Entropy is a bitch, and we’re all going to see the effects of it in 3, 2, 1 … Everybody is talking about the avalanche of new, cheaper, large language models (LLM’s) and other AI implementations. DeepSeek, also known as CheapSeek seems to stand out because it’s based on the ‘heavy lifting’ done by other LLM developers and is quite literally a cheap knock off. So can it be good?
People seem to both like the idea of a cheaper ‘production line for LLM’s’, but also wonder: do I still get enough bang for my buck? Well, I hate to disappoint, but you already didn’t and depending of your perspective: it just got progressively worse!
Inbreeding models is how we make AI
People in the know will probably fight me on this, bus as I understand it. After learning from a know dataset AI will try to not only validate it’s outcome (the growing model) by comparison of the original data, but more and more rely on growing autonomously and measuring it’s growth by checking itself with older and newer incarnations of itself.
Basically making an AI is a self confirming series of operations where statistical ‘truth’ is created by a recursive version of the telephone game, with a cheat sheet of passed on in post-it notes. In math we have a concept called entropy, the nature of models to degrade to a natural state of disorder. And data, or model entropy is a thing too!
To be sure: the LLM training method is holy effective! Chat GPT is pretty impressive. BUT, as several experts already have indicated: the maturity and capability of AI don’t only grow through computational power, but also on more data. The lack of more data creates validation poverty or draught, giving data entropy a better foothold on the quality of the model. Without new data, the AI model will become more a cartoonish version of its previous self.
If we know this lack of variety in nature as well. It’s called inbreeding, or procreation within a small gene pool, where some attributes will become stronger and stronger, because they are rewarded by gene domination. However, this also goes for the tiny mistakes that might not be apparent at first, but will become even dominant.
Can we trust cheap AI models?
So, we start with the premise that basing newer versions of yourself on increasingly limited and self exaggerating data and perhaps corrupt base models might be bad for the quality of LLM’s as we use them now. How do we expect newer and cheaper AI models to perform? Based on the data we see greater methods to be faster, generated more cheaply, and that they are using the huge shortcut of ‘not doing your own homework’ and copying the homework of others.
By a quick glance at the cheap models we at least see that the short comings of base models are quite apparent in models like DeepSeek. It seems to have copied the wrong answers so verbatim that it’s awkwardly apparent that we’re not talking to something that did all it’s homework. And really cute: it thinks its ChatGPT. Aww.
Coming back to the trust issue: I think we should be able to trust an AI model, but this relates to a deeper question: How do we use AI models?, or phrased more accurately: what are AI models to us? My short answer:
AI models are not a source of fact, but can be a source of inklings, or notions, or sparks for ideas, even reappropriated creativity.
The main take away is: please don’t use any of the AI models, or LLM’s as a source of truth, accuracy, or even understanding. It is really like talking to a severely limited parrot like entity that has the capacity to learn how to fool you to seem intelligent, but just learned really well what you might like.
From a face only a mother can love to awesome derps!
If you want to recognise an AI a thing that acts human, you will always be disappointed. You will always stare into the uncanny valley and (hopefully) never trust anything that comes from that source. And that it okay! Because the current LLM’s are great to do other, more wonderful tasks, like randomly offer up things that others might, or might not have said. Given a smart prompt, they can even generate things you might not meant to get, but got anyway.
In that regard, AI is a great way to do broad discovery of your own ideas, or perhaps a very advance version of dice, or magic eight balls, giving your train of thought random directions. And given that your model is not too much inbred, you might want to have it generate the lists and steps based on all the other lists and steps on the internet. Make sure have it mention its sources, because AI is worse than Wikipedia, it’s a warped and execrated reflection of humanity on the web.
