Recently there has been an influx of AI generated images, texts and what not in our daily lives. So I thought this is a good time to make people a little more literate about what an LLM is, how they work, how your complex question is answered by a chatbot which sounds erringly similar to a human etc. This will be a long post so please pick some time and grab some snacks.

(with all the AI generated slop anti-intellectualism is on the rise so please take some time and read this and prevent yourself from going down in that hole)

What are LLMs?

LLMs stand for Large Language Models. These are Machine Learning Models trained to do only one thing: predict the next word given a sequence of words as input.

Example:

Input - "My dad went out but he forgot car's"

(Read the sentence in your mind and you will also, most likely, say the next word should be Key, that's the same behavior we want from our model)

Output - "Key"

Now, after getting "Key" we append this new word to the previous sequence and feed it back to the model to get the next word and repeat this cycle.

But why they got popular now?

Before LLMs, we had many different approaches for text generation. From Recurrent Neural Networks to Long Short-Term Memory and even Rule-based text generation. These models/systems had a bottle neck: they didn't work well on long sequences and they were hard to run efficiently on GPUs. But in 2017 Vaswani et al published a research paper titled "Attention is all you Need" that introduced a new architecture of "Transformers". This was the changing point. Transformers gave us the ability to parallelize the whole process on modern GPUs which in turn increased the performance of such models. OpenAI was one of the first labs to adopt these new architecture and they released ChatGPT-2 in 2019 which showed how much potential this new architecture holds. This was the same year Covid started and since most of the people were home the use of such chatbot became very popular. At this time these models were great at text fluency and could do some general task but clearly failed at complex tasks or reasoning. The researchers realized we just need to make bigger models, which they did in the coming years and the capabilities also increased, from basic reasoning to advance and now competing with PhD level humans in terms of domain knowledge.

How can a simple 'next work prediction' lead to this?

What I said earlier: "LLMs are trained to predict next word given an input sequence" is true but it is not the only reason the models work so well. With more and more researchers and labs chiming in the LLM train the original Transformer (also called Vanilla Transformers) started getting new upgrades. These upgrades ranged from specific data curation, improvements in training pipeline, architectural changes, hardware improvements and much more. From 2019 to 2024/5 researchers believed one thing: throw large amount of data and increase model size and that would work. But such large models are expensive to run (details in next section) so they now came the time to explore new efficient methods. We got o1 in September of 2024, that model introduced a whole new way for LLMs to generate new words. The simple idea was: before allowing the model to answer a question let it think. Yes, allowing a model to think before its final output is showed to the user drastically improved performance. If you have ever used thinking mode/ max mode etc. in a chatbot you are asking the model to think before it answers your question. And by "think" here I mean the model is trained to plan out what it understands about the given problem, try different approaches, self verify and finally answer what it finds as a correct response. This thinking is done by generating words only but the end user is not shown those words (only in the case of proprietary models - open source models do). These all came from the fact that we can see the task of generating new word from a statistical point of view rather than a rule-based point of view.

How do we run such models?

We need GPUs - Graphics Processing Unit(s). CPUs are powerful too but they have very low cores (workers) which slows down the whole process and overall they are designed to handle multiple tasks at the same time through context switching and other wizardry which is too much right now. Since the underlying task in a Transformer is to do thousands of matrix operation (we store the weight of a model in matrix form) GPUs are the perfect fit here. GPUs are designed to do such boring repetitive tasks in blazing speed and in parallel. Parallel means you can assign the same task to multiple cores (workers) in a GPU and they will all do it parallelly and give output in the same time. This is much more efficient than doing it sequentially where you have to wait for one output to proceed to the next step. We need to store and run the whole model on a GPU and a model can have a size from 2GB to 1TB (1TB = 1024GB). The current price of an Nvidia rack that can run a huge model (500+GB) goes in range of 3-5 Million USD. This is just the cost of GPUs, you will also need cooling system, power management, CPUs, RAM etc. After you have got all of these, you will need to pay the electricity cost which can vary very much based on the area/country.

How LLMs work in modern Chatbots:

So now the next time you to ChatGPT and ask "What's the weather in Delhi today?", this is what happens:

• The LLM recognizes that weather is a varying thing and it needs to fetch the latest weather update from a forecasting agency/website.
• It uses 'tool-call' to search the web - Basically every model instance is given a set of 'tools' which it can use: from searching web to generating an image or excel sheet.
• The web search returns the latest weather report of Delhi. The model uses that to give information to the user. (Basically you did a web search with keeping LLM as a middle man)

Some technical words and their meaning:

Tokens - Language has very nuance and there are potentially thousand if not millions of words in English language alone. So researchers found a way to convert this into sub-words aka Tokens that a model can always recognize.

Example: "Thinking" -> "think" + "ing" (the model might have never seen thinking but it may have seen think and ing so it will know what we are trying to say here)

I honestly don't know what other words a layman might have heard without knowing its meaning - so please comment down below and I will try my best to explain them.

That's it for now. If this gets enough interaction I will post a next part explaining more topics (maybe the geopolitics aside, the recent ban of Anthropic's Claude Fable or even the internal AI lab drama stuff)

Fun fact: Anthropic's model family is called 'Claude' and it is named after Claude Shannon, who was the first person to formalize the idea that language can be treated as a statistical source and studied via next word prediction all the way back in 1950s

Thanks for reading. Please comment any of your doubts or opinions.

Reminder: This is a non-technical post so I have simplified many things which do not represent my understanding of the matter so don't call me out - I know stuff.

(My DMs are open if you wanna have more discussion around this)