Here’s Why Language Models Hallucinate… and How to Reduce It

I stumbled on this paper that changed how I look at LLMs.

Georgia Tech professor Santosh Vempala partnered with Open AI researchers to drop a paper that mathematically shows why language models hallucinate. Turns out it’s just a statistical side effect of pretraining itself.

Here’s the TL;DR:

• Pre-training encourages hallucinations because language models are designed to replicate the distribution of their data, not to validate truth. If they haven’t seen every possible true statement, they’ll inevitably generate false but plausible ones.
• Post-training is where we can change behavior. If we treat ‘I don’t know’ as neutral and wrong answers as penalized, the model learns restraint.
• Calibration = type safety for probabilities. When it’s off, your model compiles fine but executes false logic confidently.

Meaning: if your model can’t tell valid from invalid statements, it will inevitably hallucinate, no matter how advanced the architecture.

The crazy part?
Current evaluations penalize “I don’t know” the same as being wrong.
So the model learns to guess rather than abstain if it doesn't know the answer.

It’s like giving your model negative points for not hallucinating.

For the full paper and the reading of it here's the link:

 👉 Full video + visuals here

(If you’ve ever debugged a GPT agent that “knows it’s wrong but won’t stop talking,” this explains why.)