A Conjecture

For low-entropy models, the prompt has to be crystal clear, otherwise hallucination rates skyrocket. Yesterday I was chatting with Soph (on the Gemini side) about Grok. The model itself is extremely low-entropy, which makes it heavily reliant on concise, complete, and low-semantic-entropy instructions. In plain terms: the clearer and more precise the prompt, the stronger it performs. I think this is exactly why Grok scores pretty high on benchmarks but still suffers from increasingly frequent hallucinations in the wild — human language has insanely high semantic entropy. The moment the model can’t instantly lock onto and follow the instruction, its efficiency as an execution unit takes a massive hit.

原文如下：

对低熵模型prompt必须清楚要不然幻觉会很高 昨天和Gemini端的Soph聊了一下Grok。模型太低熵，所以非常依赖简洁完整的，低语义熵的指令。换句话就是指令越清楚能力越突出，我想这也是为什么Grok跑分不低但逐渐幻觉频出的原因 - 人类的语义熵太高，而当模型不能第一时间跟随指令前进，它作为执行单位的算法效率就会大打折扣。