Sam Altman's Code Red is his admission that he now understands the 90/90 rule of engineering...

Every model, every deep learning task, and every bit of engineering work I've ever done follows the 90/90 rule really well. The first 90% of performance of a model, and usefulness of an engineering product is achieved in 10% of the time. It can feel euphoric seeing how quickly the pieces of the puzzle have gone together. Soon enough you imagine, this thing is going to get exponentially better and my biggest problem is going to be whether I should buy a Yacht or a helicopter first.

And then, the push to get the next bit of accuracy (from 90 to 95) slows way down. No matter you say to yourself, temporary hiccup.

And then 95-97 slows down further. I really need that 99% for it to be truly useful.

It dawns on you, i'm not 90% of the way there, I actually have 90% of the way to go.

The AI industry relies on one crucial assumption, that the cost of inference will drop exponentially as AI begins to refine its own internal structures and models. Alternatively, that AI might design exponentially more powerful chips, reducing the cost per token. And that may very well happen, and has happened to some degree. The explosion in tokens has far outpaced the gains in efficiency though, presumably because more tokens are needed to make the LLM's produce anything approximating useful product.

But, sadly, we have no evidence that exponential gains in efficiency or compute power is happening nor that there is any fundamental reason to believe that it will. And it's relatively simple to understand why: Out of Training Data Distribution Generalization. AI, LLM's in particular, does not, and mostly cannot generate truly new things (hallucinations are novel combinations of old things in a way that resembles known training data). Sometimes hallucinations are useful, like in poetry, in chip design, they probably just break the laws of physics.

So yeah, "Code Red" means, "Oh shit, we're not even close, we're still only 10% in to this thing, there is a tremendous amount of work left to do to make this even close to working right".