A new AI coding benchmark's authors said they never ran the "write clean, careful code" prompt. I ran it 22 times — it did worse than the plain one.

There's a benchmark called SlopCodeBench that I think is a genuinely good idea. Instead of giving a coding agent one spec, it gives it a small one, then changes the requirements four or five times, testing after each round. To clear a stage you have to build the new thing without breaking what you already built. It's much closer to how requirements actually arrive than a single frozen prompt is.

It got attention recently because nothing finished it — not even the problems marked Easy.

But the write-up had a footnote I couldn't stop thinking about: they'd only run the bare "just solve it" prompt. The benchmark also ships a prompt variant that tells the model to write clean, careful code and avoid the usual mess — extra defensive checks, single-use variables, trivial wrappers, if/else ladders, a pile of helper methods. They said they hadn't tested it.

That's the prompt everyone reaches for. So I ran it.

Setup: two Easy problems, two prompts, two effort levels, everything else held constant. 30 runs attempted, 22 valid — 7 died on an agent-side crash and 1 aborted. I threw the crashes out rather than count them, which matters more than it sounds: a crashed run writes zero lines of code and then vacuously passes every error-handling test, because a program that does nothing handles all input correctly. Left in, they'd have quietly flattered whichever arm they landed in.

Result: it didn't help, and on one problem it clearly hurt.

• Problem 1: neither prompt cleared a single checkpoint. Four runs each. Nothing.
• Problem 2: the plain prompt cleared checkpoint 1 in 4 of 4 runs. The careful-code prompt managed it in 1 of 5.

Same model, same problem, same starting code. The only thing that changed was being told to be careful.

Nor does more compute rescue it. At high effort the careful prompt wrote 78% more code than the plain one at low effort (313 → 556 mean lines) and landed in exactly the same place. More care and more thinking both bought substantially more code for the same or worse results.

My read on why: the benchmark never tells the agent which tests failed. That's the whole design. And you can't find a requirement you were never shown by thinking harder about the ones you were. Careful prompting and extra compute both operate on the information you already have — neither one manufactures the feedback channel that's missing. If anything, "be thorough" spends the budget widening code you can't verify.

Two caveats, because they're load-bearing:

1. Two problems, not the whole benchmark. 36 exist.
2. I had to run a newer agent version than the benchmark pins, because the pinned one can't even make a valid request to this model. So my numbers are not comparable to the pass rate being quoted around — only the two arms against each other, which share a version.

And the thing I'd most want said out loud: this is not "the model is bad at coding." The benchmark deliberately forbids the feedback loop. It measures one-shot, no-feedback, hidden-spec work, which is close to the opposite of how anyone sensible uses these tools. What it's really measuring is how far an agent gets with the loop switched off — and the answer is that the prompt you'd instinctively reach for doesn't move that needle.

Full breakdown with the charts: https://youtu.be/IUWJ9Rnm1zE

If anyone's run the careful-code variant on the harder problems, I'd like to know whether it flips — n is small and I'd rather be corrected than quoted.