Every week the same argument flares up: open models are catching the frontier, or open models are nowhere close, depending on who you follow. I got tired of the vibes, so I built a benchmark

harness and ran the actual test. Same tasks, same automated graders, open model against Claude's frontier, thinking on for everyone.



The model is Ornith, an MIT-licensed family from deepreinforce, post-trained for agentic coding. It is a reasoning model with native tool calling, and it runs on your own hardware for the cost of electricity. I tested two sizes: the 9B dense model on a single consumer card, and the 35B mixture-of-experts model on four used Tesla P100s. That second box is the fun part. Four server cards from 2016, about $700 all in, and the whole 36 GB model sits in video memory and generates at 43 tokens a second.



\*\*The headline.\*\* On hard competition math, scored with a grader that brute-forces the answer itself so I cannot mis-key it, Ornith-35B hit 92%. Claude Sonnet 4.5 hit 88%. Claude Opus 4.7 hit 96%. The free model on decade-old GPUs came in ahead of Sonnet and within a single problem of Opus, and it matched both on coding (95% on the full HumanEval set) and on agentic tool use. The 9B told the same story on a clean suite: 27/29 against Opus's 28 and Sonnet's 29, all three within a task or two of each other.



\*\*Why you should not trust that yet.\*\* A clean number is the thing to be most suspicious of. The samples are small, so a 96-to-92 gap is one problem. The easy suites ceiling out, which means they literally cannot rank the models at the top. And while double-checking, I caught my own math grader marking correct answers wrong whenever they were written in a different form, like \`5\` versus \`x = 5\`. I rewrote the scorer and re-scored every model from its saved answers. The fix moved everyone up. The lesson is the reason I am telling you instead of quietly editing it out: read your failures before you trust your pass rate.



There was one more surprise. On the 9B, most of what looked like a capability gap on the hard tasks was a temperature setting. At the model's default it dropped four hard problems. I lowered the temperature, changed nothing else, and three of the four came back. The chasm people imagine was, in large part, one config value.



\*\*Where the frontier is actually still ahead.\*\* Here is the part worth keeping. It is not raw capability on a problem you can score. It is two specific things. First, efficiency: Opus reached 96% spending about 350 tokens per problem; Ornith reached 92% spending about 12,000. Same answer, roughly thirty times the words. Second, long-context synthesis: the small model can recall a single fact buried deep in a long document, but combining several scattered facts started failing reliably past about 37,000 tokens, and it failed the same way on a higher-quality cache and a bigger budget. That one is the model's ceiling, not a setting.



\*\*So tier them.\*\* Open local models are far closer to the frontier than the discourse admits, on any task you can score objectively, and they are free and private and run on your desk. The frontier's durable edge is now narrow and specific: conciseness, and holding a long tangled context together. So do not pick a winner. Run the open model for the private, offline, high-volume agentic work where it already holds up. Reach for Claude when the job is long, tangled, or expensive to get wrong. The gap is no longer a wall. It is a specialty.

  
Full breakdown on our YouTube channel in tonight's Evening Context. And I am genuinely curious: are you running anything local yet, or still all-API? Where have you landed?  


\*All numbers are from runs on the hardware described, with Claude run through the identical harness on the Anthropic API.

Ornith: does it really come close to Claude? I tested it.