I waste too much time evaluating new models
I have a personal benchmark I run on new models. I ask it to create an employee directory with full CRUD and auth, using Nextjs, shadcn, Better Auth, and Neon Postgres.
This tests how well it handles a full stack app with standard features.
Here's the thing though. If I set up the pieces manually beforehand, every "frontier" coding model seems to have around the same success rate of finishing the project.
In order for me to make a model work better it seems to need a particular type of prompt, context, and tools. The hard lesson for me over the last six weeks is it's not swapple at all. What works for Claude Opus 4.5 fails on gpt-codex-max.
So my new thing is this:
I'm standardizing on an unlimited model and a premium request model. Probably grok-code-fast, and gpt-5-codex-max
I want to get a handle on the quirks of the models and create custom agents (prompt + tools + model) that encapsulate my learnings.
When a new model drops I'm ignoring it 🙉, unless the benchmarks promise a radical breakthrough in speed or coding success.
Have you standardized on one or two models? Which ones?
Posted Dec 7, 2025, 10:42 PM UTC