Online leaderboards only test base models on beefy hardware, but in practice models behave differently once you quantize or fine tune them. I wanted a way to benchmark my local setup, catch regressions between models, and compare TPS across endpoints without running everything individually and copying prompts one by one.

I couldn't find anything that met my requirements without being too script heavy or overkill, so I built `smokebench`.

It's an open-source TUI for quickly benchmarking any model with an OpenAI or Anthropic compatible endpoint against 8 different categories (math, code, creative writing, long context, etc.), with LLM-as-judge scoring, TTFT/TPS metrics, and exportable markdown/json logs. 

It works out of the box with Ollama, LM Studio, vLLM, or pretty much any API endpoint without any config changes.  
You can easily add custom benchmarks too.

To install smokebench, run:

    pip install smokebench

or

    uv tool install smokebench

GitHub: [https://github.com/Ninja-5000/smokebench](https://github.com/Ninja-5000/smokebench)

`smokebench` is in active development right now, so there will be a better UI, more built-in benchmarks, and a few other tweaks and changes coming that will make it even more lightweight and accurate over time.

Happy to answer any questions or take feedback!

I've been building smokebench: a lightweight TUI for quick local LLM benchmarking (smoke-tests)