Eight tiny LLMs on a $250 Jetson Orin Nano Super — what I learned about running inference at the edge

I spent the last week running 8 small language models, from 135M parameters all the way to 1.2B -- on a single Jetson Orin Nano Super 8GB.

The models I tested:

- SmolLM2-135M
- SmolLM2-360M
- Qwen2.5-0.5B
- LFM2.5-350M
- LFM2.5-1.2B
- Qwen3-0.6B
- Llama3.2-1B
- Gemma3-1B.

All running on both llama.cpp CUDA and Ollama, across all four Jetson power modes - 7W, 15W, 25W, and MAXN.

Why both backends? Because I wanted to know if theres  any real, noticeable difference between llama.cpp and Ollama inference and it turns out llama.cpp beats Ollama at sub-1B and almost same 1 B models.

Here's what I found.

At SmolLM2-135M Q4_K_M under llama.cpp at 25W:

- up to 165 tok/s (Ollama: 121 tok/s), 29.6 output tok/J (Ollama: 21.3)
- 0.31 s TTFT at ctx=2048 (Ollama: 0.46 s) -- llama.cpp is 1.37× faster on throughput, 1.39× on tok/J
- 487 total tok/J at ctx=2048, gen=64: best in suite

At LFM2.5-350M Q4_K_M under llama.cpp at 25W:

- 115 tok/s -- nearly matching SmolLM2-360M (369 MB) in only 219 MB
- Ollama drops to 28 tok/s at the same mode -- 4.20× gap, purely a kernel issue
- 17.16 output tok/J (Ollama: 6.39)
- 0.39 s TTFT at ctx=2048 (Ollama: 0.50 s)

At LFM2.5-1.2B Q4_K_M under llama.cpp at 25W:

- 54.1 tok/s: leads the ~1B class (15 % over Llama3.2-1B at 47.1, 33 % over Gemma3-1B at 40.8)
- Ollama: 21.8 tok/s -- llama.cpp is 2.48× faster
- 6.37 output tok/J (Ollama: 3.94), 1.03 s TTFT (Ollama: 1.11 s)
- Only 698 MB -- smallest footprint in the 1B class

Benchmark Methodology

- For each model × prompt × gen combo, aiperf sends 20 single-concurrency requests with synthetic prompts at the exact target token count.

- Power is sampled from tegrastats VDD_CPU_GPU_CV (mW → W) at 500 ms intervals. Tegrastats samples are assigned to exact prefill/decode phase windows using per-request nanosecond timestamps from profile_export.jsonl (aiperf's stats).

- Clocks were locked with jetson_clocks at all modes. Each run's power and clock speed was capped through nvpmodel and monitored for thermal stability (no sustained throttling; junction temp ≤ 73 °C).

- Latency percentile used throughout: all TTFT, ITL, and request latency (RL) values reported use the p50 (median) over the 20 requests per combo.

Analysis [here](https://www.smolhub.com/posts/jetson-nano-super-benchmark-non-reasoning/)

Tiny Jetson Orin Nano Super Benchmark Across 8 models | The Ollama vs llama.cpp story