Serving 200 to 300 custom HF models on a single H100 node with bursty traffic. Here’s what broke first.

We’ve been running a reference deployment focused purely on long tail custom models from HF. Not foundation models, not 24/7 traffic. Think small fine tuned models that get hit sporadically.

Right now we’re serving around 200 to 300 distinct custom models on a single H100. Traffic is bursty. Most models sit idle most of the time.

A few things we’ve learned:

1.“Scale to zero” is not enough by itself.

If your restore path replays container pull, framework init, weight load, CUDA context, kernel warmup, you are just hiding the cold start, not solving it.

2.Warm pools quietly turn into manual capacity planning.

A lot of setups end up prewarming with dummy calls. At that point you are basically running your own warm GPU fleet.

3.Multi model scheduling becomes the real problem.

It’s less about raw throughput and more about deterministic restore and eviction policy under memory pressure.

4.Billing alignment matters more than peak latency.

For bursty workloads, users care more about not paying for idle VRAM than shaving 50 ms off steady state latency.

Not sure how others here are handling long tail deployments.

Are you prewarming? Keeping models resident? Relying on autoscale?

What’s your restore time from zero to first token for a 10 to 20 GB model in practice?