LiteLLM + vLLM debugging note: the backend might not be the problem

I spent way too long chasing vLLM last week when the problem was actually in front of it.

My setup: LiteLLM in Docker, two vLLM backends, and apps only talking to one OpenAI-compatible URL.

The three things I’d check first next time:

1. Restart LiteLLM after model config changes.

   I added an API key to a vLLM backend and kept getting 401s even after updating LiteLLM and restarting vLLM. LiteLLM was still using the old cached keyless deployment. `docker restart litellm` fixed it immediately. If your model config lives in LiteLLM’s DB, treat key, URL, and param changes as restart-required.

2. Make sure LiteLLM sends the exact served model name.

   The LiteLLM alias is for your clients. The value after `openai/` is what gets passed to vLLM. That needs to match `--served-model-name`. If not, LiteLLM can give you 404s while vLLM works fine directly. Check `/v1/models` on the backend and copy that value.

3. Don’t point a container at `localhost` unless you mean the container.

   With LiteLLM in Docker, `http://localhost:8000\` means “inside the LiteLLM container”. Use the backend’s real IP/hostname, `host.docker.internal` with `host-gateway`, or `172.17.0.1` depending on your setup.

One more: vLLM being `active` in systemd does not mean it is ready to serve. Wait for `/v1/models` to respond.

Hopefully useful to someone else who is busy debugging the wrong machine.