Hey, I wanted to share my local llama-swap setup with you, as i finally came around to creating a boilerplate for it  
  
  
 the boilerplate dockerizes the entire setup and makes managing multiple LLM models much easier.  
  
  
The key features:  
\- Fully dockerized llama-swap setup that runs in a container  
\- Docker-in-docker support for spawning vLLM containers on demand  
\- Merge config system that automatically combines YAML configs from subfolders, making it easy to organize models by provider or type  
\- Examples for three different model setups: local GGUF files with llama-cpp, GGUF models from HuggingFace with llama-cpp, and vLLM containers running in Docker  
\- Traefik reverse proxy integration with automatic SSL and routing (it assumes you have a running Traefik Instance)  
, plus instructions for running standalone   
  
  
I added the merge\_config logic to make everything more organized, since managing a single big config file gets messy when you have lots of models. Now you can put your model configs in separate subfolders like models/ibm/, models/deepseek/, etc., and it will automatically find and merge them into one config file.  
  
  
The vLLM setup uses docker-in-docker to spawn containers dynamically, so you get proper isolation and resource management. All the volume mounts use host paths since it's spawning containers on the host Docker daemon.  
  
  
This post and the boilerplate was written with AI assistance.  
  
  
  
I just wanted to get this out there for now, as it took some time to get it running,   
but right now im pretty happy with it.  
  
  
I left my Model configs in , they are configured for a system with 2x3090 + 128GB DDR5 RAM  
the Model Configs that use local gguf files would need to have the model downloaded ofc. the Configs that reference hf repositorys should work right away.  
  
  
Would love some Feedback. please bear in mind that i mostly published it to be able to link it, because i came around multiple posts/comments already that were referencing llama-swap and vllm (over the past months) and i was getting a bit tired to explain my setup :D so its not really polished but should give people a good starting point.  
you probably can use it to use other dockerizable inference engines aswell (iirc in the llama-swap repo someone wanted to have ik-llama support in llama-swap )  
  
  
(the last part after AI disclaimer was written by human, as you can probably tell haha)

  
I hope im allowed to post it like this, if not feel free to tell me to remove it (or the link) 

Llama-swap + vllm (docker) + traefik(optional) setup