It's ALIVE! New AI homelab passed the wife test!

Recently decided to invest in getting a homelab setup for local AI models so we can bring as much of our AI chats and basic work back in house as possible. My wife loves to ask ChatGPT for stuff when she is doing creative writing (she uses it to get feedback and critiques while drafting). She also likes to use image generation from time to time. So I decided we might be able to do this at home (plus I wanted to find justification to invest on some new hardware). As of today, I have the full stack working and it officially passed the wife test as she says the workspace setup for her creative writing assistant is (in her words) giving better answers than ChatGPT ever did. Here is the stack I setup to do this. Really thrilled with how it has come out so far and how power efficient is actually turned out to be!

Happy to answer questions or take feedback on what I can improve here from some that are probably more versed than I am.

Client docker server is a N97 with 16 GB of RAM running on Ubuntu server with docker.

AI client stack running Gluetun with split tunneling to allow for local network access while providing privacy for all containers running inside.

Inside the stack:

• openwebUI - primary chat user interface for the house
• SearXNG - metasearch for both the home users and for openwebUI
• Playwright - web scraper for gathering the raw data from the search results (I also set the search to Bypass Embedding and Retrieval so it hands the full result set to the AI models)

To run the AI models I am running a Strix Halo system with 96GB of RAM with 80GB allocated to GTT for AI models

This runs 2 llama server instances

• First docker is explicit for the home AI model the family uses (currently running Gemma 4 26B A4B APEX)

• The second docker uses the router configuration so I can experiment with models during periods when the family isn't using the agent

• Context window set at 128K per user with 4 users max configured (VK set to Q8)

• This server also runs a forge container with ROCm 7.x (this was a pain to get working) for stable diffusion using Flux.1 for image generation

All of this is setup with specific workspaces in OpenWebUI that have been tuned for different purposes (analytical research and direct answers, creative writing and feedback, prompting for image generation).

So far the server is performing exceptionally well and currently seems to stay under 60GB of total usage when generating text responses and image generation. It is reasonably fast (between 20 and 40 ish tokens generated per second with about 400 tokens/s prompt processing). Image generation takes about 2-3 minutes for a 1024x1024 image as well.