I thought my RAG refresh was done once the index loaded. Turns out there was one more step

I’m starting to think a lot of RAG pipelines stop one step too early.

The usual flow is something like:

• chunk docs
• generate embeddings
• insert into a vector DB
• build / load the index
• start serving queries

Once the data is searchable, the refresh is treated as done.

But “searchable” and “ready to serve production traffic” are not always the same thing.

A small test that made me notice this: Milvus 2.6.17, single-node Docker Compose, 1M vectors, 768-dimensional embeddings, HNSW, same search params, same 16-core / 64 GB machine. The only thing I changed was the layout: 3 sealed segments before optimization, 1 segment after force merge.

Search QPS went from around 3,000 to around 5,600-6,000.

The data was already loaded and indexed before optimization. This was not a recall issue or an embedding issue. The physical layout still affected serving performance.

Before the merge, each query had to fan out across multiple segments and merge partial topK results. After the merge, there was less fan-out and less merge overhead.

This made me wonder whether RAG refreshes need a more explicit “serving prep” phase.

Something like:

• ingest new docs
• build / load index
• verify freshness
• run any needed optimization / warmup
• then route production traffic

Obviously, you would not want to run heavy optimization all the time. It can use CPU, memory, and disk I/O. But after a large KB refresh, it may be worth treating it as part of the refresh pipeline instead of leaving it entirely to background maintenance.