The Silent Killer in Your RAG Pipeline: Retrieval Quality Drift

Been running RAG systems in production for about a year now, and I've learned that the retrieval layer is where most ops headaches live. Everyone focuses on the LLM latency, token limits, or fine-tuning the reranker - but what kills you in production is when your retrieval quality slowly degrades and nobody notices for weeks.

Here's what I mean: your embedding model was solid when you launched, your BM25 baseline was reasonable, and your chunks were well-sized. But then your document corpus grows 10x, you add a new data source with a different schema, and suddenly 40% of your top-1 retrievals are off-topic. The LLM can usually paper over it (making bad context work), but now you're burning 2x tokens per query and your latency creeps up. Users complain it feels slower. You blame the LLM. You don't blame retrieval because there's no easy signal for it.

The operational fix: instrument your retrieval layer like you would any other database query. Track P@1, P@5, MRR - but do it on a sample of real queries every day. I set up alerts when retrieval precision drops below baseline (requires labeling maybe 50 production queries quarterly, but it catches drift fast). We also started logging retrieval score distributions and chunk metadata - sounds annoying but saved us twice when a data migration corrupted formatting on 15% of documents. Caught it in the daily dashboard instead of users noticing.

RAG ops isn't glamorous, but the difference between a system that works and one that slowly falls apart is whether you're measuring the retrieval side seriously. Anyone else running into this? Curious how you're monitoring it.