Architecture decisions I made building a production RAG backend for enterprise knowledge bases — what I’d do differently

I recently built a RAG system for a client that needed to make two internal knowledge sources — Document360 and SharePoint, queryable through a single interface.

Here’s the architecture I landed on and the decisions I’d revisit.

The stack:
• Document360 + SharePoint as source systems (very different ingestion patterns)
• Aurora PostgreSQL with pgvector for embedding storage
• Standard chunking + embedding pipeline (sentence-transformers)
• FastAPI serving layer

The decisions that mattered most:

1. Ingestion unification is harder than it looks. Document360 has a clean API. SharePoint is SharePoint. Getting consistent metadata, versioning, and chunk-level provenance across both required a normalization layer I hadn’t initially scoped. Build this first, not as an afterthought.

2. Aurora + pgvector is underrated for enterprise RAG. Everyone reaches for Pinecone or Weaviate, but if your client is already AWS-native and data residency matters (it always does in enterprise), pgvector inside Aurora gives you retrieval quality that’s good enough and operational complexity that’s much lower. The hybrid search (vector + BM25 fallback) is what made the difference in retrieval precision.

3. Chunk strategy is a retrieval strategy. I used hierarchical chunking, smaller chunks for retrieval, larger parent chunks for context injection. The naive fixed-size approach gave poor results on structured docs like policy documents and SOPs. This was the single biggest quality improvement.

What I’d do differently:

Add a re-ranking step (cross-encoder) before context injection. I skipped it for scope reasons and it cost retrieval quality on ambiguous queries. It’s not optional for production enterprise RAG.

Also: build an eval harness before you build the pipeline. I built mine halfway through and had to retrofit it. Painful.

Happy to go deeper on the pgvector setup, chunking strategy, or the SharePoint ingestion headaches if anyone’s dealing with similar problems.

(Full architecture writeup on my medium profile.)