I built an open-source configurable RAG pipeline comparison harness — looking for feedback
I’ve been building an open-source configurable RAG pipeline comparison harness and would like feedback from people who have evaluated or built RAG systems.
Repo: https://github.com/icpinto/RAG-benchmark
The goal is not to claim this is a definitive RAG benchmark. It is a practical harness for comparing different RAG pipeline configurations locally and seeing how design choices affect retrieval quality, latency, and operational complexity.
Current support includes:
| Stage | Implemented variants |
|---|---|
| Extraction | TXT, Markdown, PDF, HTML/URL |
| Cleaning | Basic cleaner |
| Chunking | Fixed-size, recursive character chunker, LlamaIndex fixed-size, sentence, hierarchical/parent-child, semantic, and document-structure splitters |
| Embeddings | SentenceTransformers: MiniLM, BGE small, configurable local HF model |
| Indexing | Qdrant dense strategies (dense_hnsw, dense_hnsw_tuned, dense_exact, dense_quantized, dense_on_disk, dense_payload_indexed), OpenSearch BM25 strategies (bm25_standard, bm25_field_weighted, bm25_custom_analyzer, bm25_phrase) |
| Retrieval | Dense, metadata-filtered dense, BM25, hybrid reciprocal-rank-fusion, query-adaptive hybrid routing |
| Query rewriting | None, HyDE via Ollama, multi-query paraphrases, step-back conceptual queries, decomposition, local synonym expansion |
| Reranking | None, CrossEncoder reranker, FlashRank, RankLLM, ColBERT, Cohere, Jina, diversity/MMR reranker |
| Context compression | None, top-N trimming |
| Generation | Ollama, default model gpt-oss:20b |
| Metrics | recall@k, precision@k, MRR, NDCG, latency, index size, simple citation validation, placeholder answer metrics |
Specific feedback I’m looking for:
- What would make this genuinely useful for comparing RAG pipelines?
- Which metrics are missing, weak, or misleading?
- Is the pipeline abstraction sensible, or too complicated?
- What existing tools or benchmarks should I compare against?
Posted Jun 23, 2026, 10:31 AM UTC
10