Disclosure up front: this is my own project (ClawIndex), one-person shop. Not selling anything here, the writeup is free to read and I’m mostly after criticism from people who do retrieval seriously.

The setup: a retrieval approach that does a reasoning pass over an index instead of pure vector similarity. No embeddings, no vector DB. The entire benchmark ran on self-hosted Qwen, nothing left the machine.

Benchmarked against FAISS and BM25 across 700 queries: 500 HotpotQA + 100 BEIR ArguAna + 100 BEIR SciDocs.

What it won:  
• NDCG@10 on all five dataset splits (0.934 on  
HotpotQA full set)  
• Biggest gap on multi-hop bridge questions:  
0.920 vs FAISS 0.837  
• 51/500 HotpotQA queries hit a fallback path; all still resolved to valid traced results  
Where it loses / caveats I want to be honest about:  
• FAISS beats it on MRR@10 on both BEIR  
datasets  
• SciDocs margin (0.975 vs 0.972) is within noise at n=100, no confidence intervals yet, so I'm calling it directional not a win  
• HotpotQA was the distractor setting, not fullwiki. My comparison to a published system (PRISM) may be apples-to-oranges since I haven't confirmed their corpus setting  
• \~27 seconds per query. FAISS is 7ms. This is the real cost and it's not small

Honest take: it’s slow and it’s built for a narrow job, async work where a traceable, fully-local answer beats a fast one (contract review, compliance, anything that can’t touch an external API). For real-time search, FAISS wins, no contest.

Full tables and method in the link. Genuinely want to know what I’m measuring wrong or what else I should test!

I benchmarked my reasoning-based retrieval system against FAISS and BM25 on 700 queries, running everything on local Qwen. Results + where it loses