We've been benchmarking our Verbis Graph on CINECA's Leonardo supercomputer, and we saw a pretty significant improvement

Over the past few weeks we've been refining the retrieval pipeline behind Verbis Graph and rerunning the benchmarks.

Latest NFCorpus results:

• Hit Rate: 92.26% (previously 89.16%)
• Mean Recall: 46.57%
• MRR: 0.578 (up from 0.303)

We also ran our first evaluation on SciFact:

• Hit Rate: 98.0%
• Mean Recall: 98.0%
• MRR: 0.601

The number we're happiest about is the MRR improvement. Doubling it means relevant documents are appearing much earlier in the ranking, which is exactly what we were trying to improve. For GraphRAG systems, getting the right evidence near the top is usually more important than simply retrieving it somewhere in the top 1000.

These experiments are running on CINECA's Leonardo HPC infrastructure, mainly because we're interested in understanding how graph-based retrieval behaves at larger scales.

Next on our list:

• ontology-enhanced retrieval
• finance and legal benchmarks
• reproducible GraphRAG evaluations on HPC

I'm curious how others here are evaluating GraphRAG systems.