retrieval metrics tell you what came back, not what the generator used. how are people actually measuring the second one?

retrieval metrics tell you what came back. none of them tell you whether the generator actually used it. i have been stuck on measuring the second one and want to know what others landed on.

the gap

context precision, recall, mrr, ndcg all measure whether the retriever returned the right thing. none measure utilisation.

those come apart more than i expected. in a run i was debugging, the top result scored 0.910 and contributed nothing to the answer, while a 0.340 result is what actually answered it. by retrieval metrics that run looks fine.

it matters because the two failure modes need opposite work:

what you see	what is actually wrong
high score, not used	retrieval is fine. the problem is your prompt, context ordering, or position in a long context
low relevance, used	the generator grounded on bad evidence

without separating them you tune the wrong stage. i have definitely spent days tuning a retriever that was already working.

the hard part: estimating "used"

method	cost	fails how
lexical overlap, answer vs chunk	zero extra inference	completely fooled by paraphrase
embedding similarity	cheap	conflates topical similarity with actual use. a chunk on the same subject scores high either way
nli entailment per chunk	one small model pass per chunk	entailed is not used. flags a chunk that supports a claim the model got elsewhere
leave one out ablation	n extra generations per trace	closest to causal, but it is a batch eval method, not a debugger
attention attribution	needs logits or internals	off the table for anything behind an api

i went with lexical overlap at a 0.2 threshold, because i wanted it to run while debugging with zero extra inference, and i label it as a heuristic in the ui rather than presenting it as ground truth.

i am not happy with it. it is the weakest part of what i built.

two questions

1. has anyone actually validated nli entailment as a proxy for utilisation? it looks like the right middle ground and i cannot find anyone who checked it against leave one out.
2. does anyone run leave one out in a real eval loop, or does the cost kill it in practice every time?

adjacent finding, for anyone doing graph retrieval

i built the corpus as a knowledge graph with typed weighted edges and per type recency decay. the nodes retrieved constantly and never used turned out to be the over connected ones, not the irrelevant ones.

once i ingested per pr file lists, the repo node touched everything and started appearing in almost every result while never being the evidence for anything. hub suppression had to become a property of node plus relation rather than of the node, since dropping the whole node also dropped its useful edges. the naive version was silently discarding 47% of results.

---

check here as well: https://graphsight.vercel.app/

the tool this came out of is graphsight, mit, runs locally, renders a run as a graph with used and ignored items drawn differently. more interested in the measurement question than the tool.

if you think lexical overlap is too weak to be worth shipping at all, say so. i would rather hear it here than from a user.