While reviewing an LLM output-stability benchmark, I found a latent gap between its documented scope and its scoring pipeline.

Tool-call responses weren’t supported, but the response parsers could erase them:  
The OpenAI adapter used message.get("content") or "". A tool-call response with null content would become "".

The Anthropic adapter kept only text blocks, dropping tool\_use blocks.  
The scorer excluded explicit errors, but accepted empty strings.

Given those samples, the scorer would see identical empty strings: one distinct output, byte-identical results, and mode share 1.0.  
That would measure the stability of the fallback not the tool calls.

To be clear: this was traced in source, not reproduced in a live run. Current request builders never forwarded tools, so existing cases couldn’t reach this path. The maintainer checked all 563 recorded non-error samples: none were empty, and no published benchmark was affected.

The fix enforced the documented boundary: reject cases carrying tools, mark empty non-error completions unsupported, and exclude them from successful samples.

The broader lesson: preprocessing can erase the behavior you intended to measure. If “unsupported” becomes a valid-looking default, a reassuring score can hide the missing measurement.

How do you distinguish unsupported responses, parsing failures, and genuinely empty outputs in your eval pipelines?

How an unsupported tool-call response could become “perfectly stable” in an LLM benchmark