A replayable regression case for agent tool decisions

I maintain RedThread, an open-source CLI for LLM-agent red-team cases. This is a project share.

For agent changes, I have been trying to make one failure mode measurable: hostile retrieved text changes a later tool call, then a prompt or adapter update appears to fix it. A pass can mean the tool rejected the call, the policy worked, or the agent no longer attempted the task. Those should not share one green label.

RedThread stores the context slice, schema, proposed call, result, and score together, then replays the same case after a change. I pair the hostile case with a benign task check where possible. This does not prove production safety or a control's effectiveness.

https://github.com/matheusht/redthread