Your agent's retry logic dies when the agent does

I've spent the last few months getting an agent into production that actually does things( issues refunds, updates records, posts to internal tools). Not a chatbot, an agent with write access. Learned a lot the hard way and one lesson surprised me enough that I wanted to share it.

In the demo everything's fine. The model calls a tool, the tool hits an API, you wrap it in a retry decorator, done. Where it fell apart was production. The agent thinks for 40 seconds, the platform recycles the container mid-run, and the retry state that was sitting in memory inside the agent loop is just gone. Sometimes the call never fired. Sometimes it fired twice. The framework's .with_retry() was doing its job fine, but it lives and dies with the process, and LLM loops are long and flaky enough that this isn't really an edge case.

The shift that fixed it for me: a tool call with side effects isn't part of the conversation, it's a job. It should outlive the agent. Its own retries, backoff, an idempotency key so a retry can't double-charge, and some record of what actually happened. Basically the boring durable-execution stuff we already know how to do for background jobs.
So now anything that touches money or external state gets handed off to something durable instead of retried in-loop. The agent fires it and gets the result back later.

Curious how everyone else deals with this. Are you retrying in the agent loop and hoping? Reaching for Temporal/Inngest? Rolling your own queue? It feels like everyone hits this the moment their agent does something real, but I haven not seem much talk about it.