Handling AI Latency: What I Changed When the LLM Took 10 Seconds to Reply
Because your RAG pipeline can be brilliant while your users are already opening another tab
There is a particular moment in enterprise Gen AI projects that I have come to recognise.
The engineering team is finally proud of the system. Retrieval is working. The application is finding the right documents. Hybrid search has improved recall. Reranking has cleaned up the context. Security filters are respected. The prompts have survived several rounds of evaluation, and the answers are considerably better than what the first prototype produced.
Then somebody outside the project team uses it. They type a question and stare at the screen.
The engineers know that a great deal is happening. The query may be rewritten, embedded, sent across multiple indexes, filtered for access rights, reranked, assembled into context and finally passed to the model.
The user sees a rotating circle.
At around ten seconds, somebody asks the question nobody on the engineering team wants to hear:
I have seen versions of this problem repeatedly. The immediate response is usually to treat it as a backend performance issue. We profile vector search. We look at model latency. We introduce caching. We parallelise calls. We debate whether the reranker is worth another few hundred milliseconds.
All of that work is valid. What changed for me was realising that not every second of AI latency can be engineered away, and that the seconds which remain have to be designed.
That sounds like a UX observation.
I stopped measuring the experience with one latency number
For years, application teams have discussed response time as though it were one continuous measurement: request goes in, result comes back, stopwatch stops.
That became misleading in Gen AI systems.
Consider two applications that both take twelve seconds to produce a complete answer.
In the first application, the user submits the question and sees a spinner for eight seconds. At second nine, a full answer appears. In the second, the application acknowledges the request immediately. The interface shows that enterprise sources are being searched. Two seconds later, the first part of the answer appears. The user begins reading while the model continues producing the rest. The complete response still takes twelve seconds.
On a backend dashboard, the difference may look modest. To the user, these are two completely different products.
That distinction changed our discussions. Instead of asking only whether we could reduce a twelve-second request to eight seconds, we began asking what the user should experience during those twelve seconds.
Streaming was the first change that consistently paid off
When a model can start generating reasonably quickly, I have found streaming difficult to argue against.
The traditional implementation waits for the complete model response and then renders it. The user experiences the entire inference duration as dead time. With streaming, the response begins appearing while generation is still underway. OpenAI’s Responses API, for example, exposes server-sent events specifically so applications can deliver output as it is generated rather than buffering the completed answer.
The interesting part was not simply that streaming made the application appear faster. It changed what the user could do.
Once the first sentence appeared, the user could start judging whether the system had understood the request. They could begin reading while generation continued. More importantly, they could discover early that the answer was heading in the wrong direction.
That is why I began treating Stop as part of latency design.
If the opening two sentences reveal that the model misunderstood the question, there is little value in forcing the user to watch another thousand tokens arrive. Let them stop the generation, correct the request and continue. That saves attention as well as inference cost.
But on the more sophisticated RAG systems, we quickly discovered the limitation. Sometimes there was nothing to stream.
The model was fast. The six seconds before the model were not.
One of the more frustrating tests involved an application where streaming itself was working perfectly.
The user submitted a question. The streaming cursor appeared. Then the cursor sat there doing absolutely nothing.
The reason was obvious once we traced the request. The system was doing considerable work before inference. Retrieval had to run. Candidates were filtered. Results were reranked. Context was constructed. Only then could the generation request begin.
That changed what we showed on the screen.
I stopped being satisfied with generic messages such as Loading… or Thinking… Instead, where the backend actually knew its current state, we exposed that state in simple language.
The screen might move through Searching, Retrieving, Comparing, Verifying and finally Drafting. In a policy application, it might show Policy Library, then 12 Sources, then Cross-checking, and finally transition into the streamed answer.
There is an important boundary here. I would not describe this as exposing chain of thought. I do not want an interface inventing a theatrical representation of the model’s private reasoning.
I want it reporting observable work. The retriever really did start. Fourteen documents really were returned. Reranking really did complete. A tool really was called. Generation really did begin.
“Searching 14 policy documents” tells me something about the system.
“Thinking deeply…” tells me nothing.
This distinction became more important as agentic interfaces became increasingly animated. There is a temptation to make the AI look busy because we assume visible effort makes waiting acceptable. I would rather make the actual pipeline visible. Enterprise users do not need theatre. They need confidence that the request is progressing.
AWS’s current guidance is unusually direct on this point as well: when a tool invocation interrupts a streaming response, the interface should surface progress rather than leave the user staring at an unexplained pause.
Then I realised we were making users wait for work they never needed to watch
The next change came from a completely different kind of application.
Imagine somebody working through a support queue. They open a ticket, review it and click Auto-Categorise.
The first version of the workflow behaved like a chat application. The button became disabled. A spinner appeared. The user waited eight seconds for the LLM to return the category. Then they could continue.
Technically, nothing was wrong with it. From a workflow perspective, everything was wrong with it.
Why did the person need to remain on that ticket while the model categorised it? They had already expressed their intent. There was no long answer to read and no conversation to follow. We had converted a background operation into a synchronous interruption simply because an LLM happened to be involved.
So we changed the interaction. The user clicked Auto-Categorise, and the interface immediately let them continue. The AI work ran behind the workflow. When it completed, the updated category appeared and a small confirmation was available.
The feature suddenly felt fast even though the model itself had not become faster. The control I cared about most was not a progress animation. It was Undo.
When an AI action changes application state optimistically, reversibility becomes part of the trust model. If the model chooses the wrong category, the user must be able to correct the result immediately rather than begin another workflow to repair the AI’s mistake.
That project gave me a useful rule I now apply much more broadly: before optimising the latency of an AI operation, ask whether the user needs to wait for the operation at all. CRM enrichment, metadata generation, document tagging, classification and similar tasks often do not belong on the synchronous path.
Once AI becomes embedded inside the workflow rather than becoming the entire workflow, quite a lot of latency can disappear from the user’s experience without disappearing from the infrastructure.
Longer-running AI forced us to stop designing for waiting altogether
Streaming and visible process states work well when the delay is measured in seconds. They become absurd when the work lasts several minutes.
A multi-agent investigation may invoke several models and tools. A large document review may have hundreds of pages to process. A financial-analysis agent may gather data, reconcile figures, investigate anomalies and prepare a report before anything useful can be returned.
There is no loading animation good enough to justify keeping somebody on that screen. Once the work crossed into genuinely long-running territory, I stopped trying to entertain the user through the wait. We converted the interaction into a job.
The user submitted the task and continued working elsewhere. The job remained visible in the product. When it completed, the notification did not merely say Finished. It carried the next useful action: Review, Copy, Share, Open Report.
The infrastructure world is moving in this direction as well. Google introduced Agent Executor in May 2026 specifically around durable agent execution, including workflows that may run for hours or days and need to survive interruption and resume reliably.
Once execution is durable, the UX no longer has to pretend the user and the agent are participating in one continuous synchronous session.
The user can leave. The agent can work. The application can reconnect the two when something useful is ready. That is a much healthier interaction model than placing a larger spinner in the middle of the page.
I eventually stopped thinking about “AI loading” as one UX state
Across these projects, what emerged was not a universal latency pattern. It was a set of different interaction modes. When the work is genuinely quick, the interface should simply feel immediate.
When generation can begin quickly but completion takes time, streaming works well because the user can begin consuming the answer. When retrieval or tools create several seconds of silence before generation, I expose real process states and then transition into streaming.
When the AI is performing a reversible workflow action that does not require continued attention, I prefer to move it into the background and give the user confirmation and a way back.
When the process is long-running, I make it a persistent asynchronous task and let the user leave. I no longer try to force all five behaviours into the same chat-shaped interface.
That may be the most important conclusion I took from the work. Gen AI latency is not one problem because Gen AI applications are not one kind of application.
A conversational research assistant, an invoice-processing workflow, a coding agent and a financial-analysis job should not share the same waiting experience merely because all four happen to call an LLM.
None of this excuses a slow backend
I still want retrieval to be faster. I still profile embedding, search and reranking separately. AWS recommends exactly that for RAG workloads because retrieval can silently consume the latency budget intended for reasoning and tool use.
I still ask whether independent calls can run in parallel, whether context can be reduced, whether connections can stay warm and whether the task really needs the largest available model. AWS’s current agent-performance guidance treats model selection, concurrency, retrieval optimisation and streaming as complementary parts of the same performance problem rather than substitutes for one another.
But I no longer make UX wait for backend perfection. That was the mistake in some of the earlier projects. We would spend substantial engineering effort removing hundreds of milliseconds while leaving the user staring at exactly the same spinner. The benchmark improved. The experience barely changed.
Conversely, I have seen applications remain computationally expensive while becoming substantially more usable because the user could see meaningful progress, begin consuming results earlier or simply continue with other work.
Backend latency and perceived latency are related. They are not the same engineering problem.
The spinner was telling us that the interaction model was wrong
I used to look at the infinite spinner and think we needed to make the model faster. Now I often look at it and ask why the user is being asked to wait. Sometimes the answer is legitimate. The person needs the model’s output before they can continue, and reducing TTFT really matters.
Sometimes the application is silent because a complicated retrieval pipeline is doing useful work that could be surfaced more honestly. Sometimes the user could have moved on immediately while the LLM completed a background action. And sometimes the task is sufficiently long-running that pretending it is still an interactive request is simply the wrong product design.
We have spent years building interfaces around deterministic software. Click something, compute quickly, return the result. Gen AI introduces a different rhythm. Retrieval takes time. Tool calls take time. Reasoning loops take time. Multi-agent coordination takes time. Generation itself unfolds over time.
Trying to hide all of that behind the same spinning circle is not simplicity. It is loss of information.
These days, when I review an enterprise AI application, I still ask how long the request takes. But I ask another question immediately after it:
What are we asking the user to do while it takes that long?
That question has changed more Gen AI experiences for me than shaving another 300 milliseconds off a vector query. The RAG pipeline can be excellent. The answers can be accurate. The architecture can be sophisticated. But while the engineering team is admiring everything happening behind the request, the user is judging the only thing they can see.
Posted Jul 27, 2026, 7:17 AM UTC