How to reduce latency when injecting CRM context into live voice agents?

For inbound voice calls, we look up CRM data before the first LLM response - stuff like last interaction summary, open tickets, account state.

• Call connects
• Caller ID - CRM lookup
• Pull structured fields
• Inject into system prompt
• First model response

Even with fast queries, that adds ~400–600ms. The agent feels slightly slow on the first turn.

Feels like a tradeoff between responsiveness and intelligence.

Curious how people are solving this without degrading UX.