Hey folks,

I’m working on a real-time voice agent and running into what looks like a cold-start issue, but I’m not able to clearly pinpoint where it’s coming from.

Stack :

• LiveKit (self-hosted on EC2)
• Deepgram STT → model: nova-2
• LLM: gpt-4o-mini, gemini-2.5-flash-lite, or llama-3.3-70b-versatile
• ElevenLabs TTS → model: eleven_turbo_v2_5 (mostly), fallback sometimes to eleven_turbo_v2

Problem :

What I’m seeing consistently:

• First call after long idle (like morning or after some inactivity) → high latency
• After 1–2 calls → everything becomes fast and stable

So the pattern is:
Idle → first call slow → rest fast

What I’ve already ruled out :

• LiveKit is self-hosted on EC2 → always running → shouldn’t have cold start behavior
• I’m already doing pre-warm calls → still seeing this

My understanding so far :

I checked a bit about cold starts and most of the discussion points to serverless systems.

But in my case:

• LiveKit → not serverless (self-hosted)
• Deepgram / ElevenLabs → I couldn’t find any place where they explicitly say they are serverless

They mention things like:

• multi-tenant cloud
• managed APIs

But nothing clearly saying:
“we scale to zero” or “serverless”

Where I’m stuck :

Even though they don’t explicitly say serverless, behavior looks very similar to:

• connection setup cost
• model/resource initialization
• first request overhead

Also saw in Deepgram docs that:

• WebSocket connection has a one-time setup latency

So trying to understand:

Questions :

• Has anyone seen this exact pattern with Deepgram (nova-2) or ElevenLabs (turbo v2 / v2.5)?
• Do these systems internally behave like “cold start” even if not labeled serverless?
• Or could this still be coming from LLM / connection reuse issues?

Would appreciate if anyone has seen this or has any concrete proof / explanation 🙏