Bypassing GC thrashing in high-frequency on-device LLM streaming

Spent the last few days profiling on-device token streams via the AI Edge SDK. Streaming local LLM responses (Gemini Nano) at 30 tokens/sec allocates hundreds of temporary string objects per second. This triggers constant garbage collection sweeps on the main thread and drops UI frames.

Bypassed the allocation spikes by using conflate() on the Kotlin Flow and updating the UI state in chunks rather than single characters:

// Buffer intermediate text tokens on NPU background thread
generativeModel.generateContentStream(prompt)
    .flowOn(Dispatchers.Default)
    .conflate() // Merges emissions if downstream collection is slow
    .collect { chunk ->
        // Batch append to avoid constant Compose layout invalidations
        uiState.updateTextBatch(chunk.text)
    }

I've pinned the open-source sample repository containing this Hilt-injected chat implementation on my profile (u/yogirana5557) if you want to clone/run it.