Local Inference on Apple Silicon with sub-second wait times

With oMLX, it’s possible to have fast, local, and quite decent quality roleplay in SillyTavern using Gemma 4 models across hundreds to thousands of turns (depending upon response and input length) over contexts extending 262,144 tokens. But SillyTavern's stock settings change early bytes of your prompt on every turn. This busts prefix caches, which prevents oMLX from reusing any of the work it did on the previous turn and results in ever-increasing waits for tokens the longer the conversation goes. Here are six UI changes take measured cache reuse from 0% to ~99%+ (for sufficiently-long contexts) cutting per-turn wait times dramatically from minutes to seconds.

And I've found Gemma 4 with a good character card to be quite pleasant to interact with. Gemma 4 26B A3B has replaced gpt-oss-120B as my daily driver on my Mac, with roughly equivalent quality and the benefit of vision. It's just as fast as 120B at full precision (they are both roughly the same size, since GPT-OSS-120B is quantized with MXFP4), and Gemma 4 notably lacks the nanny, policy-oriented attitude GPT-OSS displays.

Who this is for

I assume you're running Gemma 4 (the 26B MoE or the 31B dense; I did not extensively test E2B/E4B, though they should work but with smaller context) on an Apple Silicon Mac via oMLX, pointed at SillyTavern in Chat Completion mode. Your roleplay turns take minutes before the first token begins streaming once the conversation goes on for a while, and you're wondering why cloud APIs feel instant but your M4 Max doesn't. Sure, prefill times are a well-known Mac Achilles’ Heel due to being mostly a compute-bound operation, but the answer isn't just your hardware. It’s also your prompt construction.

I won’t go into installation of oMLX here. But if you use a Mac for SillyTavern with a decent amount of RAM (>24GB), you should try it out: https://omlx.ai . And make sure you strictly follow Google's guidelines for sampling parameters; I used "force sampling" in oMLX to ensure SillyTavern's defaults didn't affect the model:

• temperature = 1.0
• top_p = 0.95
• top_k = 64

What's happening

oMLX caches the computational work (KV cache) from your prompt in fixed 1024-token blocks. On your next turn, if the prompt starts with the same tokens, oMLX reloads that cached work from SSD or filesystem cache in RAM (if you have lots of RAM) instead of recomputing it. This is called prefix caching. LMstudio also claims to support this, but so far I’ve had terrible luck with prefix caching working reliably using MLX (though it works OK with llama.cpp).

The problem: SillyTavern's defaults inject different content near the beginning of your prompt on every turn. Once one byte changes, every block after it is a cache miss. Your 80,000-token prompt gets fully recomputed from scratch because 50 tokens shifted at position 91.

One of the worst offenders is keyword-triggered World Info / lorebook entries placed at the default position (↑Char — Before Character Definitions). Different keywords fire on different turns, the injected content changes, and the entire prompt downstream is invalidated.

Some fixes:

1. Move your lorebook entries to u/D ⚙️ depth 0 (biggest single win)

This is the one that mattered most in my testing. For every lorebook attached to your character:

1. Top toolbar → World Info (globe icon) → pick the lorebook in the dropdown.
2. For each entry, change Position from ↑Char to u/D ⚙️ (at Depth, System role).
3. Set the depth field to 0.

If you're running Seraphina (a stock character), she ships with a lorebook called Eldoria with four keyword-triggered entries. Pretty easy to edit such a small lorebook. Some others I’ve checked out include dozens or hundreds of entries and are a bit more challenging. 

Why depth 0? It injects the content right before the assistant's reply. It’s past the point where the prompt naturally changes between turns (your latest message). So the churn is confined to the very tail of the prompt instead of poisoning the whole preceding cache.

2. Set Example Messages Behavior to "Always include examples"

User Settings (gear icon) → find "Example Messages Behavior" → change from "Gradual push-out" to "Always include examples".

The default gradually trims example messages as context fills, which shifts every token below them. Even if your current character has no examples, the next one will, and this'll bite you. This one was subtle, but I had to dig deep into context to understand what was happening. The idea of this implementation, I think, makes sense when dealing with tiny 4k or 8k context sizes from circa 2022/2023: reduce overall context length by eliminating examples once the character has produced tokens following the example pattern. But this kind of subtle “trimming of history” busts the cache of a modern turn-taking instruction-tuned model.

3. Move Author's Note off "In-chat @ Depth"

If you use Author's Notes: Extensions → Author's Note → change Position from "In-chat @ Depth" to "Before Main Prompt / Story String".

The "@ Depth N" anchor floats backward one message every turn, shifting the note's position in the prompt and invalidating everything after it. If you don't use Author's Notes, set Insertion Frequency to 0. Once again, the goal here is just to keep as much of preceding turns identical as possible to maximize cache hit ratio.

4. Unlock and/or raise your max context

AI Response Configuration (leftmost slider icon) → tick "Unlocked Context Size" → drag "Context Size" up to your model's limit (262,144 for Gemma 4 26B/31B works fine; 128K for E4B/E2B).

When context is capped low, SillyTavern drops your oldest messages to make room. This shifts every remaining token and busts the entire cache, resulting in massive prefill times at large contexts. I didn't notice any benefit setting it larger than 262,144 for Gemma 4 26B/31B or 131,072 for Gemma 4 E2B/E4B.

I will emphasize that I barely tested E2B and E4B models. They are truly *amazing* for their size, and if you wanna try out local inference but lack much RAM on your Mac, they'll fit. Their tool usage is quite good. But they aren't remotely the storytellers their larger siblings are.

5. Check Character Notes (the hidden cache-buster)

Click a character's avatar → Advanced Definitions → scroll to Character Note (a.k.a. Depth Prompt). If there's text there with depth set to 2 or 4, change it to depth 0, role System. Or clear it if you don't use it.

I audited 21 community cards and found 5 with depth=4 — a historical default from the 4k-context era that aged terribly on modern models.  This setting alone can add many seconds of prefill time to each prompt after the first few prompts.

6. Kill dynamic macros

Any macro that changes every turn — such as {{time}}, {{date}}, {{random:…}}, {{roll:…}} — anywhere in your character card, persona description, or main prompt will bust the cache if used badly. Search for them in:

• Character card (Description, Personality, Scenario, System Prompt Override)
• Persona Management → active persona description
• AI Response Configuration → prompt list (Main Prompt, NSFW, Jailbreak, Utility)

If you need a live clock in your RP, move it into Author's Note at @ Depth 1 as user role — the damage is confined to the last couple of blocks instead of the whole prompt.

Bonus: Turn off the Summarize extension on 128k+ models

Extensions → Summarize → set Source to "None".

Summarization busts the cache two ways: the summary's position drifts every turn, and the summary text itself changes when it regenerates. On a 256k context you can hold hundreds of turns of full roleplay turns as raw history. Nothing needs compressing. 

One caveat: summarize-off works well for basic narrative and voice work (RP, creative writing, storytelling). But if you need the model to recall specific proper nouns, IDs, or hard facts from 150+ turns ago, raw context alone may silently confabulate. The model might confidently give you wrong names that sound right, for instance. For that kind of recall, pin your most-important facts in a small block at u/D ⚙️ depth 0 or use Vector Storage. More on this below.

Server-side: oMLX hot cache tuning

oMLX features a “hot_cache_max_size” setting (the in-RAM KV block pool). I had some counter-intuitive performance findings when benchmarking. Rather than include it all here, the TL;DR: leave oMLX hot cache turned off for most setups. It competes with system RAM and filesystem cache. If you do set it, make sure it’s large enough to hold your entire KV cache. A small “hot cache” is the worst of both worlds.

What you get

Measured on a 310-turn SFW Seraphina roleplay, Gemma 4 26B-A4B (full precision, no quantization) on an M4 Max 128GB:

• Before fixes: 0% cache reuse. Every turn fully recomputed. Multi-minute waits at long context.
• After fixes: 95–99% cache reuse in steady state. Only ~1024 tokens of new prefill per turn regardless of how long the chat is.
• Wall time at 190k per-turn prompts: ~25–30 seconds for a 2000-character reply, most of that simply waiting on streaming to complete at 49 tokens per second down to 30 tokens per second late in the conversation. Would have been several minutes without caching.
• Character voice after 310 turns: rock solid. No drift, no degradation. Mostly because it’s being injected every turn, really.

The cache works better at larger prompts; the fixed-size trailing block miss (1024 tokens) becomes a smaller fraction of the total, so hit rate actually climbs as your chat grows.

The coherence footnote (for the curious)

During the stress test I planted specific facts early in the chat and tested whether the model could retrieve them later without being reminded:

• At 114 turns / 75k tokens of context distance: the model recalled every specific physical descriptor of a brass key — shape, number of teeth, material, wear pattern. Clean retrieval. ✅
• At 190 turns / 180k tokens of context distance: asked for four family names planted earlier. The model confidently produced four completely wrong names that sounded plausible for the genre. No hedge, no uncertainty, declined an explicit "say so if you can't remember" off-ramp. ❌

The failure mode was confident confabulation. The model finds the right shape of the answer (four fantasy surnames) but loses the specific tokens and falls back on what its training data says fantasy surnames should sound like.

This matters for your RP: character voice and narrative tone will stay solid for hundreds of turns because the lorebook injection at depth 0 re-anchors it every turn. But specific plot facts — character names, locations, items, promises made 150+ turns ago — will drift into plausible-sounding wrong answers. If that matters to your story, pin your important facts in a small stable block at u/D ⚙️ depth 0 that you update manually as the plot develops. Think of it as the GM's cheat sheet.

The historical confabulation smells like a normal LLM artifact -- I work with them all day, every day for a living anyway -- but it also might be an artifact of turboquant in oMLX. The aliasing inherent to KV cache quantization of any sort, as well as RoPE and YARN, make it very difficult for the model to accurately predict correct choices among many similar-looking vectors.

One-liner version

Open World Info, move everything to u/D ⚙️ depth 0. Check your character cards for depth-4 Character Notes. Turn off Summarize. Set oMLX's hot_cache_max_size to 0 or ≥ your working KV size. Enjoy your 95%+ cache hit rate and responses that feel like they start generating instantly.

Happy roleplaying.

(Yep, I used/abused Claude Code to help me work through the options that were busting cache in SillyTavern. This compressed days of troubleshooting into a few hours of testing last night, curled up on my recliner with my MacBook pro M4 Max 128GB playing the role of personal space heater.)