Spent 6 months building agent stacks. The hardest part isnt the agents, its the context layer between them

Im going to skip the throat clearing.

I lead a small team building vertical agents in legal tech. weve built five so far, two in production, one that almost shipped and got killed by enterprise procurement. the longer i do this, the more convinced i am that the conversation in this sub is mostly focused on the wrong layer.

People argue endlessly about react vs reflexion vs whatever the new orchestration paper this week is. fine. those matter at the margin. but the actual production failure mode in every system weve shipped is not the agent reasoning. its that agent A doesnt know what agent B did 20 minutes ago, and the user has to manually paste context between them. or worse, the user gives up and goes back to chatgpt because at least that has memory now.

Context fragmentation is the real bottleneck.

I think this happens because most of us came up training models, not designing operating systems. we treat memory as a vector store you bolt on the side. but in production what you actually need is something closer to a shared context bus that every agent can read from and write to, scoped per user or per project, with provenance. nobody has shipped a clean version of this yet inside a coherent product. its all bespoke per deployment.

The cut that matters in practice is not "do you have memory" but "how does the context actually get into the system in the first place". four broad paths the field is betting on right now, each with very different tradeoffs:

1. Chat-driven memory. ChatGPT memory rollout, Claude Projects, Cursor's per-project memories. the system learns from whats said inside the chat surface itself. cleanest signal because the user is literally typing their intent. but its scoped to one app and only covers what they remembered to say. everything that happened in slack, in a doc, in a meeting outside that surface, is invisible to it.
2. Schema-driven connectors. MCP servers, OpenAPI integrations, the connector ecosystem (zapier, paragon, etc). agent pulls structured context from gdrive/notion/linear on demand. coverage is wide on paper, in practice it covers whatever the user took the trouble to connect, and its still pull-based, the agent has to know what to ask for. MCP is moving the spec in the right direction but the memory ergonomics arent there yet.
3. OS-level observation. AirJelly on macos, screenpipe in the OSS lane, what limitless was doing on the pendant side before meta bought them in december, what apple keeps gesturing at across WWDC keynotes but hasnt put into siri at any usable depth. always-on capture at the screen/audio layer, local OCR + embedding, the system gets a continuous timeline of what the user actually did instead of what they remembered to log. noisiest signal of the four but the only one that captures events that never made it into any app. closest to ground truth, hardest to do well.
4. Curated knowledge index. Notion AI, mem.ai, obsidian + a rag plugin. retrieval over notes the user already wrote down. signal quality is high because the user already filtered, but its lagging and partial. you only see what got into the vault, which is a small fraction of what actually happened.

If im honest, the path im rooting for from a backend-agent-builder perspective is #3, and its not because i love always-on capture on my desktop. the privacy and battery tradeoffs are real, the products on this path are still rough at the edges, and most of them are pitched at the wrong audience right now (productivity end users) not the right one for our problem (agent infra). but my agents dont need the user's curated notes. they need to know "what was the user actually doing at 2pm tuesday when they pinged me about contract X". paths 1, 2 and 4 all require the user (or some upstream system) to have already created the artifact. path 3 doesnt. for a set of agents thats supposed to feel coherent across a workday, having a single per-user timeline that every agent can read from changes the shape of whats possible. the products on this path are early and consumer-facing today, but the architecture is the one id want to build my own context bus against, not the connector-graph one were all defaulting to.

MemGPT got attention for the sliding window stuff but the deeper insight buried in that paper, that memory has to be hierarchical, hasnt been picked up enough by application teams. whichever path wins, the layering question still has to be solved on top of it.

The team that figures out the right primitive for cross agent context will win this. its not going to be the team with the cleverest agent loop. agent loops are commoditizing fast. context isnt.

Im going to keep building agents either way but my money is on context being the real moat for the next 18 months.