My agent kept "remembering" things wrong. The fix was embarrassingly simple

Six months ago I spent a weekend wiring up a vector store to give my coding agent persistent memory. Embeddings, retrieval pipeline, similarity thresholds, the whole stack. I got it working and it felt like progress.

Then I asked the agent about a design decision I'd documented three weeks earlier — why we'd chosen a particular auth approach. It cited the right document. But the answer was wrong. It had retrieved something that was topically similar to the auth decision but was actually about a different service with different constraints. The cosine similarity was fine. The answer was not.

I spent an hour trying to debug it. I couldn't just "open the memory and look." The knowledge was in embedding space. I could see what was retrieved after the fact, but I couldn't understand why or fix the underlying confusion without restructuring and re-indexing.

I switched to something dumber: I gave the agent direct CLI access to a folder of markdown files.

iwe retrieve -k decisions/auth-flow --depth 2

That command returns the auth-flow document with its linked child documents inlined — not based on semantic similarity, but based on the structure I had already built by organizing and linking the files. The agent gets exactly the subgraph I would navigate to if I were looking this up myself.

The retrieval failure went away. Not because structured retrieval is smarter than embeddings — it isn't, for all kinds of queries. But for architectural knowledge, the structure I'd already created by organizing notes was a much stronger signal than cosine distance.

The other thing I didn't expect: the agent started maintaining the structure itself. I give it a simple instruction in the system prompt — when you learn something durable, write it as a new linked document in the right place. Now when I ask about a decision I don't remember documenting, sometimes it's already there because the agent filed it correctly during a previous session. The knowledge base is shared. The agent and I are working from the same files.

The tradeoffs are real and worth saying plainly:

• You need to start with organized, linked notes for this to work at all. The retrieval is only as good as the structure you've built.
• Fuzzy or exploratory queries — "what did we discuss that's vaguely related to caching?" — are worse than embeddings. You have to know roughly what you're looking for.
• It requires ongoing maintenance of link structure. Links don't create themselves (unless you train the agent to help, which I now do).

But the debugging story changed completely. If the agent gets something wrong, I open the markdown file, fix it, and the agent gets the corrected version next call. No re-indexing. No wondering which embedding is stale. No black box. And just like with code, I regularly ask the agent to review the knowledge base and restructure content as needed — rename things, split documents that got too big, fix broken links. It's the same workflow I already use for code, just applied to knowledge.

Curious if others are experimenting with a similar approach, still prefer embeddings, or mix the two. Would love to hear what's working.