How are people handling AI agent memory in production with sensitive data?

To give some background, I currently live in the EU, the motherland of bureaucracy and red tape.

I’ve been looking at how different AI agent frameworks handle memory, and I think people are underestimating how quickly it becomes a security problem once agents start interacting with real data.

If an agent can read documents, summarise emails, browse websites, call tools, and modify files, then long-term memory slowly turns into a collection point for sensitive information.

And the problem is not only accidental retention.

A malicious instruction hidden inside a webpage, email, or document could potentially poison memory, extract previous context, or manipulate future behaviour. If the agent inherits broad user permissions, things can escalate fast.

What makes me uncomfortable is that many memory systems still feel optimised primarily for convenience: store more context, remember automatically, retrieve aggressively.

But where is that data actually living?
Is it encrypted?
Can it be audited or purged?
Can you verify what wrote it and when?

To me, AI memory feels less like a UX feature and more like infrastructure.

So I started building a small open-core Python CLI around a different approach:
local encrypted storage, explicit ingestion, and user-controlled long-term memory.

The design is intentionally boring.

Memory should have a lifecycle: import deliberately, encrypt locally, retrieve intentionally, and minimise what gets sent back to public models.

I’m not claiming this solves every AI security issue, but I do think safer defaults matter:
memory should not be invisible, automatic, and plaintext by default.

Curious how others are approaching this.

Am I overthinking the problem? Are people quietly building similar safeguards internally?