I run a personal home-assistant agent with persistent memory and a library of "skills" — markdown files it loads on demand when doing specific kinds of work (server admin, scheduled-job maintenance, recipe digitization, network monitoring, etc.). Each skill has a main file plus optional supporting documents. When the agent decides a task matches a skill, it reads the main file in its entirety into context. There's no caching between sessions: every time it consults that skill, it pays the full token cost again.

Sounds fine until you have a few dozen skills and some of them grew organically over months of accumulated incident notes. Mine were 27–48KB each (~7–12k tokens). A routine session touching two or three of those was burning most of my context budget before any actual work started — which also meant more frequent compaction, more lost nuance, and (on local models) slower turns.

This affects both those who use frontier models and those with local LLM systems, either in cost or lost time as the local LLM loops through compression more frequently (and unnecessarily.)

The fix: lean core + on-demand references

The pattern is simple:

Keep in the main body (target: ≤ ~10K characters):

• What the skill is for / trigger conditions
• Environment facts you need cold (hostnames, ports, credential locations, versions)
• Safety rules and high-frequency pitfalls — the stuff that causes real damage if missing
• A domain index: a small table mapping "if you're doing X → read references/x.md"

Move to reference files:

• Command tables with worked examples
• Incident histories and case studies ("here's what happened last Tuesday")
• API listings, edge-case catalogs, long troubleshooting trees
• Anything you'd only want in one specific sub-task

Reference files load explicitly, only when needed. For me this dropped six fat skill bodies from ~56k tokens of recurring cost to ~10k — an 80% reduction — while keeping every fact reachable. Nothing was deleted; old material went into archive/ folders inside each skill so it survives but never loads by accident.

Worked example: my scheduled-job audit procedure

One of my skills documented how to audit scheduled background jobs (think cron-style automation for the agent). The original body was ~48KB because two months of actual audits had been appended to it as lessons learned. Here's how I split it:

The new main body (~10K chars) keeps:

• The audit contract: agree scope before starting big reviews; treat the scheduler's job metadata as current state, not canonical truth (the real prompts may live elsewhere); check three evidence layers — current metadata, historical run output, and the platform's actual capabilities
• The report format: confirmed defects vs likely defects vs user policy decisions, with no speculation presented as fact
• A stop-condition list for when a human reviewer is involved
• An index table pointing at four reference files

The references got:

• monitor-script-pattern.md — full pattern for change-detector scripts that gate expensive LLM runs on cheap hash comparison
• safe-withdraw-pattern.md — the worked "how do I safely pause a job mid-cycle" walkthrough from a real incident
• reviewer-stop-conditions.md — the detailed rules about what an advisory reviewer can and cannot override
• prompt-corruption-case-files/ — archived transcripts of two times a long prompt got truncated by the runner and what we did about it

Judgment call that mattered most: which pitfalls stay in core. I kept anything that had actually caused damage or confusion (e.g., "job metadata says X but the last 20 runs show Y — always read the runs") and moved everything else to references. The test I used: if the agent does this task type without reading any reference file, will it get hurt? If yes, keep it in core.

Things that bit me

1. Stale cross-references. Skills reference each other ("see the network monitoring skill's access patterns"). After restructuring, those pointers either dangle or drag in a whole sibling body. Fix: point at the specific reference file, not the parent skill. I now run a regex check over every rewritten body to confirm all references/*.md pointers resolve to real files before calling a restructure done.
2. Backups are non-negotiable. Every rewrite gets a .bak-<date> next to the original. My skills encode hard-won operational knowledge; one bad compression pass shouldn't be able to destroy it.
3. Archive, don't delete. Incident notes have value even when they're no longer active procedure. An archive/ subfolder keeps them out of the load path without losing them.
4. Measure first. I ranked my skills by (load count × size) from usage telemetry before touching anything. Three "medium" skills I'd have guessed were fine turned out irrelevant; the actual offenders were obvious once you look. Don't trim what you rarely load.
5. This is complementary to tool-schema bloat. If your agent also exposes lots of MCP/plugin tools, the tool catalog inflates context too — separate problem, same remedy philosophy: whitelist aggressively, defer the rest behind search.

The underlying principle: treat your skill bodies like hot-path code and your references like cold storage. The main file should be the table of contents plus everything dangerous to miss. Everything else is one lookup away.