Advanced Claude Code Workflow: Building 'Densely' for Lossless Context Compression and Tokenizer Optimization with CLAUDE.md and Hooks

Workflow value: 95/100
Status: active · Freshness: 70/100 · Confidence: 1.00 · Level: advanced
Categories: Quality Control, Token Saving, Context & Memory, Debugging, Shipping, CLAUDE.md, Hooks, MCP
Original source: r/ClaudeCode post/comment

What problem this solves

Overcoming Claude's context window limitations and high token costs by losslessly compressing large tool outputs before they consume agent history, and accurately understanding Claude's tokenizer behavior for efficient token usage.

Summary

This workflow details how to build and integrate a lossless context compression tool ('densely') for Claude agents. It emphasizes using a structured CLAUDE.md with hard behavioral rules, setting verifiable goals for Claude Code, and leveraging PostToolUse hooks to automatically compress large tool outputs. It also includes a method for reverse-engineering Claude's tokenizer using the count_tokens endpoint to optimize compression efficiency.

Why it is useful

This workflow is highly valuable because it provides a concrete, open-source solution ('densely') to a critical problem in LLM development: context window limitations and high token costs. It demonstrates a robust and repeatable workflow for using Claude Code itself, emphasizing structured prompting with CLAUDE.md and verifiable goals, which are highly reusable principles for any complex Claude Code project. Furthermore, it offers novel technical insights, such as leveraging PostToolUse hooks for proactive context management and a method for reverse-engineering Claude's tokenizer using the count_tokens endpoint to achieve optimal compression. The detailed validation with real numbers and byte-exact verification adds significant credibility, making the tool and the underlying principles highly transferable and beneficial for advanced Claude Code users and agent developers.

Workflow

1. Define hard behavioral rules in CLAUDE.md (e.g., state assumptions, surgical diffs, verifiable goals) for Claude Code development.
2. Phrase development tasks as verifiable goals (e.g., 'compress+decompress must round-trip byte-exact on these 5 real files').
3. Iterate with Claude Code, using the verifiable goals as acceptance criteria, until the checks pass.
4. Implement a PostToolUse hook to intercept large tool outputs before they are added to the agent's history.
5. Set up an MCP server to host the compression/decompression logic.
6. Integrate the compression tool (e.g., 'densely') with the PostToolUse hook to automatically compress tool outputs.
7. Optionally, reverse-engineer Claude's tokenizer using the count_tokens API endpoint to identify optimal single-token words for custom compression alphabets (e.g., batch exact-match plus bisection).
8. Measure and verify token savings and byte-exact round-trip integrity using real-world data and Claude's own count_tokens counter.

Tools / artifacts

• densely (context compression tool)
• CLAUDE.md file
• PostToolUse hook
• MCP server
• count_tokens API endpoint
• GitHub repository (alibaizhanov/densely)
• Blog post (densely.dev/blog/bpe-slack)
• Real-world files for testing (e.g., macOS install.log, npm registry JSON, package-lock.json, source code)

Validation signals

• Real numbers measured on Sonnet 5's own counter
• Byte-exact round trip verified per row (sha256-verified)
• Specific token savings reported (93.5% for install.log, 82.7% for npm JSON, 62.5% for package-lock.json)
• 131k tokens saved in two days of normal use
• Claude Code loops until verifiable checks pass
• Public GitHub repository and detailed blog post for transparency and verification
• Initial calibration issue caught by a commenter and subsequently addressed, leading to improved accuracy

Limitations

• The technical depth of the post might be overwhelming for beginner users.
• The initial Reddit score and comment count are low, likely due to the post's recency.

Rate this workflow

Upvote this post if the workflow is useful, reproducible, or worth recommending.

Downvote if it is vague, outdated, unsafe, overhyped, or not reproducible.

Reply if it worked for you, failed, is outdated, or has a better alternative.

---

This post was generated automatically from the workflow library database.