[Research] I've been working on an attention mechanism that keeps KV cache at ~1.5GB regardless of context length — update post

DWARF uses a fixed circular buffer — about 1.5GB, always, regardless of context length. The tradeoff is you don't get full attention over the whole context, but the physics-derived offset set recovers most of what matters.

Core result: a fixed ~1.5GB KV cache at any context length (versus ~52GB for a standard 7B at 100K tokens), achieved by computing attention at 44 physics-derived dyadic offsets rather than all past positions.

DWARF models outperform standard Transformers in several metrics, including reduced training cost.

Code has been public for two weeks with 500+ clones.

Paper is written and LaTeX-compiled, available upon request. Trying to submit to arXiv cs.LG and need an endorsement

(Please DM if you are able and willing to help.)

GitHub: https://github.com/Lanerra/DWARF