Position-Agnostic Pre-Projection with Content Skip for Transformer Attention — nonlinear feature construction before Q/K/V, with learned attention bypass

I'm sharing a paper on a simple modification to transformer attention blocks, motivated by a limitation in how position interacts with content.

**The motivation:** RoPE and other positional encodings are applied to Q/K vectors that already contain content information — position and content are entangled. For text, this works well enough. But for multimodal architectures, it's a problem: text is sequential, images are 2D spatial, video is spatiotemporal, and point clouds have no natural ordering. A shared attention layer has to force all modalities through the same positional scheme, or use separate projections per modality. What if we could separate content preparation (modality-independent) from positional awareness (modality-specific)?

**The idea:** Insert a small nonlinear MLP before Q/K/V projections that operates in a purely content-focused, position-agnostic space — before RoPE is applied. Additionally, a content skip connection lets these enriched features bypass attention entirely where the model learns it's beneficial. The pre-projection does double duty: enriching Q/K/V inputs AND providing a content-only residual that bypasses positional attention.

**Key results (frozen probe on Pythia, base model frozen):**

| Method | Params | LAMBADA | PPL |

|--------|--------|---------|-----|

| Baseline (160M) | — | 0.128 | 78.4 |

| LoRA (r=480) | 26.5M | 0.126 | 63.8 |

| Pre-proj + skip | 24.8M | **0.180** | **47.9** |

| Baseline (410M) | — | 0.466 | 29.2 |

| LoRA (r=640) | 94.4M | 0.488 | 17.7 |

| Pre-proj + skip | 88.1M | 0.484 | **17.0** |

**Most interesting finding:** The content skip learns to activate more strongly in later layers — consistent across both model sizes. Early layers barely use it, final layers use it heavily. The model discovers on its own where content should bypass positional attention. This aligns with prior work showing early layers handle syntax (position-dependent) while later layers handle semantics (content-dependent).

**Why it matters:**

- Nonlinear feature construction gives attention access to features that no linear projection can extract

- Position-agnostic design cleanly separates content preparation from positional encoding — each modality can apply its own positional scheme at Q/K while sharing the pre-projection

- The content skip is especially relevant for modalities where position is undefined or unhelpful

- No K/V cache overhead

All experiments on a single RTX 4080 12GB.

Paper + code: https://github.com/cs-cmyk/preprojection

Happy to answer questions. Also looking for an arxiv endorsement in cs.CL or cs.LG if anyone is willing.