I've been working on a fundamentally different LLM architecture. No attention layers. No FFN blocks. Instead, every token lives in complex phase space, and language processing happens through wave-like interference between specialized "phase banks."

Open-sourced here: https://github.com/gowrav-vishwakarma/qllm2

The core idea: language as wave interference

In a transformer, a token is a real-valued vector that gets refined through attention + FFN layers. In this model, a token is a complex number -- it has a magnitude (how "important/activated" it is) and a phase angle (what "kind of meaning" it carries). These two properties are naturally separated and jointly processed.

This isn't just a gimmick. It changes how every operation works:

• Embeddings: Each token gets a [real, imag] vector. The model learns that semantically similar tokens align in phase, while different meanings sit at different angles.
• Transformations are rotations: When context modifies a token's meaning (like "bank" shifting meaning based on surrounding words), that's a phase rotation -- a complex multiply. Rotations compose naturally, are always invertible (no information loss), and reduce to GEMM.
• Similarity is coherence: Instead of dot product, we use phase coherence: Re(a * conj(b)) / (|a| * |b|). This measures both directional alignment AND magnitude relationship.
• Multiple banks interfere: A "semantic bank" and "context bank" process each token independently, then combine via learned interference (constructive where they agree, destructive where they conflict). A tiny router decides per-token how much weight each bank gets. Think MoE but at the representation level.

What the phase system actually gives you

After reading through the code and training results, here's what the phase representation concretely provides:

1. Natural magnitude/phase decomposition = implicit attention High-magnitude phase states dominate downstream processing automatically. The model doesn't need explicit attention to decide "which tokens matter" -- magnitude handles salience, phase handles identity. The SemanticPhaseBank uses 512 learnable concept vectors and retrieves them via phase coherence -- this is essentially a learned associative lookup that runs in O(seq concepts), not O(seq^(2).)

2. Context as phase modulation The ContextPhaseBank computes a causal windowed average (window=8) of nearby tokens and then complex-multiplies it with the current token. This is elegant: the local context literally rotates the token's meaning in phase space. A word appearing after "not" gets rotated differently than after "very." No attention needed.

3. Rotation-based state evolution The backbone SSM evolves state via: h[t+1] = damping * R(theta) @ h[t] + gate * B @ x[t] where R(theta) is a Cayley-transform rotation. The state naturally oscillates, and the damping factor (learned, per-dimension, range [0.5, 1.0]) controls how fast old information decays. This is why SSMs struggle with long-range recall -- but the model compensates with a separate Phase-Coded Memory (1024 learned slots, chunked top-k retrieval) and an Episodic Memory (sliding window via FlashAttention SDPA).

4. Zero trig in the hot path Every rotation uses the Cayley transform: cos_like = (1-a^2)/(1+a^2), sin_like = 2a/(1+a^2). This is just arithmetic -- no sin(), no cos(), no exp(). Every operation is a matmul or elementwise op. Perfect for Tensor Cores.

Results (178M params, TinyStories, 10k samples, A6000)

Metric	Epoch 1	Epoch 2	Epoch 3 (partial)
Train PPL	200.86	32.75	~26 (and dropping)
Val PPL	76.47	48.92	--
Train CE	5.30	3.49	~3.26

Training used only 10k samples (0.5% of TinyStories). Starting PPL was 55,000 (random). It dropped to val PPL 49 in 2 epochs (40 min on A6000, no compile). Overfiting simply needs data now ...

Epoch 1 generation:

"The quick brown house. They run and start to get a smile. Mom were very excited. Now mommy and big yellow room. There said and She are friends. Tim, she started to save the garden."

For context: A 22M-param GPT-2 trained on the full 2.1M TinyStories dataset for 20k steps reaches val PPL ~11. We're at 49 with 0.5% of the data and 2 epochs. The learning curve is steep and still dropping -- we just need more data/epochs to converge.

Why this approach might be better

• O(n) complexity: Linear-time backbone. Theoretical 256K context. No quadratic attention.
• GEMM-only math: No trig, no softmax in the backbone. Everything is matmul/elementwise.
• Interpretable: You can inspect which bank each token routes through, what concepts are retrieved from memory, how coherent the phase states are. The model ships with "philosophy metrics" (Manas/Buddhi/Viveka/Smriti from Indian philosophy) that track mind activity, discernment, stability, and memory quality.
• Modular: Banks, backbone, coupler, memory, and objectives are all registered components. Add a new bank type with a decorator. Swap the backbone. Change the coupling strategy. All via config.
• Consumer-GPU friendly: Medium model trains on RTX 4090 / A6000 with batch 48-64.

Honest limitations

• Training throughput is ~2x slower than an equivalent transformer. The SSM backbone loop is sequential per-step. A custom Triton kernel would help but doesn't exist yet.
• In-context learning will be weaker. Fixed-state SSMs compress context into a fixed vector. The episodic memory (O(n buffer_size) sliding window) helps with copying but isn't a full replacement for O(n^(2)) attention.
• Not validated at scale. 178M params on 10k samples is a PoC. Need full dataset + larger models + benchmarks.
• Bank ablations not done. We use semantic + context banks but haven't proven both are needed. Could be that one bank suffices.
• Pure PyTorch. No fused CUDA/Triton kernels. Backbone loop is Python. Lots of low-hanging performance fruit.

What's next

• Full TinyStories training (2.1M samples) for proper PPL comparison
• Bank ablations (semantic-only vs semantic+context vs 4-bank)
• Triton kernel for the oscillatory SSM recurrence
• Scale to 1B+ params
• Long-context evaluation (4K / 16K / 64K tokens)

Tech stack

PyTorch | torch.compile compatible | GPT-2 BPE tokenizer | uv package management | Clean modular codebase

Looking for feedback, collaborators, and people who want to try architectures beyond transformers.