Retrofitting latent chain-of-thought onto Qwen2.5-3B — it learns a well-formed thought trajectory, but not a question-specific one

Fine-tuned Qwen2.5-3B on a Coconut-style objective: the hidden state is recycled into inputs_embeds for K steps before any token is emitted, with a PonderNet halt head choosing K.

What came out, measured against the pre-objective checkpoint:

• Best depth moved 2 → 4 steps; benefit of thinking went −1.11 → −3.56 nats
• Shuffling the thought steps costs +1.39 nats and hurts 12/12 held-out prompts
• But transplanting a different problem's latents costs only +0.055 nats, and hurts just 8/12

So the trajectory is load-bearing and order-sensitive, yet largely generic — it has learned the shape of an answer without the content of the question. Unlike Huginn and Ouro this isn't pretrained looped, it's a retrofit onto an existing checkpoint, which may be exactly why.

Weights and full writeup, including what failed: https://huggingface.co/Cymela/hyper-3b-latent
Also visit cymela.com