I fine-tuned Qwen 14B to beat GPT-4o on NYT Connections (30% vs 22.7%)

I spent a weekend fine-tuning Qwen 2.5 14B to solve NYT Connections puzzles. Results:

What worked: Distillation. I had Sonnet solve ~350 puzzles while explaining its reasoning step-by-step, then fine-tuned Qwen on those traces. The model learned to think about the puzzle, not just output answers.

What didn't work:

• Fine-tuning on just puzzle solutions (learned format, not reasoning)
• Synthetic puzzle generation (Sonnet kept making trivial puzzles)
• Embedding similarity scoring (word associations aren't semantic)

Setup:

• QLoRA with Unsloth
• LoRA rank 32, 2.5 epochs
• ~20 min training on A100
• Total cost: ~$10

Full writeup with code: https://open.substack.com/pub/john463212/p/teaching-a-14b-oss-model-to-beat

Happy to answer questions about the approach!