Week 2: Building a Small Language Model from Scratch(Positional Embeddings, RoPE, and Model Distillation) - June 30 - July 4

https://preview.redd.it/h0h0kuq6fz9f1.png?width=922&format=png&auto=webp&s=adff81fdbb4ff3138a504a84b18e4440a6185b98

Hi everyone,

I’m currently working on a hands-on series where I’m building a small language model from scratch. Last week was all about tokenization, embedding layers, and transformer fundamentals. This week, I’m shifting focus to something crucial but often overlooked: how transformers understand order.

Here’s the breakdown for June 30 – July 4:

• June 30 – What are Positional Embeddings and why do they matter
• July 1 – Coding sinusoidal positional embeddings from scratch
• July 2 – A deep dive into Rotary Positional Embeddings (RoPE) and how DeepSeek uses them
• July 3 – Implementing RoPE in code and testing it on token sequences
• July 4 – Bonus: Intro to model distillation, compressing large models into smaller, faster ones

Each day, I’ll be sharing learnings, visuals, and code walkthroughs. The goal is to understand the concepts and implement them in practice.

If you'd like to follow along more closely, I’m posting regular updates on LinkedIn. Feel free to connect with me there https://www.linkedin.com/in/prashant-lakhera-696119b/

Would love to hear your thoughts, questions, or suggestions.