I recently wrote a blog on implementations of the various algorithms used in Tokenization in Llms.

Hey everyone,

I've been getting into systems development and low level development a lot lately, so I've been writing a lot of raw cpp dealing with memory management and gguf parsing.

I recently wrote about implementations of a couple tokenization algorithms.

Link to blog

Give it a read if you're interested. Thanks :D