Building the GPT Tokenizer from Scratch
Andrej Karpathy · Lecture · YouTube title: Let's build the GPT Tokenizer
Why it's here
Recommended as the best deep dive on tokenizers, worth the two hours for anyone working with LLMs.
What it covers
Andrej Karpathy builds the GPT tokenizer from scratch, implementing byte pair encoding along with encode and decode. He shows how many odd LLM behaviors trace back to tokenization, and covers Unicode byte encodings along the way.