Build GPT from Scratch with PyTorch

Andrej Karpathy · Tutorial · YouTube title: Let's build GPT: from scratch, in code, spelled out.

Why it's here

It is the go-to recommendation for learning how transformers and LLMs actually work, praised for its clarity and Karpathy's teaching.

What it covers

Andrej Karpathy builds a GPT from scratch in code, following Attention Is All You Need and GPT-2/3, in about two hours using PyTorch. He connects it to ChatGPT and assumes familiarity with his earlier makemore videos.