This wiki covers Andrej Karpathy's published teaching corpus on language models — seven open-source repositories and a nine-lecture YouTube series ("Neural Networks: Zero to Hero"). Together they trace the technical lineage from "what is backpropagation" through to "here is a working reproduction of GPT-2 (124M)."

The corpus is unusually coherent. The same patterns and abstractions recur across repos — Block, MultiHeadAttention, configure_optimizers, estimate_mfu, from_pretrained — at progressively bigger scales. Reading any one repo in isolation works, but reading them in order shows you the underlying ideas being refined.

Reading guide

If you're starting from zero and want the full arc, the order is:

1. zero-to-hero-arc The lecture map. Read this first.
2. repos/micrograd Scalar autograd. The conceptual root.
3. backpropagation and value-class The algorithm and its data structure.
4. repos/makemore First real LMs. Bigram → MLP → ... → Transformer.
5. repos/ng-video-lecture Character-level GPT on Tiny Shakespeare.
6. repos/nanoGPT Production-grade GPT-2 implementation.
7. repos/build-nanogpt Faithful GPT-2 reproduction with every optimization.
8. repos/llama2-c Llama 2 in PyTorch + pure C inference. The "modern" architecture.
9. repos/llm-c Same training task as build-nanogpt, in pure C/CUDA.

If you want to learn a specific concept, jump to the concept page; each one cross-references the repos that demonstrate it.

The architecture, in pieces

The transformer architecture as Karpathy teaches it, broken into independent pieces:

Training, in pieces

Inference

Three "model families" to compare

The corpus contains three subtly different transformer architectures, useful to compare against each other:

Same skeleton, different organs. Once you know the skeleton (the transformer block wrapped in residuals and a stack), swapping organs is straightforward.

What's not in this wiki

Things outside the scope of the corpus:

In scope: dense, decoder-only, pretraining + base inference, up to GPT-2 / Llama 2 scale. Within that scope it's the most complete teaching resource available.

Cross-reference conventions

Every page in this wiki uses markdown reference links: [name](name.md) for concepts, [name](repos/name.md) for repos. The link text is usually the unqualified name; the path tells you whether it's a concept or a repo page.

For agents post-processing this wiki: every page is a self-contained topic that can be rendered as a single HTML page. Internal links between pages are the primary structural signal of the wiki graph. The concepts/ flat layout was rejected in favor of having concepts at the wiki root and repos in a subdirectory — concepts are first-class citizens, repos are case studies that ground them.