What You'll Build
A sampling loop that generates new names from the trained model.
Depends On
Chapter 11 (the trained model).
How Generation Works
After training, the parameters are frozen. We start with the BOS token, feed it through the model, get a probability distribution over next tokens, sample one, feed it back in, and repeat until the model produces BOS again ("I'm done") or we hit the maximum length. Same generation loop from the bigram chapter. Only the source of the probabilities has changed.
// --- FullTraining.cs (add below the training loop from Chapter 11) ---
const double Temperature = 0.5;
Console.WriteLine("\n--- inference (new, hallucinated names) ---");
for (int sampleIdx = 0; sampleIdx < 20; sampleIdx++)
{
List<List<Value>>[] keys = model.CreateKvCache();
List<List<Value>>[] values = model.CreateKvCache();
int tokenId = tokenizer.Bos;
var sample = new StringBuilder();
for (int posId = 0; posId < maxSequenceLength; posId++)
{
List<Value> logits = model.Forward(tokenId, posId, keys, values);
var scaledLogits = logits.Select(l => l / Temperature).ToList();
List<Value> probabilities = Softmax(scaledLogits);
double r = random.NextDouble();
double sum = 0;
int nextToken = -1;
var probabilityValues = probabilities.Select(p => p.Data).ToList();
// Softmax probabilities can sum to slightly less/more than 1 due to floating point.
// Rescale r into the actual total so we never fall off the end of the loop.
double totalProb = probabilityValues.Sum();
r *= totalProb;
for (int i = 0; i < probabilityValues.Count; i++)
{
sum += probabilityValues[i];
if (r <= sum)
{
nextToken = i;
break;
}
}
if (nextToken == -1)
{
nextToken = probabilityValues.Count - 1;
}
tokenId = nextToken;
if (tokenId == tokenizer.Bos)
{
break;
}
sample.Append(tokenizer.Decode(tokenId));
}
Console.WriteLine($"sample {sampleIdx + 1, 2}: {sample}");
}
Notice how model.CreateKvCache() replaces the manual array-initialisation loop we would have needed. The model knows how many layers it has; the caller doesn't need to.
Temperature
Temperature controls the "creativity" of generation. Before softmax, we divide each logit by the temperature:
- Temperature = 1.0. Sample directly from the model's learned distribution. Normal randomness.
- Temperature < 1.0 (e.g. 0.5). Sharpens the distribution. The model becomes more conservative, more likely to pick its top choices. Names will be more "typical".
- Temperature -> 0. Always picks the single most likely token (greedy decoding). No randomness at all.
- Temperature > 1.0. Flattens the distribution. More diverse but potentially less coherent output.
What You Should See
After training for 10,000 steps, the model generates plausible-sounding names like:
sample 1: kamon
sample 2: ann
sample 3: karai
sample 4: jaire
sample 5: vialan
These names don't exist in the training data. The model has learned the statistical patterns of names (consonant-vowel patterns, common endings, typical lengths) and is generating new examples from that learned distribution.
The Connection to ChatGPT
From the model's perspective, your conversation with ChatGPT is just a funny-looking "document". When you type your prompt, you're initialising the document. The model's response is a statistical document completion: the same next-token prediction we've built here, just at enormously larger scale with post-training layered on top to make it conversational.
Running the Final Model
dotnet run -c Release -- full
(Or just dotnet run -c Release. The dispatcher defaults to full when no chapter is given.)
The full training run (10,000 steps) typically takes 5-15 minutes on a modern CPU in Release mode, and much longer in Debug mode. Always use -c Release for training. The per-step loss bounces around, but watch the avg column. That's the running average, and it should trend downward from ~3.3 toward ~2.37. Every 1000 steps, a [milestone] line prints the current avg alongside its value at the previous milestone. The running average is smooth but not monotonic, so expect the occasional milestone-to-milestone wobble even while the overall trend is down. After training, you'll see 20 generated names.
What to Try Next
Now that you have a working model, here are some experiments worth running. Each one isolates a single variable so you can see its effect clearly.
Add a second transformer block. Change
layerCountfrom 1 to 2 inFullTraining.cs. The parameter count roughly doubles, but watch the loss. A second block lets the model refine its representations further. The comment in the code already hints at this.Increase the sequence length. Change
maxSequenceLengthfrom 8 to 16. This lets the model see full-length names during training instead of truncating them at 7 characters. Training takes roughly twice as long, but you should see longer and more varied names during inference.Experiment with temperature. Try
temperaturevalues of 0.1, 1.0, and 2.0 in the inference loop. At 0.1 the model plays it very safe (repetitive but well-formed). At 2.0 it gets creative (diverse but sometimes incoherent). Compare the outputs side by side to build intuition for how temperature shapes generation.Remove RMSNorm. Comment out the
RmsNormcalls inModel.csand retrain. Watch what happens to the loss. Does it still converge? Does it converge more slowly, or does it blow up entirely? This shows you what normalisation is actually doing for training stability.Swap the nonlinearity. Replace
xi.Relu()in the MLP block with something else. Tryxi * xi(squaring), or even just remove the nonlinearity entirely (delete the ReLU line). The loss will tell you how much the choice of activation function matters at this scale.
Performance Optimisation Notes
The course code above prioritises clarity over speed. Martin Skuta's C# MicroGPT repo (linked in Credits) includes several C#-specific optimisations worth understanding once the concepts are solid:
Replace LINQ in Hot Paths. The course code uses .Select(...).ToList() and .Sum() throughout (for example, the ReLU step in the MLP block and the temperature-scaling and sampling code in inference). LINQ allocates an enumerator and a closure delegate on every call, which adds up quickly in a training loop running millions of operations. Rewriting these as plain for loops that append to a pre-sized List<Value> is the first optimisation to reach for - it's mechanical, preserves readability, and typically gives a noticeable speedup before you ever touch SIMD.
SIMD Vectorisation. The Value.Dot method in the repo uses System.Numerics.Vector<double> to process multiple elements per CPU instruction. This gives a significant speedup for the dot products that dominate the computation.
Iterative Backward Pass. We already used this: the explicit Stack<T> instead of recursion. This avoids stack overflow on deep graphs and eliminates function call overhead.
Zero-Allocation Hot Paths. The repo's Value.Dot pre-allocates the _inputs and _localGrads arrays once per node instead of creating intermediate Value objects for each multiply-and-add. This keeps garbage collection pressure low during training.
Backward Loop Unrolling. The Backward method can special-case nodes with 1 or 2 inputs (which covers ~99% of the graph: Add, Mul, ReLU, Pow) to avoid loop setup overhead.
Parallel Gradient Reset. Parallel.ForEach(paramsList, p => p.Grad = 0) uses multiple cores to zero out gradients.
These optimisations don't change the algorithm. They just make the same computation run faster. When studying the code, understand the clean version first, then read the optimised version as "the same thing, but faster".
From MicroGPT to ChatGPT
Everything in this course is the algorithmic essence of how LLMs work. Between this and a production model, nothing changes in the core algorithm. What changes is scale and engineering:
| Aspect | MicroGPT | Production LLM |
|---|---|---|
| Data | 32K short names | Trillions of tokens of internet text |
| Tokenizer | 1 char = 1 token (27 tokens) | BPE subwords (~100K tokens) |
| Computation recorder | Scalar Value objects |
Tensor operations on GPUs |
| Parameters | ~4,000 | Hundreds of billions |
| Training | 1 document per step, CPU | Millions of tokens per step, thousands of GPUs |
| Architecture | ReLU, learned position embeddings | GeLU/SwiGLU, RoPE, GQA, MoE |
| Post-training | None | SFT + RLHF to make it conversational |
If you understand what you've built in this course, you understand the algorithmic essence. Everything else is efficiency.
Glossary
| Term | First Appears | Definition |
|---|---|---|
| Value | Ch. 1 | A wrapper around a double that records how it was computed, enabling automatic gradient computation |
| Computation recorder | Ch. 1 | What we call the Value class and its Backward method collectively. The ML community typically calls this an "autograd engine" (short for automatic gradient computation) or "automatic differentiation". Same thing, different name |
| Gradient | Ch. 2 | How much the final loss would change if you nudged a particular value. Stored in Value.Grad
|
| Backward / Backpropagation | Ch. 2 | The algorithm that computes gradients by walking the computation graph in reverse |
| Chain Rule | Ch. 2 | The calculus rule that lets you multiply rates of change along a path |
| BOS | Ch. 3 | Beginning of Sequence token, a delimiter marking where documents start and end |
| Token | Ch. 3 | A discrete symbol (in our case, a character) that the model processes |
| Bigram | Ch. 4 | A model that predicts the next token using only the current token |
| Logits | Ch. 5 | Raw, unnormalised scores output by the model, one per vocabulary token |
| Softmax | Ch. 5 | A function that converts logits into a probability distribution |
| Linear | Ch. 5 | A matrix-vector multiplication, the fundamental learned transformation |
| Embedding | Ch. 6 | A learned vector associated with each token or position |
| Cross-Entropy Loss | Ch. 6 | The specific formula for computing the loss: -log(probability of correct token). This is the "loss" from the Big Picture |
| Adam | Ch. 7 | An optimiser that uses momentum and adaptive learning rates |
| RMSNorm | Ch. 8 | Normalisation that rescales a vector to unit root-mean-square |
| Residual Connection | Ch. 8 | Adding a layer's input to its output, creating a gradient highway |
| Attention (self-attention) | Ch. 9 | The mechanism where tokens compute relevance scores and exchange information with other tokens in the same sequence |
| Causal attention | Ch. 9 | Attention where each token can only look at positions before it, not ahead |
| Query / Key / Value (Q/K/V) | Ch. 9 | Three projections of each token used in the attention computation |
| KV Cache | Ch. 9 | Stored keys and values from previous positions, enabling efficient sequential processing |
| Head | Ch. 10 | One independent attention computation operating on a slice of the embedding |
| MLP | Ch. 10 | A two-layer feed-forward network for per-position computation |
| Transformer Block | Ch. 10 | Attention + MLP, each with RMSNorm and residual connections |
| Temperature | Ch. 12 | A scaling factor that controls the randomness of generated text |
References
These are the primary sources behind the claims and concepts in this course. If you want to verify something, dig deeper on a topic, or just see where the ideas originally came from, start here.
The Transformer Architecture
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). "Attention Is All You Need."
https://arxiv.org/abs/1706.03762
The paper that introduced the transformer. Our model uses the same core structure: multi-head self-attention and feed-forward layers on a residual stream.
The Adam Optimiser (Ch. 7)
Kingma, D. P., & Ba, J. (2014). "Adam: A Method for Stochastic Optimization."
https://arxiv.org/abs/1412.6980
The original paper describing the momentum, adaptive learning rate, and bias correction that our training loop uses.
RMSNorm (Ch. 8)
Zhang, B., & Sennrich, R. (2019). "Root Mean Square Layer Normalization." Advances in Neural Information Processing Systems 32 (NeurIPS 2019).
https://arxiv.org/abs/1910.07467
The paper proposing RMSNorm as a simpler alternative to LayerNorm. Our RmsNorm function implements the core idea from this paper.
GPT-2 (parameter count reference)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). "Language Models are Unsupervised Multitask Learners."
https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
The GPT-2 paper. The largest GPT-2 variant had 1.5 billion parameters.
Numerical Gradient Checking (Ch. 1)
PyTorch's torch.autograd.gradcheck function, which verifies analytical gradients against numerical finite differences:
https://docs.pytorch.org/docs/stable/generated/torch.autograd.gradcheck.gradcheck.html
This is the same nudge-and-measure technique used in our GradientCheck.cs.
Karpathy's micrograd video (Ch. 1-2)
Karpathy, A. (2022). "The spelled-out intro to neural networks and backpropagation: building micrograd."
https://www.youtube.com/watch?v=VMj-3S1tku0
A 2.5-hour walkthrough of the Value class and backpropagation. If you want a video companion to Chapters 1 and 2, this is it.
Karpathy's microgpt blog post
Karpathy, A. (2026). "microgpt."
https://karpathy.github.io/2026/02/12/microgpt/
The blog post that accompanies the original Python implementation. Covers the same progression as this course with additional mathematical detail.
Credits and Acknowledgements
This course is built on the work of others. It wouldn't exist without them.
Andrej Karpathy created the original microgpt - a 200-line Python implementation that distills a GPT into its bare algorithmic essence. The pedagogical progression used in this course (bigram -> MLP -> attention -> full transformer) follows the approach Karpathy developed across multiple projects including micrograd, makemore, and nanoGPT. His blog post and accompanying YouTube videos were invaluable references for the explanatory content throughout this course.
Martin Skuta (@martinskuta) wrote the C# implementation of microgpt that this course is based on. His translation from Python to C# - including the SIMD-vectorised dot product, the iterative backward pass, and the zero-allocation optimisations - demonstrated that the algorithm translates cleanly to .NET with no external dependencies. The Value class, the Gpt function structure, and the parameter dictionary layout in this course all derive from his work.
Jonas Ara (@jonas1ara) contributed the F# translation of the C# implementation to the same repository.
The training dataset (names.txt) is from Karpathy's makemore project.
This course was created and refined by Gary Jackson with the assistance of Claude (Anthropic). Gary provided the creative direction, pedagogical priorities, and iterative feedback that shaped the course structure, while Claude drafted and revised the content, code examples, and explanations.