Introduction
Andrej Karpathy uploaded several lecture videos on YouTube and the accompanying code on GitHub. I think they are excellent lectures, even better than many paid online courses. Here's the link: Neural Networks: Zero to Hero. So, I will summarize them and try to get meaningful insights from them. I'm going to cover all of them lecture by lecture (I hope...)
The first lecture is about backpropagation in neural networks.
The Spelled-out Intro to Neural Networks and Backpropagation
In the video, Karpathy said that backpropagation is what you need to train neural networks, and everything else is mainly for efficiency. That is why he explained and demonstrated backpropagation in the very first video.
I totally agree with him. Training neural networks and LLMs is essentially about reducing loss. And fundamentally, all methods for reducing loss are related to backpropagation, directly or indirectly.
Building Micrograd
Karpathy built a small project called micrograd. You can see the code here. This is made up of just a few simple lines of code, but it shows us how neural networks are built under the hood. In the video, he demonstrated how to build Micrograd and how it works step by step.
Differentiation and Derivative
We all learned differentiation and derivatives, right? What do differentiation and derivatives actually mean?
limh→0f(x+h)−f(x)h \lim_{h \to 0} \frac{f(x + h) - f(x)}{h}
Differentiation and derivatives tell us how much f(x)f(x) changes when xx changes. That is, they show the effect of a variable and the slope or gradient of a function. Then, if the derivative is 0, that point may be a local maximum or minimum of the function—not always, but it can be.
Actually, Karpathy didn't explain differentiation and derivatives in detail in the video. However, I think this is one of the most important aspects for understanding neural networks. So I'm going to explain this in more detail.
Linear Regression and Derivative
This concept is fundamental to linear regression as well. What is the core idea of linear regression? The goal is to minimize the residual sum of squares (RSS).
RSS=∑i=1n(yi−y^i)2=∑i=1n(yi−β1xi−β0)2 \begin{align*} RSS &= \sum_{i=1}^{n} \left( y_i - \hat{y}_i \right)^2 \\ &= \sum_{i=1}^{n} \left( y_i - \beta_{1} x_i - \beta_{0} \right)^2 \end{align*}
This is what linear regression is all about: finding the optimal β\beta values. Then, how to find them? This is where the derivative comes in. The RSS formula is a kind of quadratic function, so when it comes to quadratic functions, the minimum point is the point at which the derivative becomes 0. Therefore, if we differentiate the equation and find where the derivative is zero, we can find the best β1\beta_1 and β0\beta_0 .
To find the optimal β0\beta_0 and β1\beta_1 , we take the partial derivatives of RSS with respect to each parameter and set them equal to zero. I will demonstrate how to derive them.
RSS(β0,β1)=∑i=1n(yi−β1xi−β0)2 RSS(\beta_0, \beta_1) = \sum_{i=1}^{n} \left(y_i - \beta_1 x_i - \beta_0 \right)^2
First, differentiate the RSS with respect to β0\beta_0 :
∂RSS∂β0=−2∑i=1n(yi−β1xi−β0) \frac{\partial RSS}{\partial \beta_0} = -2 \sum_{i=1}^{n} \left(y_i - \beta_1 x_i - \beta_0 \right)
Then, differentiate the RSS with respect to β1\beta_1 :
∂RSS∂β1=−2∑i=1nxi(yi−β1xi−β0) \frac{\partial RSS}{\partial \beta_1} = -2 \sum_{i=1}^{n} x_i \left(y_i - \beta_1 x_i - \beta_0 \right)
At the minimum point, both partial derivatives are zero:
∂RSS∂β0=0∂RSS∂β1=0 \begin{align*} \frac{\partial RSS}{\partial \beta_0} &= 0 \\ \frac{\partial RSS}{\partial \beta_1} &= 0 \end{align*}
This gives us the normal equations:
∑i=1nyi=β1∑i=1nxi+nβ0∑i=1nxiyi=β1∑i=1nxi2+β0∑i=1nxi \begin{align*} \sum_{i=1}^{n} y_i &= \beta_1 \sum_{i=1}^{n} x_i + n\beta_0 \\ \sum_{i=1}^{n} x_i y_i &= \beta_1 \sum_{i=1}^{n} x_i^2 + \beta_0 \sum_{i=1}^{n} x_i \end{align*}
Solving these equations gives the optimal values:
β1=∑i=1n(xi−xˉ)(yi−yˉ)∑i=1n(xi−xˉ)2 \beta_1 = \frac{ \sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y}) }{ \sum_{i=1}^{n} (x_i - \bar{x})^2 }
β0=yˉ−β1xˉ \beta_0 = \bar{y} - \beta_1 \bar{x}
Here, xˉ\bar{x} is the mean of the input values, and yˉ\bar{y} is the mean of the target values.
Neural networks are also built from these kinds of linear expressions, usually combined with nonlinear activation functions. However, the way we calculate the parameters is totally different because neural networks are much more complicated and have so many parameters. So it is almost impossible to find the parameters in this way.
Finding Your Way in Pitch Darkness
Assume that while hiking in the mountains, we get lost and trying to get down the mountain. But it is night, so we are in pitch darkness. We can only see a few inches around us. In this case, how can we get down the mountain? The answer is simple: by following the slope downward. At least, if we can see the slope around us, we can tell which way leads downward. This is how we find the optimal point when the equation is so complex that we are not able to solve for the optimum analytically.
def f(x):
return 3*x**2 - 4*x + 5
f(3.0)
xs = np.arange(-5, 5, 0.25)
ys = f(xs)
plt.plot(xs, ys)
This is an example from Karpathy's code. The function is f(x)=3x2−4x+5f(x) = 3x^2 - 4x + 5 . Its derivative is df(x)dx=6x−4\frac{d f(x)}{d x} = 6x - 4 .
If we solve the equation 6x−4=06x - 4 = 0 , the derivative is 0 at x=2/3x = 2/3 . Then, if xx is at some other point, how can we move xx to find the minimum point of f(x)f(x) ?
The answer is simple. If the derivative value at a certain point is greater than 0, we have to decrease xx ; if it is less than 0, we have to increase xx . Therefore, if we consistently subtract λ(6x−4)\lambda (6x - 4) from xx , with a proper learning rate λ\lambda , f(x)f(x) will converge to its minimum.
This finite-difference approach is useful for building intuition and for gradient checking.
h = 0.000001
x = 2/3
(f(x + h) - f(x))/h
The output is 2.999378523327323e-06, which is almost zero. It is not perfectly exact because floating-point numbers have limited precision, and because this is a finite-difference approximation, but it is close enough for this simple demonstration.
Similarly, when you have an expression with several variables, you can get the slope with respect to a specific variable in this way.
h = 0.0001
# inputs
a = 2.0
b = -3.0
c = 10.0
d1 = a*b + c
c += h
d2 = a*b + c
print('d1', d1)
print('d2', d2)
print('slope', (d2 - d1)/h)
Output:
d1 4.0
d2 4.0001
slope 0.9999999999976694
This is also Karpathy's code. This example shows how
dd
changes when
cc
changes from 10.0. The gradient is 1.0, of course, since the derivative
ddc(ab+c)\frac{d}{dc}(ab + c)
is 1.
Calculating the Gradient
Karpathy demonstrates a hands-on example of how to calculate the gradients. Let's see one of his examples.
a=−2.0b=3.0d=a×be=a+bf=d×e \begin{align*} a &= -2.0 \\ b &= 3.0 \\ d &= a \times b \\ e &= a + b \\ f &= d \times e \end{align*}
When ff is the final output, we should calculate the gradients with respect to all intermediate values. Let's do this one by one.
The gradient of ff with respect to itself is 1 because ddff=1\frac{d}{df}f = 1 . Easy, right?
The important thing is that addition passes the gradient through. For example, if x=a+bx = a + b and ddxf(x)=k\frac{d}{dx}f(x) = k , the gradients of aa and bb are also kk . For subtraction, the subtracted term receives the negative of the upstream gradient. And when it comes to multiplication, the upstream gradient is multiplied by the other variable. If x=a×bx = a \times b , the gradient of aa is k×bk \times b because the local derivative with respect to aa is bb .
Therefore, the gradient of ee is ddef\frac{d}{de}f . ff is d×ed \times e , so the gradient ee is d=−6.0d=-6.0 . On the other hand, the gradient of dd is 1.0.
Finally, the gradient of aa is the contribution from ee plus the contribution from dd . So it is −6+3=−3-6 + 3 = -3 . The gradient of bb is also the contribution from ee plus the contribution from dd , and that is −6−2=−8-6 - 2 = -8 .
The computation graph for the final output looks like this:
What if we want to minimize ff by tuning the value of aa ? By subtracting the gradient, −3λ-3\lambda , ff will get smaller since the function with respect to aa is an upward-opening quadratic function. If λ\lambda is 0.1, we subtract -0.3 from aa . Then aa becomes -1.7. As a result, ff becomes -6.63, which is smaller.
Now, let's apply this algorithm to a neural network.
Neural Networks and Gradient Descent
I organized another example from Karpathy in the image above. This is a very simple neural network architecture that he made. Actually, this is not the whole story yet. What we want to minimize is the loss function. So, if oo is y^\hat{y} , gradient descent should minimize ∑i=1n(yi−oi)2\sum_{i=1}^{n} \left( y_i - o_i \right)^2
In this way, Karpathy shows hands-on code that runs gradient descent on a simple MLP.
Here's the MLP training loop Karpathy builds in the Micrograd lecture:
class Neuron:
def __init__(self, nin):
self.w = [Value(random.uniform(-1,1)) for _ in range(nin)]
self.b = Value(random.uniform(-1,1))
def __call__(self, x):
# w * x + b
act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b)
out = act.tanh()
return out
def parameters(self):
return self.w + [self.b]
class Layer:
def __init__(self, nin, nout):
self.neurons = [Neuron(nin) for _ in range(nout)]
def __call__(self, x):
outs = [n(x) for n in self.neurons]
return outs[0] if len(outs) == 1 else outs
def parameters(self):
return [p for neuron in self.neurons for p in neuron.parameters()]
class MLP:
def __init__(self, nin, nouts):
sz = [nin] + nouts
self.layers = [Layer(sz[i], sz[i+1]) for i in range(len(nouts))]
def __call__(self, x):
for layer in self.layers:
x = layer(x)
return x
def parameters(self):
return [p for layer in self.layers for p in layer.parameters()]
x = [2.0, 3.0, -1.0]
n = MLP(3, [4, 4, 1])
n(x)
xs = [
[2.0, 3.0, -1.0],
[3.0, -1.0, 0.5],
[0.5, 1.0, 1.0],
[1.0, 1.0, -1.0],
]
ys = [1.0, -1.0, -1.0, 1.0] # desired targets
for k in range(20):
# forward pass
ypred = [n(x) for x in xs]
loss = sum((yout - ygt)**2 for ygt, yout in zip(ys, ypred))
# backward pass
for p in n.parameters():
p.grad = 0.0
loss.backward()
# update
for p in n.parameters():
p.data += -0.1 * p.grad
print(k, loss.data)
With his Micrograd code, you can see how the gradient descent algorithm works step by step. You can also calculate the gradient of each variable on your own. I strongly recommend doing these hands-on examples. After watching the video, I was able to clearly understand how gradient descent works, why we should use zero out gradients, why ReLU function is the most efficient, and so on. This is definitely worth your time.
Conclusion
I have shown Karpathy's demonstrations of gradient descent. As he said, this is the core concept for training neural networks. The rest is just for efficiency. Reducing loss using gradients: this is what makes neural network training possible and, ultimately, helped usher in the AI era.