Here we are introducing Neural Networks and Backpropagation.
We will start with a 2-layer example:
f=W2max(0,W1x)f = W_2 \max(0, W_1 x)
x∈RD,W1∈RH×D,W2∈RC×Hx \in \mathbb{R}^D, W_1 \in \mathbb{R}^{H \times D}, W_2 \in \mathbb{R}^{C \times H}
The maxmax here creates some non-linearity between W1W1 and W2W2. It is called the activation function.
In practice, we will usually add a learnable bias at each layer as well.
Here’s an example of a 3-layer one.
f=W3max(0,W2max(0,W1x))f = W_3 \max(0, W_2 \max(0, W_1 x))
x∈RD,W1∈RH1×D,W2∈RH2×H1,W3∈RC×H2x \in \mathbb{R}^D, W_1 \in \mathbb{R}^{H_1 \times D}, W_2 \in \mathbb{R}^{H_2 \times H_1}, W_3 \in \mathbb{R}^{C \times H_2}
The networks we are talking about here are more accurately called fully-connected networks, or sometimes called multi-layer perceptrons (MLP).
The most important characteristic of activation functions is to create some non-linearity.
There are multiple activation functions to choose from.
The function max(0,x)max(0,x) is called ReLU. It is a good default choice for most problems.
ReLU(x)=max(0,x)\text{ReLU}(x) = \max(0, x)
We also have:
Choosing an activation function is very empirical.
In a Neural Network, we have an input layer, an output layer, and several hidden layers.
1 | import numpy as np |
The number of neurson in the hidden layers is often related to the capacity of information. More neurons, more capacity.
A regularizer keeps the network simple and prevents overfitting. Do not use size of neural network as a regularizer. Use stronger regularization instead.
f(x,y)=x+y→∂f∂x=1,∂f∂y=1f(x, y) = x + y \to \frac{\partial f}{\partial x} = 1, \frac{\partial f}{\partial y} = 1
f(x,y)=max(x,y)→∂f∂x=1(x≥y),∂f∂y=1(y≥x)f(x, y) = \max(x, y) \to \frac{\partial f}{\partial x} = \mathbb{1}(x \geq y), \frac{\partial f}{\partial y} = \mathbb{1}(y \geq x)
f(x,y)=xy→∂f∂x=y,∂f∂y=xf(x, y) = xy \to \frac{\partial f}{\partial x} = y, \frac{\partial f}{\partial y} = x
The derivative on each variable tells you the sensitivity of the whole expression on its value.
The problem now is to calculate the gradients in order to learn the parameters.
Obviously we can’t simply derive it on paper, for it can be very tedious and not feasible for complex models.
Here’s the better idea: Computational graphs + Backpropagation.
Backpropagation is generally chain rules.
Downstream=Upstream×Local\text{Downstream} = \text{Upstream} \times \text{Local}
∂L∂x=∂L∂z⋅∂z∂x\frac{\partial L}{\partial x} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial x}
1 | ================================================================================ |
Gate / Node / Function object: Actual PyTorch code
1 | class Multiply(torch.autograd.Function): |
For vector derivatives:
x∈RN,y∈Rx \in \mathbb{R}^N, y \in \mathbb{R}
The derivative is a gradient.
∂y∂x∈RN(∂y∂x)n=∂y∂xn\frac{\partial y}{\partial x} \in \mathbb{R}^N \quad \left(\frac{\partial y}{\partial x}\right)_n = \frac{\partial y}{\partial x_n}
x∈RN,y∈RMx \in \mathbb{R}^N, y \in \mathbb{R}^M
The derivative is a Jacobian Matrix.
∂y∂x∈RN×M(∂y∂x)n,m=∂ym∂xn\frac{\partial y}{\partial x} \in \mathbb{R}^{N \times M} \quad \left(\frac{\partial y}{\partial x}\right)_{n,m} = \frac{\partial y_m}{\partial x_n}
A Jacobian Matrix is the derivative for a function that maps a Vector to a Vector.
Definition
Let f:Rn→Rm\mathbf{f} : \mathbb{R}^n \to \mathbb{R}^m be a function such that each of its first-order partial derivatives exists on Rn\mathbb{R}^n. This function takes a point x=(x1,…,xn)∈Rn\mathbf{x} = (x_1, \ldots, x_n) \in \mathbb{R}^n as input and produces the vector f(x)=(f1(x),…,fm(x))∈Rm\mathbf{f}(\mathbf{x}) = (f_1(\mathbf{x}), \ldots, f_m(\mathbf{x})) \in \mathbb{R}^m as output.
Then the Jacobian matrix of f\mathbf{f}, denoted Jf\mathbf{J}_{\mathbf{f}}, is the m×nm \times n matrix whose (i,j)(i, j) entry is ∂fi∂xj\frac{\partial f_i}{\partial x_j}; explicitly:
Jf=[∂f∂x1⋯∂f∂xn]=[∇Tf1⋮∇Tfm]=[∂f1∂x1⋯∂f1∂xn⋮⋱⋮∂fm∂x1⋯∂fm∂xn]\mathbf{J}_{\mathbf{f}} = \begin{bmatrix} \frac{\partial \mathbf{f}}{\partial x_1} & \cdots & \frac{\partial \mathbf{f}}{\partial x_n} \end{bmatrix} = \begin{bmatrix} \nabla^T f_1 \\ \vdots \\ \nabla^T f_m \end{bmatrix} = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_m}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_n} \end{bmatrix}
where ∇Tfi\nabla^T f_i is the transpose (row vector) of the gradient of the ii-th component.
1 | ================================================================================ |
Jacobians can be sparse. For element-wise operations, the off-diagonal elements are always zero; the Jacobian becomes a diagonal matrix.
These allow us to propagate error (δ\delta) from the output backward through the network. We get them from simple chain rules.
Notation:
(BP1) Error at the Output Layer
δL=∇aC⊙σ′(zL)\delta^L = \nabla_a C \odot \sigma'(z^L)
Interpretation: How much the output layer “missed” the target, scaled by the activation’s sensitivity.
(BP2) Propagating Error to Hidden Layers
δl=((wl+1)Tδl+1)⊙σ′(zl)\delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l)
Interpretation: The error at layer ll is the weighted sum of errors from the next layer (l+1l+1), pulled backward through the transpose of the weights.
(BP3) Gradient for Biases
∂C∂bjl=δjl\frac{\partial C}{\partial b^l_j} = \delta^l_j
Interpretation: The gradient for a bias is exactly equal to the error at that neuron.
(BP4) Gradient for Weights
∂C∂wjkl=akl−1δjl\frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j
Interpretation: The gradient for a weight is the product of the input activation (al−1a^{l-1}) and the output error (δl\delta^l).
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。