A neural network is not magic.
It is a system that transforms input into output through layers.
The real question is:
How does that system learn from mistakes?
Core Idea
A neural network takes data, passes it through connected layers, and produces an output.
During training, it adjusts internal values so future outputs become better.
Those internal values are mainly weights and biases.
So the simplest view is:
Input → Layers → Output → Error → Update
That loop is the foundation of neural network learning.
The Key Structure
A basic neural network looks like this:
Input Layer → Hidden Layers → Output Layer
Each layer transforms the data.
A neuron usually computes:
z = w · x + b
Then an activation function transforms it:
a = activation(z)
Where:
- x = input
- w = weight
- b = bias
- z = raw score
- a = activated output
This is the basic building block.
Implementation View
At a high level, training works like this:
take input data
run forward propagation
compute prediction
compare prediction with target
compute loss
run backpropagation
update weights and biases
repeat
This is why neural networks are trainable.
They do not just compute outputs.
They use errors to adjust themselves.
Concrete Example
Imagine a small model that predicts whether an email is spam.
The input features might include:
- suspicious words
- sender reputation
- number of links
- message length
The network processes those inputs.
It predicts:
spam probability = 0.82
If the real label is spam, the prediction is good.
If the real label is not spam, the network needs to adjust.
Backpropagation tells the model which weights contributed to the mistake.
Then training updates those weights.
Perceptron vs MLP
The perceptron is the simplest starting point.
It takes inputs, applies weights, adds a bias, and produces an output.
But a single perceptron is limited.
It can only represent simple decision boundaries.
A Multi-Layer Perceptron expands this idea.
It stacks multiple layers.
Perceptron:
- one basic computational unit
- simple input-output mapping
- limited representational power
MLP:
- multiple connected layers
- hidden representations
- more expressive decision boundaries
So the MLP is where neural networks become more useful.
Forward Propagation vs Backpropagation
This comparison is the core of learning.
Forward propagation computes the prediction.
Backpropagation computes how to improve it.
Forward propagation:
- moves from input to output
- calculates activations
- produces prediction
- computes loss
Backpropagation:
- moves from loss backward
- computes gradients
- assigns error responsibility
- prepares parameter updates
Forward pass answers:
“What did the model predict?”
Backward pass answers:
“What should change?”
Why Activation Functions Matter
Stacking linear layers is not enough.
Without activation functions, multiple layers can collapse into one linear transformation.
That means the model cannot represent complex patterns well.
Activation functions add nonlinearity.
This lets neural networks learn curves, boundaries, and deeper representations.
In short:
Linear layers calculate.
Activation functions reshape.
Both are needed.
What Actually Changes During Learning?
When a neural network learns, it updates parameters.
The most important parameters are:
- weights
- biases
Weights control how strongly inputs affect the output.
Biases shift the output.
A simple update idea is:
new parameter = old parameter - learning rate × gradient
The gradient tells the direction.
The learning rate controls the step size.
That is the practical meaning of learning.
From Neural Networks to Deep Learning
A neural network becomes “deep” when it has many layers.
But depth alone is not enough.
Deep learning depends on:
- layered representations
- nonlinear activations
- backpropagation
- parameter updates
- enough data and compute
The deeper the network, the more abstract its internal representations can become.
For example:
In image models:
- early layers may detect edges
- middle layers may detect shapes
- deeper layers may detect objects
That is why neural networks became the foundation of modern deep learning.
Recommended Learning Order
If neural networks feel scattered, learn them in this order:
- Perceptron
- Multi-Layer Perceptron
- Neural Network
- Forward Propagation
- Backpropagation
- Activation Function
- Weights and Biases
- Model Parameters
- Deep Learning
This order works because you first understand the basic unit.
Then you understand the network.
Then you understand computation, learning, and deep extensions.
Takeaway
A neural network is a trainable system of layered transformations.
The shortest version is:
Neural Network = layers + activations + parameters + learning
Forward propagation makes predictions.
Backpropagation computes corrections.
Weights and biases store what the model has learned.
If you remember one idea, remember this:
Neural networks learn by repeatedly predicting, measuring error, and updating internal parameters.
Discussion
When learning neural networks, do you find it easier to start from the perceptron, or from the full forward/backpropagation training loop?
Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/neural-network-hub-en/
GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai