Self-Attention is not just “looking at important words.”

It is a matrix operation.

And that is exactly why Transformers scale.

Core Idea

Self-Attention lets each token compare itself with every other token in the same sequence.

Each token asks:

Which other tokens are useful for updating my representation?

This matters because meaning is contextual.

A token should not stay as a static embedding.

It should become a representation shaped by the sentence around it.

The Key Structure

Self-Attention follows this pipeline:

More compactly:

Self-Attention = matching + weighting + information mixing

The full formula is:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V

This equation looks dense.

But the idea is simple:

Compare tokens.

Convert scores into weights.

Use weights to mix information.

Pseudo-code View

At a high level, Self-Attention works like this:

X = token_embeddings

Q = X @ W_Q
K = X @ W_K
V = X @ W_V

scores = Q @ K.T

scaled_scores = scores / sqrt(d_k)

weights = softmax(scaled_scores)

output = weights @ V

That is the core computation.

In real Transformer implementations, this is done for all tokens at once.

Not token by token.

That is why the matrix form matters.

Concrete Example

Take this sentence:

I love you

When updating the token “love”, Self-Attention compares it with:

The token “love” may strongly attend to “I” and “you”.

So its representation becomes more contextual.

It no longer means only the word “love.”

It becomes something closer to:

love as an action between I and you

That is why Self-Attention is powerful.

It turns isolated token vectors into relationship-aware vectors.

QKV Intuition

Each token is projected into three roles:

Query, Key, and Value.

Query:

What am I looking for?

Key:

What do I contain that others can match against?

Value:

What information do I pass forward if selected?

Search analogy:

Query = search request

Key = searchable index

Value = retrieved content

This separation is important.

The model can learn different spaces for matching and information transfer.

Step 1: Generate Q, K, and V

Given input embeddings X:

W_Q, W_K, and W_V are learned matrices.

They are trained with the model.

This means QKV is not manually designed.

The model learns how to project tokens into attention roles.

Implementation-wise, this is just matrix multiplication.

Conceptually, it creates three different views of the same token.

Step 2: Compute Attention Scores

The model compares Query and Key vectors.

For one token:

score = q · k

A larger dot product means stronger similarity.

Example:

The first key matches more strongly.

But these are still raw scores.

They are not probabilities yet.

Step 3: Scale and Apply Softmax

Dot products can become large when vector dimensions grow.

Large scores can make Softmax too sharp.

That can make training unstable.

So Self-Attention scales the scores:

score = (q · k) / √dₖ

Then Softmax converts scores into weights.

Example:

scores = [14, 12]

softmax(scores) ≈ [0.88, 0.12]

Now the model has attention weights.

These weights say how much each token should contribute.

This matters in practice.

Without scaling, attention can collapse too aggressively onto one token.

Step 4: Weighted Sum of Values

The final output is a weighted sum of Value vectors.

z = Σ αᵢvᵢ

Example:

values = [10, 20]

weights = [0.88, 0.12]

output = 0.88 × 10 + 0.12 × 20 = 11.2

The first value contributes more.

The second value contributes less.

That is the basic meaning of attention output.

It is not a simple average.

It is selective information mixing.

Self-Attention vs Cross-Attention

Self-Attention:

• Query, Key, and Value come from the same sequence
• models relationships inside one sequence
• used in Transformer encoders and decoders

Cross-Attention:

• Query comes from the decoder
• Key and Value come from the encoder
• models relationships between two sequences
• used in encoder-decoder models

In short:

Self-Attention = inside the same sequence

Cross-Attention = between different sequences

This difference matters when reading Transformer code.

If Q, K, and V come from the same tensor, it is Self-Attention.

If Q comes from one tensor and K/V come from another, it is Cross-Attention.

Naive vs Matrix View

Naive view:

Each token compares with every other token one by one.

Matrix view:

All token relationships are computed at once.

Naive logic:

for token_i in tokens:
    for token_j in tokens:
        compute_similarity(token_i, token_j)

Matrix logic:

scores = Q @ K.T

That single matrix multiplication computes all pairwise token scores.

This is why Transformers are GPU-friendly.

They replace sequential loops with dense linear algebra.

Why Matrix Computation Matters

The attention matrix contains token-to-token relationships.

If the sequence length is n, the score matrix is n × n.

Each row means:

How much one token attends to every token.

Each column means:

How much that token is attended to by others.

This structure is powerful.

But it also creates a cost problem.

Full Self-Attention grows roughly with O(n²).

Longer context means more computation and memory.

So the same design that makes attention expressive also makes it expensive.

That is why efficient attention methods exist.

Important Conditions and Limits

Self-Attention needs positional information.

By itself, attention compares token content.

It does not automatically know token order.

Self-Attention also gets expensive as sequence length grows.

For short and medium sequences, full attention is powerful.

For very long sequences, memory and compute become major constraints.

Another important point:

Attention weights are not always perfect explanations.

They show how information is mixed.

But they should not always be treated as human-level reasoning traces.

Implementation Perspective

In real models, QKV projection is often implemented as one combined linear layer.

Instead of computing three separate matrix multiplications:

Implementations often compute:

QKV = XW_QKV

Then split the result into Q, K, and V.

This is faster and cleaner.

The math stays the same.

The implementation is optimized.

That is the developer mindset:

Understand the formula.

Then recognize the optimized tensor layout in code.

Takeaway

Self-Attention is the core operation behind Transformers.

It works by projecting tokens into Q, K, and V.

Q and K compute relevance.

Softmax turns relevance into weights.

Weights mix V into contextual outputs.

The shortest version is:

Self-Attention = compare tokens → weight information → update representations

If you understand QKᵀ and weighted Values, you understand the heart of Transformer computation.

Discussion

When reading Transformer code, which part feels most confusing?

QKV projection, Softmax attention weights, or the final matrix multiplication with V?

Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/self-attention-qkv-matrix-en/

GitHub Resources
AI diagrams, study notes, and visual guides:
https://github.com/zeromathai/zeromathai-ai