For illustration only For decades, machines read language the way a goldfish swims — forward only, forgetting everything behind. Attention changed that completely. We closed Week 1 by watching the bigram model fail in slow motion. It could see exactly one token back, make a locally reasonable prediction, and then immediately forget everything it had just processed. The result was text that felt written by someone with no short-term memory. Each word plausible in isolation, the whole sequence globally incoherent. The long-range dependency problem wasn’t a bug in the bigram’s implementation. It was a fundamental consequence of its design. Week 2 begins with the mechanism that solved that problem permanently. Self-attention is the architectural innovation that transformed language modeling from sophisticated pattern-matching into genuine contextual understanding. It sits at the heart of every transformer, and therefore at the heart of every LLM you’ve ever used. Understanding it conceptually is the single most important step in this entire series. Why Everything Before Attention Struggled To appreciate what attention actually solved, you need to understand what came before it and why that approach hit a wall. The dominant approach before transformers was the recurrent neural network , or RNN. An RNN reads a sequence one token at a time, left to right, maintaining a hidden state , a fixed-size vector that’s supposed to compress and carry forward everything the model has read so far. At each step, the new token updates this hidden state, which is then passed to the next step. It’s a chain of handoffs, one after another. Think of it like a relay race where every runner must summarize the entire race story so far, on a single index card, before passing it to the next runner. By the time runner 50 gets the card, the story from runner 1 has been compressed, overwritten, and paraphrased so many times that it barely resembles the original. In language terms, by the time an RNN reaches the word “expensive” near the end of a long sentence, the subject “books” from the beginning may have been squeezed out of the hidden state entirely to make room for everything that came in between. This is the long-range dependency problem — any situation where the correct interpretation of a word depends on context that appeared many tokens earlier. RNNs could technically process the full sequence; they just couldn’t reliably remember it. The hidden state was a bottleneck, and long sequences meant lossy compression. Self-attention discards this sequential chain entirely. Instead of reading one token at a time and passing information forward through a bottleneck, attention processes all tokens simultaneously and allows every token to communicate directly with every other token in the sequence, regardless of distance. The Core Intuition: Every Token Gets to Ask About Every Other Token Here is the mental model that makes self-attention genuinely click. Imagine you’re at a professional networking event with twenty people in the room. Instead of passing notes one person at a time down a line, everyone can speak directly to everyone else at once. Each person decides who in the room is most relevant to their current conversation and pays proportionally more attention to those people, while tuning out the less relevant voices. That is self-attention. Every token in the sequence looks at every other token simultaneously and decides how much to borrow from each one when building its own updated representation. The result is that each token’s final representation is not just its own isolated embedding. It is a weighted blend of information from all the other tokens in the sequence, where the weights reflect how relevant each other token is to understanding the current one. This is why the mechanism is called “self”-attention: the sequence attends to itself. No external memory bank, no separate encoder feeding information in. Every token enriches every other token by attending directly to the full sequence in a single operation. How the Weights Are Decided — Conceptually The word “attention” implies focus, and focus implies a decision about what matters. So how does the model decide which tokens deserve the most influence over any given token’s representation? The intuition maps cleanly onto a library search. When you walk into a library with a research question, your question acts as a query , it describes what you’re looking for. Each book in the library has a catalog entry, a key , that summarizes what it contains. You compare your query against every key to identify which books are most relevant, then you actually read those books to extract their content, the values . Self-attention works the same way. For each token, the model generates a query representing what that token is “looking for” in context. Every token also generates a key representing what it has to offer. Attention scores are computed by measuring how well each token’s key matches the current token’s query. Tokens with strong matches receive high scores, tokens with weak matches receive low scores. Those scores are then normalized into weights using the softmax function we met on Day 6, so they behave like a probability distribution that sums to 1. Finally, each token computes a weighted sum of the values across the sequence, collecting more from the highly relevant tokens and barely registering the irrelevant ones. The result is a fresh, context-enriched representation for every token, built from the full sequence in a single pass. Tomorrow, we’ll formalize this with the actual Q, K, and V matrices. For now, hold this picture: every token asks a question, compares that question against every other token’s offering, decides how much attention to pay to each, and constructs its updated representation accordingly. A Concrete Example: Resolving What “It” Actually Means Think about a simple sentence like this: “The cat was sitting on the sofa and it jumped on a mat.” For us, it’s effortless. We know “it” is the cat. Our brains naturally track the subject across the sentence because we understand that cats jump and furniture doesn’t. But for an AI, this is a specific challenge called coreference resolution. The model has to decide if “it” refers to the cat, the sofa, or even the mat itself. A bigram model has no hope. It sees “it” and samples whatever most commonly follows “it” in its training data. An RNN might have partially overwritten “cat” in its hidden state by the time it reaches “it,” depending on how much intervening information updated the state. The connection is fragile. Self-attention changes the game. Instead of hoping the memory of the cat survives the trip across the sentence, the word “it” is allowed to look back at every other word simultaneously. It essentially asks, “Which of these other words explains what I am?” It sees the sofa (a static object) and the cat (a living thing that jumps), recognizes the statistical relationship between “cats” and “jumping,” and creates a direct, high-strength link between the two. The distance between the words becomes irrelevant; the connection is made because the context demands it, not just because the words are near each other. Critically, this works at any distance. Whether the relevant context is 3 tokens away or 300, self-attention bridges the gap in one operation because it’s not threading information through a sequential chain. It draws directly from the entire sequence at once, and distance costs nothing. Parallelism: Why This Also Transformed Training Speed The benefits of self-attention aren’t limited to accuracy. One of its most consequential properties is computational and it transformed what was practically possible to train. RNNs are sequential by design . Computing the hidden state at step 50 requires completing steps 1 through 49 first, in order. This means the computation cannot be parallelized across the sequence length. On modern GPU hardware which, as we covered on Day 5, is specifically optimized for doing thousands of things simultaneously — sequential processing is an enormous waste of capability. Self-attention computes the relationships between all token pairs in a single matrix operation, with all positions processed in parallel. Queries, keys, attention scores, weighted value sums. Everything happens simultaneously across the entire sequence. This maps perfectly onto GPU architecture, where thousands of cores can execute the same operation on different slices of data at the same time. This parallelism is a substantial reason why transformers could be scaled to their current sizes. Training a model with hundreds of billions of parameters using sequential recurrence would require impractical amounts of wall-clock time even on the world’s largest GPU clusters. With attention’s parallelism, the computation distributes efficiently and training runs that might take years with an RNN can complete in weeks with a transformer. Busting the Myth: “Attention Means the Model Understands Language” Self-attention is powerful enough and its outputs impressive enough, that it’s easy to overattribute intelligence to it. A persistent misconception, particularly among non-technical observers, is that because the model “attends” to the right parts of a sentence, it must understand what that sentence means in a meaningful cognitive sense. It doesn’t, not in that sense. Self-attention is a learned statistical weighting mechanism. It has no built-in knowledge of grammar, causality, physical reality, or meaning. What it has learned, through exposure to trillions of tokens during training, is which token relationships tend to be statistically useful for predicting subsequent tokens correctly. It attends to “animal” when processing “it” not because it understands biological fatigue or physical locomotion but because patterns in its training data consistently rewarded attending to subject nouns when processing pronouns in similar clause structures. A related misconception is that attention is the complete model. It isn’t. After attention mixes contextual information into each token’s representation, the outputs still flow through standard feed-forward neural network layers that perform further transformations. Attention is one crucial piece of a larger architecture, the piece that handles context aggregation. The rest of the transformer handles everything else, and we’ll build through those components over the next two weeks. The practical consequence of understanding attention as statistical rather than cognitive: the model will confidently fail when the statistical patterns it learned during training are absent or misleading for a given input. It has no internal signal that says “I’m outside my training distribution here.” Knowing this helps you anticipate where confident-sounding outputs should be verified rather than trusted. Practical Takeaway: How This Changes the Way You Write Prompts The conceptual model of attention has immediate, actionable implications for how you interact with LLMs and for how you design systems around them. Because each token’s representation is built by attending to the full context, the quality of any token’s representation depends on what else is present in that context. This means critical information should be present and prominent. When you bury an important constraint at the very end of a long prompt, the tokens that generate the model’s response will attend to that constraint but they will also be competing with every other piece of context in the window. Placing key constraints early, and reinforcing them, when necessary, increases the effective influence those tokens have on the model’s output. It also explains why context windows matter and why retrieval-augmented generation works. Both approaches give the attention mechanism more high-quality, relevant tokens to draw from when constructing each token’s representation. When the relevant information is present in the context, attention can find it. When it isn’t, the model can only fall back on patterns baked into its weights during training which is precisely when hallucination becomes more likely. For engineers building applications: when a model loses track of an instruction you specified at the top of a long conversation, it’s often because the attention weights for that instruction have been diluted by the accumulated volume of subsequent tokens. Structural techniques, clear section markers, repeated key constraints, concise context, all work because they shape what the attention mechanism is most likely to focus on during generation. Architect’s Note For the engineers thinking carefully about architectural choices: the decisive advantage of self-attention over sequential recurrence isn’t only long-range dependency. It’s the quality and stability of the gradient signal during training. In an RNN, the gradient must propagate backward through every sequential time step to reach parameters associated with early inputs. Across long sequences, this gradient passes through hundreds of weight matrices, causing it to either decay toward zero, the vanishing gradient problem, or amplify toward instability. Self-attention eliminates this structurally. Because every token has a direct learned connection to every other token in the sequence, the gradient path between any two positions is short and direct, regardless of their distance in the sequence. This is what makes training deep transformers on long sequences stable and tractable and it is one of the primary reasons the transformer displaced the LSTM, which used gating mechanisms as a partial workaround for vanishing gradients, almost entirely within a few years of its introduction in 2017. Think about a time when a state-of-the-art LLM lost track of something important you specified earlier in a long conversation. Based on what you now understand about how attention works, what do you think happened and how would you restructure your prompt to prevent it? Drop your example in the comments. The Attention Breakthrough — How Language Models Finally Learned to Focus was originally published in GoPenAI on Medium, where people are continuing the conversation by highlighting and responding to this story.