A new study suggests that instead of endlessly inflating models, it may be more efficient to increase the frequency of specific tasks in training data to anchor rare skills in smaller models.

A new study from researchers at Anthropic, Stanford, and other institutions explains why larger language models learn certain tasks that smaller ones fail at. The finding goes beyond the conventional wisdom that big models simply learn faster.

In some cases, small models can't reliably learn rare tasks even with extremely long training runs. Even well-known scaling laws show that a small model never reaches the loss of a large one, no matter how much data you throw at it.

Common tasks crowd out rare ones

To isolate the mechanism, the researchers tested a mix of tasks with varying frequency and complexity. A model with N neurons gets assigned the N "most useful" features, where usefulness is based on how often a task appears and how important it is. Frequent, simple tasks get priority. Rare, complex ones get dropped. In the experiments, only models that were large enough learned tasks that made up just 0.25 percent of the training data.

The core of the paper is its explanation of why size helps. As long as frequent tasks aren't well-learned yet, they pull the model strongly in their direction at every training step, overwriting much of what the model picked up about rare tasks. Once a large model has mostly mastered the frequent tasks, that pull fades. The freed-up capacity goes to rare tasks, and learned signals are more likely to stick.

Small models rarely reach that point, according to the study. They fall into an "update-and-forget" loop. A rare example gets briefly learned, then largely erased by the next training steps on frequent tasks. When the next rare example shows up, the model starts over from scratch.

One experiment was designed to cleanly separate this effect. The total frequency of a rare task stays constant, but the gap between individual observations varies. The larger the gap, the more the signal decays in narrow models. Wide models hold onto it better between observations and build on it.

Real language models show the same pattern

To test the theory during pre-training, the team trained OLMo models ranging from 4 million to 4 billion parameters on up to 210 billion tokens from the Dolma corpus. They mixed two artificial tasks into the data, a number comparison and a modular addition, with frequencies ranging from about 1,000 instances per batch down to one instance every ten batches.

Only the larger OLMo models picked up the rare tasks by learning the rule behind them and applying it to new cases, rather than just memorizing individual examples.

This was especially clear with modular addition, where the researchers observed what's known as grokking. A model memorizes a task first, then suddenly clicks on the actual principle after more training. Only the bigger models hit that moment, and only when the task showed up often enough in the data.

A look inside the models tells the same story. In the one-billion-parameter model, every training step that included the rare task pushed clearly toward the right answer. In the 20-million-parameter model, that signal drowned in noise from everything else. Almost no real learning took place.

Memorization turns out to be a stepping stone

The study treats memorization as a prerequisite for generalization, rather than an unwanted side effect. A model needs to hold onto individual observations long enough for a broader pattern to take shape across many batches.

This offers a practical alternative to just making models bigger. Instead of scaling up the model, the frequency of a target task in the training data can be increased to anchor a specific skill, the research suggests.

There's more than one theory for why model size helps. In May, an MIT team tied scaling laws to model geometry, where models store more concepts through superposition than their dimensions should allow.

This new study starts from a different angle, focusing on what a model can actually learn from a given data mix during training. The older debate about whether abilities truly "emerge" in sudden jumps past a certain size, or whether that's partly a measurement artifact, is still playing out.