In the last year or two, the most important trend in modern AI came to an end. The scaling-up of computational resources used to train ever-larger AI models through next-token prediction (pre-training) stalled out. Since late 2024, we’ve seen a new trend of using reinforcement learning (RL) in the second stage of training (post-training). Through RL, the AI models learn to do superior chain-of-thought reasoning about the problem they are being asked to solve.

This new era involves scaling up two kinds of compute:

1. the amount of compute used in RL post-training

2. the amount of compute used every time the model answers a question

Industry insiders are excited about the first new kind of scaling, because the amount of compute needed for RL post-training started off being small compared to the tremendous amounts already used in next-token prediction pre-training. Thus, one could scale the RL post-training up by a factor of 10 or 100 before even doubling the total compute used to train the model.

But the second new kind of scaling is a problem. Major AI companies were already starting to spend more compute serving their models to customers than in the training phase. So if it costs a factor of 10 or 100 times as much to answer each question, this really does affect their bottom line. And unlike training costs, these costs can’t be made up in volume. This kind of scaling is known as inference-scaling (since it is scaling the compute used in the output, or ‘inference’, stage).

It is critical to find out how much of the benefits are coming from each kind of scaling. If further improvements in AI capabilities are mainly going to come from inference-scaling, this would have many implications for the trajectory of AI, including for AI companies, AI governance, and AI risk.

In past writings, I’ve mainly focused on the inference-scaling, treating RL as primarily enabling larger and larger amounts of inference to be productively used to answer a question — improving capabilities, but at a steep cost every time the AI system is deployed. However, it is probably more common in the AI discourse for people focus on the training costs of RL — treating it as a relatively cheap way to train a smarter model. In support of this view, people often point to the fact that even when reasoning models have thinking turned off, they still have superior performance on maths or science benchmarks compared with the base model that they were trained from.

For example, Anson Ho and Arden Berg from Epoch AI focus on this angle in their attempt to quantify the algorithmic improvement from reasoning. However, they ultimately suggest that about half the benefit comes from the RL training (the shift from the base model to the no-thinking reasoning model) and half comes from scaling up the number of tokens that are used in reasoning. But as they note, despite producing zero hidden ‘reasoning tokens’ the no-thinking version of the reasoning model still had a chain of thought that was about twice as long as the base model. Ho and Berg address this qualitatively, as creating some uncertainty about their claim that half the gains come from each source.

But it is possible to do better — we can use this data as part of the analysis for finding out what fraction of performance increase is due to each type of scaling.

While it would be easy for people inside the leading AI companies to perform this analysis, it is more challenging with open data. A key thing is needing to have access to both a reasoning model and to the base model it started training from. In all my analysis I’ll follow Ho and Berg in using Anthropic’s Sonnet 3.7 as the reasoning model and assuming that Anthropic’s October 2024 release of Sonnet 3.5 (commonly called Sonnet 3.6 by the community) was its base model. (If that’s wrong and Sonnet 3.7’s base model incorporated improvements beyond ‘Sonnet 3.6’, that would only help my main conclusion.)

As always, it is instructive to first look at the raw scatter plot. Using the same Epoch AI dataset, here is a chart showing how benchmark performance (for MATH level 5) varies with the number of output tokens:

We see similar results on the other two benchmarks I analysed. There was a somewhat better showing from the RL boost (at a fixed compute budget) for GPQA Diamond, but most of the gains were still coming from inference-scaling, requiring more than 10x as much compute each time the model is used:

So that’s the simple case that most of the gains from RL-scaling are driven by the inference-scaling it opens up.

One obvious piece of further work would be to extend this analysis to models from other companies.

And an even more important piece of further work would be to examine the changes between generations of reasoning models. In particular, how would scaling up the RL post-training by a factor of 10 affect the inference-scaling trend-line? It would presumably move it up or left and might change its slope (though the ARC-AGI leaderboard suggests the slope usually stays the same). This extra RL training would presumably also enable longer chains of thought and thus more inference-scaling. The relationship between these two kinds of changes would determine whether further scaling of RL changes the importance of the RL boost compared to that of inference-scaling.

Alas, it is challenging for people outside the major labs to get the data to check this as simply using data from publicly released models will include gains from all other non-RL sources, inflating the apparent RL boost.

For now, I think the graphs we’ve seen here make a strong case that most of the gains from using RL to train reasoning models have been coming from scaling-up the number of tokens used to solve the task. This creates a flow of costs that have to be paid every time you use the model, substantially changing the business model, the risk profile, and the policy response.