AI agents are improving rapidly at autonomous software development and machine learning tasks, and, if recent trends hold, may match human researchers at challenging months-long research projects in under a decade. Some economic models predict that automation of AI research by AI agents could increase the pace of further progress dramatically, with many years of progress at the current rate being compressed into months. AI developers have identified this as a key capability to monitor and prepare for, since the national security implications and societal impacts of such rapid progress could be enormous.
While some progress is being made on measuring AI R&D capabilities, significant disagreement remains in interpreting the implications of these measurements. There are reasons to think that if AI systems become able to automate AI R&D, the resulting acceleration of progress would be economically transformative, but others are skeptical of such dramatic effects. To better understand how to interpret relevant evaluation results, and identify where experts have the greatest disagreements, we have worked with the Forecasting Research Institute to conduct an initial pilot survey of 8 AI forecasting domain experts1 (henceforth “experts”) and 10 “superforecasters”2, eliciting their predictions of future acceleration and societal impacts conditional on several hypothetical evaluation results.
This pilot study tasked experts and superforecasters with estimating the likelihood of (several variations on) two key events:
We believe these would be unprecedented outcomes if realized, with far-reaching consequences for the world. If we saw a three-fold increase in the pace of AI progress—when maintaining the current rate of progress is already demanding a potentially unsustainable pace of investment—it might indicate that AI systems are having truly transformative effects on R&D productivity, which could rapidly propagate to other parts of the economy. Because we are very uncertain about where this might lead and how it would change society, we picked a few robust and broad indicators of extremely rapid growth (unprecedented increases to global energy consumption) and instability (increase in risk of COVID-level disasters and unprecedented reductions in global energy consumption) as forecasting targets, to serve as proxies for the possibility of broad transformative changes to society.
We worked with FRI and other experts to operationalize these questions as carefully as we could for this pilot. This involved developing a definition of “3X acceleration of progress” in terms of acceleration of the pace of growth in “effective compute” (see below) that hopefully gets close to our intended meaning. Despite these efforts we still see indications that some participants may have interpreted these concepts differently in their rationales (see the full write-up for details on the operationalization and reviews of participant rationales).
We asked forecasters to predict:
What is the probability that, before 2029, there is a two-year period where the relative increase in effective compute seen in that two-year period is equal to or greater than the relative increase in effective compute seen in the 6 years between the beginning of 2018 and the beginning of 2024?
Intuitively, we are attempting to capture the idea that during some two year period before 2029, the amount of progress that happened in one year between 2018 and 2024 now happens every 4 months. “Effective compute” defines the progress made between two models as the compute scale-up factor that would have been needed to match the more powerful models capabilities without algorithmic innovations.
In more detail, we define the effective compute scale-up between an earlier weak model and a later more capable one as “what factor more compute would have been needed to reach the performance of the more powerful model using algorithms available at the time when the earlier model was developed, if a team had ~3 months to make adaptations to the increased amount of compute?”
This definition combines progress from larger compute spend and improvements to algorithms into one metric (if the later model used twice the compute and had algorithms that could achieve the same performance with half the compute cost, then this represents an overall “effective compute” scale-up of 4X).
A key challenge for defining this metric is defining what it means for two models developed with different algorithms to have “the same performance”, especially when comparing models of drastically different levels of capability where no individual benchmark can provide direct faithful comparisons. The previous literature has often used the perplexity of text prediction as the grounding performance measurement, but this doesn’t capture improved post-training approaches like reasoning models. For our purposes we instead ground this through a (theoretical) panel of experts using benchmarks and other investigations to determine the relative economic value of the models. See the full write-up from FRI for details.
Here we present some of the key results from the survey. For more details of the operationalizations of the forecasting questions and more detailed results see FRI’s write-up.
Experts and superforecasters both find a three-fold acceleration of effective compute scale-up by 2029 plausible but unlikely, with experts giving it a median 20% chance and superforecasters 8%. They both agree that a significant acceleration in physical compute scaleup looks unlikely, while post-training enhancements are most likely to accelerate rapidly.
We find that experts and superforecasters converge somewhat toward a higher likelihood of three-fold acceleration (to a median of 20.5% and 18% respectively, see Fig. 2) when conditioning on evaluations showing AI systems doing better than human researchers at open-ended month-long research projects. On the other hand, both experts and superforecasters agree that scores exceeding human performance on the day-long tasks in RE-bench would not be significantly informative, since many already expect the benchmark to be saturated by 2027.34 This provides some partial support for thinking that longer horizon lengths are predictive of the ability to automate AI R&D (though there may be other differences between the hypothetical research projects and RE-bench scores that forecasters view as important).5
Other conditionals see more disagreement, with experts believing that consensus among a panel of experts that AI can replace human researchers would make three-fold acceleration more likely than not, whereas some superforecasters are much more skeptical.
Experts and superforecasters also strongly disagree about the likelihood of all four extreme societal events that we asked about, with superforecasters giving a median risk estimate below 0.5% for all outcomes (compared to a median risk estimate of over 18% for all outcomes from the experts).
Conditioning on a three-fold (or even ten-fold) acceleration does not close this gap in the predictions of the two groups, though both groups forecast significantly higher probabilities of extreme societal events. The relative changes in risk were similar for the two groups, though given the superforecasters’ baseline level of risks were so low, their relative change was minimal in absolute terms.
Rationales show that experts generally took a threefold effective compute acceleration to imply artificial general intelligence (AGI), and predicted AGI causing massive change in energy consumption. Superforecasters are split on what the condition implies about AGI and skeptical about AGI being likely to cause energy consumption change or COVID-level disasters, due to technical and physical barriers and human intervention. For more about participant rationales, see the full write-up.
We found that experts and superforecasters broadly agree that dramatic acceleration of AI progress could result from automation of AI R&D, and that observing whether AI systems end up competitive with humans on month-long, open-ended research projects resolves most of their disagreement on this question. However, the two groups disagree strongly about how likely this is to lead to unprecedented societal impacts. We think it is worth more deeply exploring the source of this disagreement before expanding this pilot to a larger and more representative survey.
As this is a pilot study with only a small sample of participants, and relies on subjective judgment-based forecasts, these results should only be taken as suggestive evidence of the likely impacts of AI R&D capabilities and acceleration in effective compute scale-up. In particular, some of our aggregate statistics are fragile as participants disagree by several orders of magnitude on several questions.
Several of the “extreme societal events” considered are unprecedented, and may require intense societal preparation to handle appropriately, if they come to pass. Resolving disagreements in this area (by building longer-horizon AI R&D evaluations or tracking other acceleration metrics, and by more carefully modeling the potential impacts of such automation) is crucial for future work.
The full write-up from FRI covers these forecast results in much more detail, including:
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。