Core Assumptions About AGI Development

The report assumes that frontier AI systems will continue to be developed within the current paradigm for the foreseeable future. This paradigm is characterized by scaling computation and data, learning and search as central mechanisms, and algorithmic innovations for efficiency.

Long-run AI capability improvements to date have been driven by large-scale increases in compute, data, and algorithmic efficiency. Studies show a smooth power law relationship between these inputs and performance outcomes. For example, researchers have observed that the total compute used for the largest training runs has grown by an annual average of approximately 4× between 2010 and 2024.

This assumption motivates the report's focus on an "anytime" approach to AGI safety that can be applied at any point. It also highlights the need for oversight signals that can accurately assess whether a given AI action is good or bad.

The report assumes that AI capabilities will not cease to advance once they achieve parity with the most capable humans. There is no "human ceiling" that sets an upper limit for AI capability.

Superhuman performance has already been demonstrated in several domains:

• Chess, where AI systems reached an Elo rating of 3643 in 2024, compared to the highest human rating of 2882
• Go, which was considered much more difficult than chess, yet was mastered by AlphaGo
• AlphaFold's superhuman performance in predicting protein structures

The authors observe no principled arguments for why AI capability would necessarily stop at human-level, and many examples suggest that AI can use fundamentally different approaches than humans.

The safety approach described must leverage new AI capabilities as they become available. Initially, this will involve augmenting human work with AI assistance, but eventually most cognitive labor relevant to AI safety may need to be performed by AI to keep pace with advancing capabilities.

The timeline for the development of powerful AI systems remains highly uncertain. The report considers a broad range of timelines to be plausible, including relatively short timelines where Exceptional AGI (Level 4) might be developed before the end of the current decade.

Existing AI forecasts support a broad spectrum of timelines. For instance:

• Expert surveys conducted in 2023 estimated a 50% chance of "High-level machine intelligence" by 2047
• Compute-centric approaches have predicted human-level AI within the next decade
• Historical forecasting challenges suggest caution in placing too much confidence in any single prediction

Given this uncertainty, the report prioritizes safety approaches that can be integrated into current frontier AI development and applied whenever necessary, rather than more foundational explorations that might take many years to bear fruit.

The report assumes that as AI systems automate scientific research and development, an acceleration phase could be entered where growth becomes faster. Initial automation of R&D would enable the development of increasingly capable AI research systems, creating a positive feedback loop.

The economics literature supports a diversity of perspectives on accelerating growth, including some that predict extremely rapid growth. It has been estimated that returns to software R&D could be sufficient to produce hyperbolic growth, though the evidence is not conclusive. The possibility of rapidly accelerating growth also finds substantial support among AI researchers.

Such acceleration could drastically increase the pace of progress, leaving very little calendar time to notice and react to issues. The safety approach described must also be accelerated through AI assistance to ensure developers retain the ability to address novel risks as they arise.

The report assumes there will not be large discontinuous jumps in general AI capabilities given continuous increases in the inputs to those capabilities (compute and R&D effort). The report does not make any such assumption about the rate of AI progress with respect to calendar time.

Taking an outside view, large discontinuous jumps in highly optimized domains are rare. Empirical evidence suggests that general capabilities (as measured by broad benchmarks) tend not to show large, sudden jumps. While the phenomenon of "emergent abilities" has been documented, most cases can be explained as measurement artifacts, and dramatic gains that would enable severe harm are extremely rare.

This enables developers to iteratively and empirically test the approach and detect flawed assumptions as capabilities improve. The technical approach described doesn't need to be robust to arbitrarily capable AI systems, but can focus on foreseeable capability improvements.