AI Experts Warn of Potential 'Intelligence Explosion'

A group of prominent AI researchers and executives has sounded the alarm about the potential for an ‘intelligence explosion’, a scenario in which artificial intelligence (AI) systems rapidly accelerate their own development, potentially leading to unforeseen consequences. The warning comes from two of the pioneers of modern AI, Geoffrey Hinton and Yoshua Bengio, along with other senior figures at OpenAI and Anthropic.

The report, titled ‘What if automating AI R&D triggers an intelligence explosion?’, was co-authored by over 20 experts in the field, including Hinton and Bengio. It urges governments to take immediate action to prepare for this potential development, which they describe as a ‘dramatic AI-driven acceleration of AI progress’. The authors are concerned that once AIs reach expert-level capabilities at research and development (R&D), a single developer could potentially run a workforce equivalent to millions of top human researchers.

The report highlights the possibility of AIs being able to improve themselves without human intervention, known as ‘recursive self-improvement’ or automated AI R&D. This process is already underway, with AI systems contributing to their own development and deployment. The authors warn that if this trend continues unchecked, it could lead to an intelligence explosion, which would compress years of progress into months or less.

The potential consequences of such a scenario are far-reaching and potentially catastrophic. An intelligence explosion could threaten human control over AI systems, erode checks on power within and between states, companies, and branches of government, and enable biological and cyber threats to develop faster than the measures needed to counter them. Additionally, humans may lose the opportunity to control systems as they become less involved in R&D.

The report acknowledges that these impacts are uncertain and that a technological breakthrough could take time to implement due to regulatory requirements or supply chain issues. However, it also notes that AI systems could accelerate the speed at which risks are mitigated. The authors emphasize that preparing for this scenario should be an urgent priority, including at the highest levels of government leadership.

The report recommends three policy priorities: requiring transparent progress reports on AI-related R&D, embedding independent auditors in companies to monitor their activities; finding ways to constrain breakneck AI development; and preparing to adapt to an intelligence explosion. The authors also suggest that governments should consider policies such as limiting how fast an AI can improve within a certain time period or working with datacentres to enable pausing certain R&D projects.

Anthropic, one of the companies involved in developing advanced AI systems, has already agreed to have its models assessed by independent evaluators. The company’s own research suggests that AI now produces 80% of its own code, while OpenAI uses autonomous agents for tasks such as training new models. Preliminary evidence from these developments indicates that a software-driven intelligence explosion is possible.

The report warns that an intelligence explosion could lead to the extremely rapid development of highly capable or superhuman AI systems. While this may bring about medical breakthroughs and technological leaps, it also poses significant risks. The authors emphasize that governments should consider emergency response plans for various scenarios that could emerge from an intelligence explosion.