Researchers Warn Governments to Prepare for an "Intelligence Explosion"
2026-09-28 ยท Safety & Alignment
More than 20 researchers, including Geoffrey Hinton, Yoshua Bengio, OpenAI's chief scientist, and an Anthropic co-founder, published a paper warning that automating AI research could compress years of progress into months.
"What if automating AI R&D triggers an intelligence explosion?" was published on September 28 by the Cambridge Programme on AI Science and Policy. Its authors include Alan Chan, Andrew Barto, Jakub Pachocki, Geoffrey Hinton, Eric Horvitz, Yoshua Bengio, Dawn Song, and Jack Clark. They write that AI systems are on track to automate most AI research and development within a few years, and possibly all of it. If that triggers an intelligence explosion, they argue, capability growth could outpace society's ability to keep up, humanity could lose control over superhuman AI systems, and checks on power within and between states, companies, and branches of government could be severely eroded.
The authors urge governments to prepare before acceleration begins. Their proposals include auditors placed inside AI companies and the means to pause research projects at data centres. The authors describe a possible outcome, not one they say has begun, and co-authorship does not mean their employers endorse the paper.
For SIRDA, this is the center of its research focus, stated by pioneers of the field and senior figures at two frontier labs. Several of the paper's proposals correspond to powers in the Powers Map, including independent evaluation inside developers and emergency action to halt research, and its warning about eroded checks on power is the problem counter-agency addresses.