Regulating superintelligence: drawing inspiration from risk-management theories in the nuclear and aviation industries
British MP Darren Jones proposed a multinational treaty for the safe development of superintelligence. A commentary in The Conversation applies theories from the nuclear and aviation industries (Normal Accident Theory, High Reliability Theory) to the regulation of AI agents in complex systems.
British MP Darren Jones, a former chief secretary to Prime Minister Keir Starmer, called for a new multinational treaty governing the safe development and use of superintelligence (AI that would substantially surpass humans at most cognitive tasks). According to him, thanks to its reputation in AI safety, Britain could serve as a suitable intermediary for such cooperation. The article's author, a business scholar, builds on this call by considering whether governance and regulatory systems can keep pace with AI development while also ensuring its safety.
The article applies two theories from high-risk industries (nuclear power, aviation) to the question of AI regulation. Normal Accident Theory (Charles Perrow) describes how, in complex, interconnected systems, accidents can arise from the interaction of multiple components in ways that are hard to predict – the 1978 partial core meltdown at the Three Mile Island nuclear power plant is cited as an example. According to the author, the risk increases when AI agents become involved in such systems (financial systems, development tools, scientific databases), as their behavior can be unpredictable even when individual components meet quality standards.
Against this, the article sets High Reliability Theory (Todd LaPorte), according to which organizations operating extremely complex technologies with a low failure rate (the electrical grid, space flight, bioengineering) start from the assumption that they are not safe. These organizations monitor signs of problems, do not dismiss anomalies, learn from near-misses, build in redundancy (backup systems, kill switches), regularly drill for failures, and have independent oversight. The author proposes reframing the AI regulation question from "how to regulate AI" to "what would a highly reliable organization look like if AI were built into it everywhere" – in other words, that safety cannot be reduced to a one-off technical test of a model before deployment.
The source text is incomplete; the rest of the article was not available.
Why it matters
When AI agents become involved in complex corporate systems (financial, development, laboratory), there is a risk of failures arising from unpredictable interaction between components, not from the error of a single one of them – a one-off technical test of a model before deployment does not cover this risk. The article offers a concrete framework from high-risk industries (nuclear power, aviation) for how to reduce such risk: building a culture that assumes danger, does not dismiss anomalies, and drills for failures.
Relevant practical impact
What this means
For a business
Companies deploying AI agents into complex systems (financial, development, laboratory) face the risk of cascading failures that arise not from the error of a single component, but from unpredictable interaction between components.
Risks and complianceCheck the original
Event sources
only one source so far · 1 publisher, 1 independent. We count feeds from the same owner only once.