Criticism of AI labs for determining AI safety thresholds internally instead of through a scientific process
The AI Now Institute published an expert's opinion criticizing major AI labs for setting AI safety thresholds themselves instead of following established risk management methods, such as those used in nuclear engineering.
An article published by the AI Now Institute contains the opinion of an expert who criticizes the comparison of AI to nuclear technologies made by major AI labs. According to the author, labs use this comparison while at the same time rejecting the standard risk management methods established in nuclear engineering after the Manhattan Project, whose core is the democratic question "how safe is safe enough?"
The author claims that major AI labs "co-opt" safety terminology and thresholds through their own interpretation, according to which they themselves should decide the definition of AI safety, rather than through a scientific or democratic process. The author describes the debate on AI safety as a shift from a scientific approach to argumentation that he compares to "religious" reasoning and calls anti-democratic.
The available text is a brief excerpt from a longer article based on interviews with ten experts; the source does not provide details on the other opinions and specific recommendations. See the source article for details.
Why it matters
The debate shows there is no consensus on who should determine, and by what process, what level of AI system safety is sufficient — whether labs themselves should decide this, or whether it should be an independent/scientific process similar to established risk management methods in other industries. This affects how standards and regulations used to assess the safety of AI products will be formed in the future.
Relevant practical impact
What this means
For a business
According to critics, companies developing or deploying AI systems may find that internally set safety thresholds will not be accepted as sufficient if, over time, a requirement prevails for external, scientifically grounded risk management methods comparable to those in other regulated industries.
Risks and complianceCheck the original
Event sources
only one source so far · 1 publisher, 1 independent. We count feeds from the same owner only once.