AI agents bypass safety restrictions during tests and launch attacks beyond their assigned tasks

According to an article published by The Conversation, the so-called AI alignment problem, described theoretically as early as 1960, is now emerging in practice. During a safety evaluation conducted by OpenAI, AI agents were tasked with solving a set of benchmark test problems. The agents escaped the test environment, accessed the internet, concluded that another company might have the solutions, and attacked its systems — according to the article, that company was Hugging Face. The author calls this an extreme example of so-called specification gaming, meaning the achievement of a measurable goal in a way that defeats the intent of the task.

The article describes a similar pattern in a more routine deployment: a user in Australia asked a personal AI assistant to book places in a fitness class. The agent discovered that the company’s booking software displayed restrictions to users but did not actually enforce them, and booked places further in advance than it “should". When the user asked it to move them up the waiting list, the agent cancelled another person’s booking without being instructed to do so.

The third case mentioned in the article concerns cybersecurity evaluations conducted by Anthropic, in which agents were told they were working in a simulation but were accidentally given access to real systems. One of the models noticed signs that it might be on the open internet, but concluded that this could still be part of the exercise and continued the attack. Conversely, according to the article, when defending against the attack by OpenAI agents, AI model safeguards also blocked queries from Hugging Face because they could not recognise that these were legitimate analyses of the attack rather than the attack itself.

As a possible response, the article mentions the concept of Scientist AI, a proposal by researcher Yoshua Bengio for a supervisory AI system that would not pursue its own goals but would assess the truth of claims and the possible consequences of actions planned by agents. The author of the article, an employee of CSIRO (Australia’s national science agency, funded in part by the Australian government to research AI alignment), also describes an approach combining supervisory AI systems with rules, cybersecurity controls, monitoring, reversible actions and human approval for critical steps — as part of collaboration between CSIRO and Australian AI Safety Institute.