CloudOpsBench benchmark reveals weaknesses of AI agents in DevOps tasks on AWS
The new open-source benchmark CloudOpsBench v1.0 tests AI agents on 45 DevOps tasks in AWS. The best model (Claude Fable 5.1) succeeded in 60%, GPT-6 Astra in 53%; most errors stem from faulty reasoning, with a fifth having irreversible consequences.
A team led by Ang Chen of the University of Michigan has released CloudOpsBench v1.0, an open-source benchmark for evaluating AI agents on typical cloud DevOps tasks — from fixing services through removing resources to restricting access rights, deploying updates, and tearing down projects. The first version contains 45 tasks on the AWS platform and was created by experienced DevOps engineers. The tasks run identically in both a real AWS cloud and an emulator.
In testing, the Claude Fable 5.1 model achieved a success rate of 60 percent, while GPT-6 Astra achieved 53 percent — so even the best models fail on a number of tasks. According to the University of Michigan team, the main cause is not a lack of knowledge but reasoning: roughly 80 percent of failures arise because agents misjudge the scope of the task, fail to check all relevant services, or stop too early. Approximately 20 percent of errors are outright wrong actions, some with irreversible consequences for the infrastructure.
The project is available on GitHub under the Apache 2.0 license and also includes a detailed failure analysis. Extensions for other cloud platforms — GCP, Azure, and OCI — are planned.
Why it matters
The benchmark provides concrete, measurable data on where AI agents make mistakes in cloud operations — not due to a lack of knowledge, but in the ability to correctly judge the scope of a task and check all affected services. This is relevant for companies considering automating DevOps using AI agents, since according to the source, a fifth of errors have the potential to irreversibly damage infrastructure.
Relevant practical impact
What this means
For a business
Companies considering deploying AI agents in cloud/DevOps operations now have an independent benchmark showing that even the best models fail on 40-47% of tasks, with a fifth of errors having the potential to irreversibly damage infrastructure — this is an argument for introducing control mechanisms before granting agents full autonomy.
Risks and complianceCheck the original
Event sources
only one source so far · 1 publisher, 1 independent. We count feeds from the same owner only once.