Safety safeguards (guardrails) in AI models complicate work for offensive cybersecurity researchers

TechCrunch describes how safety restrictions (guardrails) in AI models, introduced to prevent misuse for cyberattacks, also complicate the work of legitimate researchers in offensive security (finding and testing vulnerabilities). In June, the US government imposed export restrictions on the Mythos and Fable models from Anthropic, partly because of a report claiming that guardrails preventing the misuse of models to build and launch malicious cyberattacks can be bypassed. According to the text, Anthropic itself repeatedly presented these models as tools posing particular risks, accessible only to vetted users. The export restrictions on Fable 5 and Mythos 5 have since been lifted: Fable 5 returned to standard access on 1 July, while Mythos 5 was made available again only to vetted US organizations as part of a government review.

Both Anthropic and OpenAI offer vetting programs for security researchers (OpenAI Trusted Access for Cyber, Anthropic Cyber Verification Program), whose approval allows researchers to work with models with less restrictive safeguards. According to researchers quoted in the article, however, even these programs do not work reliably.

Several offensive security experts contacted described problems: Mark Dowd (a longtime trader in zero-day vulnerabilities for governments) criticizes individual companies for arbitrarily deciding what is “safe” in security. Chris Anley from NCC Group points out that asking for an attempt to exploit a bug is a key step in confirming that it is a real vulnerability — if a guardrail rejects such a request, it harms defenders, because the same tool serves both defense and attack. Paolo Stagno from CrowdFense said that his company uses frontier models only for reverse engineering, while relying on locally run open-source models to find vulnerabilities and build exploits, to prevent sensitive vulnerability data from leaking into cloud models. Giuseppe Cali, by contrast, said that guardrails do not complicate his work because he does not use AI for offensive work at all. An anonymous researcher at a smartphone component manufacturer explained that his employer is not part of the Anthropic CVP program, making the tools almost unusable for finding vulnerabilities. Chris Thompson (RemoteThreat, Offensive AI Con) claims that guardrail behavior is inconsistent and changes from day to day even within vetted programs, so researchers spend time “negotiating” with the model instead of doing security work itself.

The source article also mentions that these restrictions are leading researchers to turn to Chinese open-source models, but no further details on this are available. You can find details in the source article.