Analysis: AI regulation based on the text of system prompts is insufficient, say authors writing for Tech Policy Press

Authors Anna Neumann, Holli Sargeant and Jat Singh published an analysis on Tech Policy Press responding to proposals in the US to establish an independent regulator for advanced AI models that would work with the industry when assessing safety. The article asks what should serve as evidence that a model is safe and criticizes an approach that relies primarily on system prompts – text instructions that developers use to set the role, priorities and constraints of a model and that take precedence over user instructions. According to the authors, these instructions are readable and can be documented, but their existence alone says nothing about how the model will actually behave.

According to the authors, system prompts provide only weak assurances about model behavior because the effect of a given instruction varies depending on the task, context, wording and presence of other safety measures – even within the same model family. They cite two cases as evidence of instability: the company xAI identified and reversed an unauthorized change to the system prompt of the model Grok that led to false claims about “white genocide”; the company OpenAI modified the system prompt of the model GPT-4o after the model was criticized for excessively sycophantic behavior. In both cases, according to the authors, further adjustments were necessary after the problem occurred.

The authors highlight two risks of regulation based on prompt text. First, there is a risk of certifying developer intent rather than actual system performance – a company can submit well-worded instructions that meet formal criteria without the model following them in practice. Second, assessing only the prompt text may create room for misleading or deliberately deceptive compliance, where the prompt formally aligns with regulatory expectations but actual model behavior is shaped by other means, such as fine-tuning or hidden code in the prompt.

The authors instead recommend assessing system behavior in the context of its deployment, rather than just the prompt text. They argue that documentation of system prompts can be useful if it shows how prompts have evolved in response to observed outputs – including versions, changelogs, reasons for changes, deployment settings and approving parties – and if it is supplemented by risk evaluations, adversarial testing and audits across different contexts and languages, repeated after every change to the prompt or deployment environment. The remainder of the article was not available.