Skip to content
worth noting Security

OpenAI published internal perplexity measurements of its models and proposed them as an industry safety standard

only one source so far

OpenAI has published the development of perplexity (cross-entropy loss) of its models from GPT-3.5 (0.99) through GPT-4 (0.91) to newer models (below 0.8) and proposes this metric as a shared safety standard for the entire AI industry.

OpenAI has published internal measurements of perplexity (expressed as cross-entropy loss) for its models from GPT-3.5 to the present, and argues for introducing this metric as a shared safety standard for the entire AI industry. According to the text, perplexity measures how well a model predicts the next token in text – a lower value means better prediction. According to OpenAI, the log-perplexity value dropped from approximately 0.99 for GPT-3.5 to approximately 0.91 for GPT-4, and for newer models it fell below 0.8.

The author of the text claims that specific new model capabilities have historically appeared as this metric decreased: GPT-3.5 enabled fluent conversation and formed the basis of ChatGPT, GPT-4 was able to productively use chain-of-thought reasoning, and models below the 0.8 threshold, according to OpenAI, exhibited capabilities in areas such as cybersecurity and mathematics. OpenAI publishes these figures even though, according to the author, this gives competitors insight into its position, because it considers this more important for creating a shared scientific basis for safety measures across the industry.

The text further contains the author's claim that a model with log-perplexity around 0.76 began contributing to solving open mathematical problems, and speculates that a similar acceleration could also occur in algorithmic pretraining research, possibly as early as 2027 – however, this is the author's speculation, not a confirmed fact. The rest of the article is not available; for details, see the source article.

What changed

Why it matters

If perplexity (or the corresponding cross-entropy loss) becomes a recognized indicator across model developers, it could serve as an early signal of when models are likely to reach riskier capabilities – according to OpenAI, for example in cybersecurity or mathematics. This would allow both companies and individuals to better estimate when it is necessary to tighten safety measures or reconsider how models are used, instead of relying solely on benchmarks that quickly become outdated.

Two audiences, two different impacts

What this means

01

For individuals

For an individual user or developer, this is an informational signal that can be used to estimate when new capabilities (e.g. in coding or mathematics) will appear in models, without requiring any immediate action.

What to do Track the publication of similar metrics as a reference point for estimating when further leaps in model capabilities will occur.
More practical updates →
02

For a business

Companies deploying models from OpenAI can use perplexity as a supplementary signal for estimating when models will reach riskier capabilities (e.g. in cybersecurity or mathematics), and time the review of safety measures accordingly.

Risks and compliance
What to decide Track future publications of perplexity metrics and related risk assessments from OpenAI as a basis for assessing risks when deploying new models.
More business impacts →
AI safety GPT-3.5 GPT-4 OpenAI Perplexity standards

Check the original

Event sources

only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.

1
OpenAI News primary source · first detected Perplexity and Intelligence