OpenAI published internal perplexity measurements of its models and proposed them as an industry safety standard
OpenAI has published the development of perplexity (cross-entropy loss) of its models from GPT-3.5 (0.99) through GPT-4 (0.91) to newer models (below 0.8) and proposes this metric as a shared safety standard for the entire AI industry.
OpenAI has published internal measurements of perplexity (expressed as cross-entropy loss) for its models from GPT-3.5 to the present, and argues for introducing this metric as a shared safety standard for the entire AI industry. According to the text, perplexity measures how well a model predicts the next token in text – a lower value means better prediction. According to OpenAI, the log-perplexity value dropped from approximately 0.99 for GPT-3.5 to approximately 0.91 for GPT-4, and for newer models it fell below 0.8.
The author of the text claims that specific new model capabilities have historically appeared as this metric decreased: GPT-3.5 enabled fluent conversation and formed the basis of ChatGPT, GPT-4 was able to productively use chain-of-thought reasoning, and models below the 0.8 threshold, according to OpenAI, exhibited capabilities in areas such as cybersecurity and mathematics. OpenAI publishes these figures even though, according to the author, this gives competitors insight into its position, because it considers this more important for creating a shared scientific basis for safety measures across the industry.
The text further contains the author's claim that a model with log-perplexity around 0.76 began contributing to solving open mathematical problems, and speculates that a similar acceleration could also occur in algorithmic pretraining research, possibly as early as 2027 – however, this is the author's speculation, not a confirmed fact. The rest of the article is not available; for details, see the source article.
Why it matters
If perplexity (or the corresponding cross-entropy loss) becomes a recognized indicator across model developers, it could serve as an early signal of when models are likely to reach riskier capabilities – according to OpenAI, for example in cybersecurity or mathematics. This would allow both companies and individuals to better estimate when it is necessary to tighten safety measures or reconsider how models are used, instead of relying solely on benchmarks that quickly become outdated.
Two audiences, two different impacts
What this means
For individuals
For an individual user or developer, this is an informational signal that can be used to estimate when new capabilities (e.g. in coding or mathematics) will appear in models, without requiring any immediate action.
For a business
Companies deploying models from OpenAI can use perplexity as a supplementary signal for estimating when models will reach riskier capabilities (e.g. in cybersecurity or mathematics), and time the review of safety measures accordingly.
Risks and complianceCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.