OpenAI chief scientist Jakub Pachocki published the lab’s internal pretraining scores for the first time on November 22, and used them to propose something no frontier lab has asked for before: shared limits on AI development that every major lab would agree not to cross. The post, titled “Perplexity and Intelligence,” argues that a single training metric can track the raw intelligence of AI models and should anchor an industry-wide agreement on when to slow down.
Pachocki’s case starts with a simple observation. Modern AI development, he writes, mostly comes down to training a compressor on a huge corpus of data, then drawing out the capabilities hidden inside it. The best measure of that compressor, in his view, is perplexity, or cross-entropy loss: how well a model predicts the next token. Lower is better. OpenAI has tracked the number on a held-out dataset drawn from its internal codebase since GPT-3.5, and it is now sharing the figures publicly for the first time.
What the Numbers Show
The chart in the post traces a steady fall in log-perplexity, with each step linked to a jump in capability. GPT-3.5, at about 0.99, could hold fluent conversation and became the basis for the original ChatGPT. GPT-4, at about 0.91, became proficient at chain-of-thought reasoning. Once losses dropped below 0.8, Pachocki writes, OpenAI saw “the emergence of transformative capabilities in fields like cybersecurity and mathematics.”

OpenAI’s log-perplexity figures, from GPT-3.5 to the 0.68-0.70 window it expects next. Graphic: GPTOnline, based on OpenAI data.
The most striking data point sits at about 0.76. Pachocki writes that once OpenAI upgraded to a model at that level, “it started producing groundbreaking results” on open mathematics problems the lab had been posing to its models for more than a year. “Seeing the resolutions to questions we’ve long thought about coming out of the model one after another in a single Slack thread felt deeply surreal,” he wrote. The chart ends with an unreleased internal model at about 0.73, below the latest public model, GPT-6.2 Astra, at about 0.76.
The metric has nothing to do with Perplexity, the AI search startup, despite the shared name. It is a decades-old training measurement, and Pachocki argues that its very age is the point: while benchmarks saturate and change, perplexity has kept tracking general intelligence across every generation of models.
The 2027 Warning
The post’s urgency comes from what Pachocki thinks happens next. AI is increasingly contributing to its own development, and he argues the biggest step change may come from AI doing algorithmic research on pretraining itself. Drawing on the math experience, he writes: “as soon as 2027 we could conceivably see a similar avalanche of fundamental pretraining advancements.”
That acceleration, he argues, is likely to outrun the tools used to watch it. “While I expect AI will also significantly contribute to progress on alignment and monitoring, I think that progress is unlikely to be fast enough to allow the full potential acceleration in pretraining,” he wrote. He also expects further transformative capabilities as models enter the 0.68 to 0.70 log-perplexity window, while admitting, “I don’t know what those capabilities will be, and we may not be testing for them today.”
What Pachocki Is Proposing
Pachocki proposes three things: shared norms for evaluating and red-teaming highly capable models before first use, shared transparency standards, and common limits based on perplexity and other measurements. Developers would commit not to cross a first agreed limit until monitoring reaches a concrete milestone, “such as the development of a successor to CoT monitors,” meaning the chain-of-thought monitors used to watch models reason. He writes: “I further propose common limits, based on perplexity and other measurements, that developers are committed to not crossing until our ability to understand and monitor AI behavior achieves a certain concrete milestone.”
OpenAI CEO Sam Altman amplified the post on X, calling it “the most important thing OpenAI has published in awhile” and saying he hopes the field can agree to common limits.

OpenAI CEO Sam Altman (left) with SoftBank CEO Masayoshi Son in Tokyo in February 2025. Photo: Office of the Prime Minister of Japan, via Wikimedia Commons (CC BY 4.0).
The Limits of the Metric
Pachocki is candid about the problems with his own proposal. Different labs train on different data and measure on different held-out sets, so perplexity numbers are not cleanly comparable across companies today. Perplexity also misses gains from chain-of-thought reasoning, which sharply improves capabilities without reducing the score. And nobody knows the floor: the inherent entropy of the data sets a limit on how low the number can go, and progress will look slower as models approach it, even as each small gain means a larger leap in intelligence.
Why It Matters
The significance is less the metric than the messenger. OpenAI is disclosing internal numbers it once treated as competitively sensitive, with Pachocki writing that the concern about helping rivals “is outweighed by the urgency of establishing shared safety measures across the industry.” The post follows his September essay on AI as an “alien mind,” and it lands as governments and labs debate how to govern systems that keep surprising their own builders. Whether rivals publish comparable numbers will decide if perplexity becomes a real shared yardstick or stays an OpenAI-only lens. For now, the lab that builds the most widely used AI models is publicly asking the field to agree on a speed limit, and warning that the next leap could arrive as soon as 2027.

