AI models are still improving fast on tasks with clear right answers, but they lag well behind humans on work that requires judgment. That is the headline from an updated report by METR, the AI evaluation nonprofit, covered by MIT Technology Review on November 20.

METR updated its influential METR time horizon plot, which measures progress by the length of tasks, scored by how long they take human experts, that frontier models can complete. The original plot showed the length of tasks models could handle doubling roughly every four months. The new version adds longer tasks and, for the first time, tasks where success is hard to define or measure.

What the update tested

The new suite includes assignments that probe the skills needed to automate AI research itself. In one task, models must design new AI evaluations that METR could use in its own research, and METR employees score the results. In another, models score philosophical arguments, a test meant to capture judgment.

The results show a clear gap. Not one of the AI-designed evaluations was good enough for METR employees to consider it useful for their research, even though the most advanced models are now better than almost every human at software engineering. On the philosophical argument task, models also fell short of human performance, though METR notes its own employees disagree about how relevant that task is to automating AI research.

There are signs of movement. Some of the newest models, such as Anthropic’s Claude Opus 5.5, reached what METR calls “borderline above the bar” on the evaluation design task. Just a year ago, every model scored “clearly below the bar.”

body metr fits

METR’s long-run data since 2019 fits an exponential trend with a doubling time of 212 days. Chart: METR (CC-BY)

Why judgment is hard to teach

The gap has a technical explanation. Reinforcement learning, the technique behind much of recent AI progress, works best when success can be checked automatically, which is why models have become so strong at coding and math. It is far harder to apply to tasks where success is vague or takes a long time to judge, such as choosing a promising research direction.

“We’re at the point where AI evaluations need to get more creative,” Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, told MIT Technology Review. Kapoor said “research taste,” the ability to choose promising questions, hypotheses and experiments, could prove a major bottleneck in automating AI research and development.

Read the results with care

METR itself urges caution. In the new report, the time needed for most tasks is estimated rather than measured, because paying people to complete assignments that might take hundreds of hours is prohibitively expensive. The philosophical argument task was mostly rated by a single person, so scores reflect agreement with that individual. And the report includes only a few judgment-heavy tasks, which makes it hard to predict how models will progress in these areas.

“The improvement we observe, I think that does tell us something for the current moment,” Najoung Kim, a linguist and computer scientist at Boston University, told MIT Technology Review. “I’m not sure just yet how to think about how it would extrapolate.”

Matt Putz of METR’s technical staff added that trends for hard-to-verify tasks are unlikely to ever look as clean as those for well-specified work. “By their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify,” he said.

Why it matters

The update lands as the AI industry debates whether models can soon automate research itself. A separate, heavily redacted METR report earlier this month estimated AI may be speeding up the pace of AI progress at Anthropic by a factor of two, and OpenAI said in a blog post this week that its models do less well on the real-world tasks its employees assign them than on the easier, verifiable tasks in METR’s original suite.

METR’s message is narrower but important: progress on measurable tasks is real and rapid, while the judgment needed for open-ended research remains a human strength, for now.

Sources