Benchmark
A standardised test used to compare model performance.
A benchmark is a standardised test suite that measures how well a model performs on specific tasks. Examples include MMLU (general knowledge), HumanEval (code generation), and GPQA (graduate-level reasoning). Benchmarks give the community a common yardstick for comparing models from different labs.
Why they matter: Benchmarks make a comparison explicit. To interpret a score, check the test version, task selection, model, tools, time or token budget, and scoring rules. A published number is not automatically an independently reproduced result.
The limits: Benchmarks only measure what they test. A model can score well on MMLU but struggle with real-world tasks that require tool use or multi-step reasoning. There is also a contamination problem: if benchmark questions leak into training data, scores inflate without genuine capability improvement.
Different tests answer different questions: ARC-AGI-3 tests adaptation in unfamiliar interactive environments. That is useful evidence about learning and planning, but it does not test every cognitive domain or prove that a business workflow operates reliably.
Before applying a score to real work: Define what counts as success and which cases enter the denominator. Keep failures and manual rescues among the eligible cases, and report excluded work separately. Time spent acting, confidence in an answer, merged changes and successful production outcomes are different measures. Anthropic’s autonomy study explains why individual tool calls do not reveal complete workflow outcomes.
Our Horizon predictions use explicit resolution criteria, separate from benchmark scores. A stronger score can inform a forecast without satisfying its milestone.
Editorial review, 18 September 2026: clarified reproducibility and the limits of transferring benchmark results to production. The original publication date is preserved.