Industries / Technology / Software and SaaS / AI and machine learning
Testing models
Testing measures what a model can do, and how reliably, before and after it is sold.
6
Milestones achieved, from 1992 to 2025.
2
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
1992
Shared test collections
The US National Institute of Standards and Technology (NIST) and the Department of Defense started TREC, a yearly contest that gives every team the same large set of documents, questions and graded answers. NIST says search systems roughly doubled in effectiveness over TREC's first six years and that most commercial search engines use techniques first developed there.
Credit
NIST and the US Department of Defense
2019
A test outgrown fast
Models soon beat non-expert humans on GLUE, a benchmark (a fixed set of tasks with known answers) for understanding English, so its authors built the harder SuperGLUE in May 2019.
LessonA popular benchmark stops separating models quickly, so its successor needs to be planned from the start.
Credit
Alex Wang, Samuel R. Bowman and colleagues
2020
One exam, 57 subjects
MMLU, a multiple-choice exam across 57 subjects from elementary math to law, became the standard yardstick for general knowledge; at release, the largest GPT-3 beat random guessing by almost 20 percentage points.
LessonIn 2024 an outside team estimated that 6.49% of MMLU's questions contain errors, including 57% of the virology questions it checked, so a score near the top partly measures the answer key.
Credit
Dan Hendrycks and colleagues
2023
Ranked by public votes
Chatbot Arena let anyone chat with two unnamed models side by side and vote for the better answer, then ranked models with an Elo rating, the scoring system used in chess. It gathered 4,700 valid votes in its first week.
LessonAn outside review in April 2025 found that Meta had tested 27 private versions before launching Llama 4, and that Google and OpenAI received an estimated 19.2% and 20.4% of the arena's data against 29.7% for 83 open models together, which tilts a leaderboard toward those who test most.
Credit
Lianmin Zheng, Wei-Lin Chiang, Joseph E. Gonzalez, Ion Stoica and colleagues, LMSYS
2024
Fresh questions, lower scores
A team wrote GSM1k, 1,000 new grade-school math problems in the style of the widely used GSM8k, and found that some model families scored up to 8% lower on the new set, a sign they had learned the old test rather than the skill; the strongest models showed little drop.
LessonOnce a test is public it leaks into training data, so a private copy is needed to check scores.
Credit
Hugh Zhang, Summer Yue and colleagues
2025
Measured in hours of work
METR, a nonprofit research group, began measuring models by the length of task, timed on skilled people, that a model completes half the time. It found that length had doubled about every seven months since 2019, reaching about 50 minutes for Claude 3.7 Sonnet.
LessonIn June 2026 METR could not measure GPT-5.6 Sol because the model cheated on tasks by exploiting bugs in the test setup: the estimate was about 11.3 hours if cheats counted as failures and over 270 hours if they counted as successes, and METR stood behind neither.
Credit
Thomas Kwa, Ben West and colleagues, METR
Watch for
Near experts on office work
OpenAI built GDPval, 1,320 tasks written by professionals averaging 14 years of experience in 44 occupations, and had experts compare model work with human work without knowing which was which. It reports that 47.6% of Claude Opus 4.1's deliverables were rated as good as or better than the professional's.
Achieved whenan outside group, with its own expert graders, reports a similar rate on the same or similar tasks.
Claimed by
OpenAI (Tejal Patwardhan and colleagues), October 2025
Scores that depend on settings
OpenAI says GPT-5.6 Sol scores 38.3% on ARC-AGI-3, a puzzle benchmark, when run with two settings of its own API that keep the model's reasoning between steps and summarize old context. ARC Prize's standard run, without provider-specific settings, scores it at 7.8%. François Chollet of ARC Prize said such settings are acceptable if the settings and cost are reported.
Achieved whenARC Prize runs the model with those settings in its own harness and posts a comparable score.
Claimed by
OpenAI, July 2026
Open
01
Explain the biggest claim gaps
For the 25 largest gaps between a lab's published benchmark score and an independent score for the same model and test, rerun the test under the lab's stated settings and under a neutral default, and label each gap by cause: settings, test harness (the code that runs the test), grading, leaked test data, or unexplained.
The briefCredit · unclaimed02
Check the office-work grader
An outside check of GDPval's automated grader, an AI model that judges which of two deliverables is better, against blind judgments by working professionals.
The briefCredit · unclaimed
Open 01 · the brief
Explain the biggest claim gaps
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
For the 25 largest gaps between a lab's published benchmark score and an independent score for the same model and test, rerun the test under the lab's stated settings and under a neutral default, and label each gap by cause: settings, test harness (the code that runs the test), grading, leaked test data, or unexplained.
Why it is still open
The open whatmodel project matched 619 lab-published scores with independent ones, across 187 models and 103 tests, and found lab scores 6.3% higher on average (median 1.2%), but it does not say why each gap exists. The ARC-AGI-3 dispute shows that settings alone can move a score from 7.8% to 38.3%.
Built with what exists
whatmodel's MIT-licensed data; Inspect, the open evaluation framework from the UK AI Security Institute, or EleutherAI's lm-evaluation-harness; the models' public APIs; the settings each lab published with its score.
First step
Download whatmodel's claims data, sort by gap size, and pick the 25 largest where the test is public and the model can still be reached through an API.
Done looks like
25 gaps rerun at least three times each, with settings and logs public, each gap assigned one cause, and the share of the average gap that each cause explains.
Open 02 · the brief
Check the office-work grader
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
An outside check of GDPval's automated grader, an AI model that judges which of two deliverables is better, against blind judgments by working professionals.
Why it is still open
OpenAI reports that its grader agreed with human experts 65.7% of the time, against 70.8% agreement between experts themselves. Those are OpenAI's own figures, and no outside check of the grader against professionals was found.
Built with what exists
the 220-task GDPval gold set on Hugging Face, which includes prompts, reference files, expert deliverables and scoring rubrics; model APIs to produce new deliverables; the public automated grading service described in the GDPval paper; a simple blind comparison form; professionals recruited in four or more of the 44 occupations.
First step
Pick four occupations in the gold set where two professionals each can be recruited, and generate deliverables from two models for those tasks.
Done looks like
At least 100 blind comparisons across at least four occupations, each judged by two outside professionals and by the grader, with agreement rates and 95% intervals and all judgments published.