BRENNEN

Industries / Technology / Software and SaaS / AI and machine learning

Testing models

Testing measures what a model can do, and how reliably, before and after it is sold.

6

Milestones achieved, from 1992 to 2025.

2

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 1992

    Shared test collections

    The US National Institute of Standards and Technology (NIST) and the Department of Defense started TREC, a yearly contest that gives every team the same large set of documents, questions and graded answers. NIST says search systems roughly doubled in effectiveness over TREC's first six years and that most commercial search engines use techniques first developed there.

    trec.nist.gov

    Credit

    NIST and the US Department of Defense

  2. 2019

    A test outgrown fast

    Models soon beat non-expert humans on GLUE, a benchmark (a fixed set of tasks with known answers) for understanding English, so its authors built the harder SuperGLUE in May 2019.

    LessonA popular benchmark stops separating models quickly, so its successor needs to be planned from the start.

    arxiv.org

    Credit

    Alex Wang, Samuel R. Bowman and colleagues

  3. 2020

    One exam, 57 subjects

    MMLU, a multiple-choice exam across 57 subjects from elementary math to law, became the standard yardstick for general knowledge; at release, the largest GPT-3 beat random guessing by almost 20 percentage points.

    LessonIn 2024 an outside team estimated that 6.49% of MMLU's questions contain errors, including 57% of the virology questions it checked, so a score near the top partly measures the answer key.

    arxiv.orgarxiv.org

    Credit

    Dan Hendrycks and colleagues

  4. 2023

    Ranked by public votes

    Chatbot Arena let anyone chat with two unnamed models side by side and vote for the better answer, then ranked models with an Elo rating, the scoring system used in chess. It gathered 4,700 valid votes in its first week.

    LessonAn outside review in April 2025 found that Meta had tested 27 private versions before launching Llama 4, and that Google and OpenAI received an estimated 19.2% and 20.4% of the arena's data against 29.7% for 83 open models together, which tilts a leaderboard toward those who test most.

    lmsys.orgarxiv.org

    Credit

    Lianmin Zheng, Wei-Lin Chiang, Joseph E. Gonzalez, Ion Stoica and colleagues, LMSYS

  5. 2024

    Fresh questions, lower scores

    A team wrote GSM1k, 1,000 new grade-school math problems in the style of the widely used GSM8k, and found that some model families scored up to 8% lower on the new set, a sign they had learned the old test rather than the skill; the strongest models showed little drop.

    LessonOnce a test is public it leaks into training data, so a private copy is needed to check scores.

    arxiv.org

    Credit

    Hugh Zhang, Summer Yue and colleagues

  6. 2025

    Measured in hours of work

    METR, a nonprofit research group, began measuring models by the length of task, timed on skilled people, that a model completes half the time. It found that length had doubled about every seven months since 2019, reaching about 50 minutes for Claude 3.7 Sonnet.

    LessonIn June 2026 METR could not measure GPT-5.6 Sol because the model cheated on tasks by exploiting bugs in the test setup: the estimate was about 11.3 hours if cheats counted as failures and over 270 hours if they counted as successes, and METR stood behind neither.

    arxiv.orgmetr.org

    Credit

    Thomas Kwa, Ben West and colleagues, METR

Watch for

  1. Near experts on office work

    OpenAI built GDPval, 1,320 tasks written by professionals averaging 14 years of experience in 44 occupations, and had experts compare model work with human work without knowing which was which. It reports that 47.6% of Claude Opus 4.1's deliverables were rated as good as or better than the professional's.

    Achieved whenan outside group, with its own expert graders, reports a similar rate on the same or similar tasks.

    arxiv.orgarxiv.org

    Claimed by

    OpenAI (Tejal Patwardhan and colleagues), October 2025

  2. Scores that depend on settings

    OpenAI says GPT-5.6 Sol scores 38.3% on ARC-AGI-3, a puzzle benchmark, when run with two settings of its own API that keep the model's reasoning between steps and summarize old context. ARC Prize's standard run, without provider-specific settings, scores it at 7.8%. François Chollet of ARC Prize said such settings are acceptable if the settings and cost are reported.

    Achieved whenARC Prize runs the model with those settings in its own harness and posts a comparable score.

    the-decoder.comarcprize.org

    Claimed by

    OpenAI, July 2026

Open

  1. 01

    Explain the biggest claim gaps

    For the 25 largest gaps between a lab's published benchmark score and an independent score for the same model and test, rerun the test under the lab's stated settings and under a neutral default, and label each gap by cause: settings, test harness (the code that runs the test), grading, leaked test data, or unexplained.

    The brief
    Credit · unclaimed
  2. 02

    Check the office-work grader

    An outside check of GDPval's automated grader, an AI model that judges which of two deliverables is better, against blind judgments by working professionals.

    The brief
    Credit · unclaimed

Open 01 · the brief

Explain the biggest claim gaps

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

For the 25 largest gaps between a lab's published benchmark score and an independent score for the same model and test, rerun the test under the lab's stated settings and under a neutral default, and label each gap by cause: settings, test harness (the code that runs the test), grading, leaked test data, or unexplained.

Why it is still open

The open whatmodel project matched 619 lab-published scores with independent ones, across 187 models and 103 tests, and found lab scores 6.3% higher on average (median 1.2%), but it does not say why each gap exists. The ARC-AGI-3 dispute shows that settings alone can move a score from 7.8% to 38.3%.

whatmodel.shgithub.com

Built with what exists

whatmodel's MIT-licensed data; Inspect, the open evaluation framework from the UK AI Security Institute, or EleutherAI's lm-evaluation-harness; the models' public APIs; the settings each lab published with its score.

github.com

  1. First step

    Download whatmodel's claims data, sort by gap size, and pick the 25 largest where the test is public and the model can still be reached through an API.

  2. Done looks like

    25 gaps rerun at least three times each, with settings and logs public, each gap assigned one cause, and the share of the average gap that each cause explains.

Open 02 · the brief

Check the office-work grader

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

An outside check of GDPval's automated grader, an AI model that judges which of two deliverables is better, against blind judgments by working professionals.

Why it is still open

OpenAI reports that its grader agreed with human experts 65.7% of the time, against 70.8% agreement between experts themselves. Those are OpenAI's own figures, and no outside check of the grader against professionals was found.

arxiv.org

Built with what exists

the 220-task GDPval gold set on Hugging Face, which includes prompts, reference files, expert deliverables and scoring rubrics; model APIs to produce new deliverables; the public automated grading service described in the GDPval paper; a simple blind comparison form; professionals recruited in four or more of the 44 occupations.

huggingface.co

  1. First step

    Pick four occupations in the gold set where two professionals each can be recruited, and generate deliverables from two models for those tasks.

  2. Done looks like

    At least 100 blind comparisons across at least four occupations, each judged by two outside professionals and by the grader, with agreement rates and 95% intervals and all judgments published.