BRENNEN

Industries / Technology / Software and SaaS

Testing

Checking that software does what it should, and finding where it does not, before its users do.

7

Milestones achieved, from 1990 to 2024.

2

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 1990

    Random input crashes programs

    One night, noise on a dial-up phone line sent stray characters into programs one of the authors was using, and the programs crashed. Barton Miller of the University of Wisconsin turned this into a 1988 class project: feed random characters to UNIX programs and count the failures. The 1990 paper reported crashing 25 to 33% of the utility programs on every version of UNIX tested, almost 90 programs on seven versions. The method is now called fuzzing. The paper won the 2022 Jean-Claude Laprie Award in Dependable Computing from IFIP Working Group 10.4. In 2026 the same group tested AI-written versions of ten Linux utilities and found them as reliable as the human-written ones, or more.

    pages.cs.wisc.edupages.cs.wisc.eduarxiv.org

    Credit

    Barton P. Miller, Lars Fredriksen and Bryan So

  2. 1996

    Ariane 5 lost to software

    On 4 June 1996 the first Ariane 5 rocket veered off its flight path about 40 seconds after the start of its flight sequence, broke up and exploded. The inquiry board traced the failure to "specification and design errors in the software of the inertial reference system", the unit that tells the rocket which way it is pointing. The reviews and tests during development "did not include adequate analysis and testing" of that system or of the complete flight control system, testing which "could have detected the potential failure." The board recommended a test environment closer to the real launcher and a reexamination of all software built into equipment.

    LessonTests that do not reproduce real operating conditions can pass a fatal flaw.

    esa.int

    Credit

    the Ariane 501 Inquiry Board; the European Space Agency, which presented its report in July 1996

  3. 1997

    A test framework takes off

    Kent Beck had built a small framework in Smalltalk for organizing and running unit tests, which are short automated checks of one piece of code. On a flight from Zurich to the 1997 OOPSLA conference in Atlanta, Beck and Erich Gamma wrote JUnit, the same idea for Java. Martin Fowler wrote in 2006 that JUnit "took off like a rocket" and that "nearly every language has at least one JUnit port."

    martinfowler.com

    Credit

    Kent Beck and Erich Gamma

  4. 2000

    Rules instead of examples

    Koen Claessen and John Hughes presented QuickCheck, a tool for the Haskell language. Instead of writing individual test cases, the programmer states a rule the code must always obey, and the tool tries it on many randomly generated inputs. The method is now called property-based testing. In 2010 ACM SIGPLAN gave the paper its Most Influential ICFP Paper Award, noting that its techniques had been adapted to other languages and sold commercially for Erlang and C.

    sigplan.org

    Credit

    Koen Claessen and John Hughes

  5. 2016

    Fuzzing that never stops

    On 1 December 2016 Google announced OSS-Fuzz, a free service that fuzzes open-source projects around the clock and reports the bugs it finds to their maintainers. Google says that as of May 2025 it had helped find and fix over 13,000 vulnerabilities and 50,000 bugs across 1,000 projects. An outside study by Zhen Yu Ding and Claire Le Goues analyzed 23,907 bugs it found in 316 projects, and found that it often finds bugs quickly and that developers often fix them quickly.

    github.comarxiv.org

    Credit

    Google

  6. 2023

    Real issues as a test

    Researchers at Princeton built SWE-bench: 2,294 problems taken from real GitHub issues, and the pull requests that resolved them, in 12 popular Python projects. An AI model solves a problem if its change to the code passes the project's own tests. The best model at the time, Claude 2, solved 1.96%.

    arxiv.org

    Credit

    Carlos E. Jimenez, John Yang, Ofir Press, Karthik Narasimhan and colleagues, Princeton University

  7. 2024

    The test had leaks

    A team led by Reem Aleithan and Song Wang checked the SWE-bench problems solved by the top agent of the time, SWE-agent with GPT-4. In 32.67% of them, the fix was already written in the issue report or its comments. Another 31.08% passed only because the tests were too weak to catch a wrong fix. Without those, the agent's success rate fell from 12.47% to 3.97%. In February 2026 OpenAI stopped reporting scores on SWE-bench Verified, its reviewed subset of the benchmark, after finding that 59.4% of 138 problems it audited had flawed tests or descriptions, and that every leading model it tested could reproduce some of the original fixes from memory.

    LessonA public test ends up in the training data, and the test's own checks need checking.

    arxiv.orgopenai.com

    Credit

    Reem Aleithan, Song Wang and colleagues (SWE-Bench+); OpenAI

Watch for

  1. An agent that writes tests

    Diffblue, which sells an AI agent that writes unit tests for Java, compared it with senior Java engineers using Claude Code on 8 real Java projects. By Diffblue's count, its agent's tests covered 80.7% of lines against 32.3%. They also caught 61.3% of mutants, small bugs planted on purpose to see whether the tests notice, against 24.2%.

    Achieved whena party with no stake in either tool reruns the comparison on projects it picks and publishes the generated tests.

    diffblue.com

    Claimed by

    Diffblue, March 2026

  2. Agents writing property tests

    Muhammad Maaz, Liam DeVoe, Zac Hatfield-Dodds and Nicholas Carlini built an AI agent that reads a Python module, works out rules its functions should obey, and writes property-based tests to check them. Run on 100 popular Python packages, 56% of its bug reports were valid bugs on the authors' own review. They reported 5 bugs, including to NumPy, with 4 patches, and 3 patches were merged.

    Achieved whenmaintainers outside the team confirm most of a large sample of the agent's reports, for example 50, without the authors filtering them first.

    arxiv.org

    Claimed by

    the authors, October 2025

Open

  1. 01

    Mutation scores in public

    A public table, updated monthly, that shows for the 100 most-downloaded Python packages what share of planted bugs (mutants) each package's own tests catch. Then, for the ten weakest, use a coding agent to write tests that catch the surviving mutants, and offer them to the maintainers as pull requests.

    The brief
    Credit · unclaimed
  2. 02

    A fair test of test-writers

    An independent comparison of AI tools that write unit tests, such as Diffblue Cover, Qodo, GitHub Copilot and Claude Code, on the same public Java projects. Score each by line coverage and mutation score, and publish every test each tool wrote.

    The brief
    Credit · unclaimed

Open 01 · the brief

Mutation scores in public

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public table, updated monthly, that shows for the 100 most-downloaded Python packages what share of planted bugs (mutants) each package's own tests catch. Then, for the ten weakest, use a coding agent to write tests that catch the surviving mutants, and offer them to the maintainers as pull requests.

Why it is still open

Mutation testing has been run at scale inside companies. Meta's ACH system applied it to 10,795 Android classes and produced 571 new tests. A search found no public table of mutation scores for widely used open-source packages.

arxiv.org

Built with what exists

mutmut or Cosmic Ray, mutation testing tools for Python; each package's own test suite; PyPI download counts to choose the packages; Hypothesis for property-based tests; a coding agent such as Claude Code or Inkling 2 Small (see the Note "Checked without being asked") to write the new tests.

  1. First step

    Run mutmut on five well-known packages and publish the five scores with their run times, to learn what the full table will cost in computer time.

  2. Done looks like

    Scores for 100 packages, each reproducible from a published script. For the ten weakest, at least 50 new tests merged by maintainers, each linked to the mutant it catches.

Open 02 · the brief

A fair test of test-writers

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

An independent comparison of AI tools that write unit tests, such as Diffblue Cover, Qodo, GitHub Copilot and Claude Code, on the same public Java projects. Score each by line coverage and mutation score, and publish every test each tool wrote.

Why it is still open

The head-to-head figures for these tools come from a vendor, Diffblue, testing its own agent. The academic comparison "Test Wars" (January 2025) covered research tools, EvoSuite, Kex and TestSpark, not the commercial agents teams buy now.

diffblue.comarxiv.org

Built with what exists

PIT, a mutation testing system for Java; the GitBug-Java set of real Java projects used in Test Wars; trial or free tiers of each tool (see the Note "Effort, not model" for GitHub Copilot's current setup); a fixed time budget per project.

  1. First step

    Pick three GitBug-Java projects, run PIT on their existing tests for a baseline, and run one tool on all three.

  2. Done looks like

    Four tools on 20 projects, each run three times, with coverage, mutation scores, tests and logs published. A second team reruns three projects and gets scores within 3 percentage points.