BRENNEN

Industries / Technology / Software and SaaS / Developer tools

Testing and checking

Tools that look for mistakes in code, either by reading it without running it (static analysis) or by watching it while it runs, and tools that drive a program the way a user would to see whether it still works.

5

Milestones achieved, from 1978 to 2016.

3

To watch for: announced, not yet confirmed outside.

3

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 1978

    A checker beside the compiler

    Stephen C. Johnson of Bell Labs described lint, a program that reads C source files and reports likely bugs and "obscurities" that the C compiler let through, such as calls that do not match a function's definition in another file. He kept these checks out of the compiler on purpose: the compiler stayed fast, while lint took "a more global, leisurely view of the program." He also wrote that many facts lint would need cannot be known in general (deciding whether a program ever calls exit is as hard as the halting problem), so its checks are a compromise.

    wolfram.schneider.org

    Credit

    Stephen C. Johnson, Bell Laboratories

  2. 2004

    A browser driven by script

    At ThoughtWorks in Chicago, Jason Huggins built a JavaScript tool, later named Selenium, to test an internal time-and-expenses web application by clicking through its pages in a real browser. Colleagues added a server and drivers so tests could be written in other programming languages. Simon Stewart's WebDriver, which controls each browser through its own client instead of JavaScript inside the page, later merged with it. In 2018 WebDriver became a W3C Recommendation, a web standard. Microsoft called it "a standard way for web developers and browser engineers to automate the browser."

    selenium.devsfconservancy.orgblogs.windows.com

    Credit

    Jason Huggins, Paul Gross and Jie Tina Wang at ThoughtWorks; Simon Stewart; the World Wide Web Consortium (W3C)

  3. 2012

    Memory errors caught cheaply

    C and C++ let a program read or write memory it does not own, a common cause of crashes and security flaws. Earlier tools that caught this while a program ran, such as Valgrind's Memcheck, slowed it about 20 times. Four Google engineers built AddressSanitizer into the compiler instead, which adds the checks to the program as it is built. On average it slowed programs by 73%, and in its first 10 months of testing it found more than 300 previously unknown bugs in the Chromium browser. It shipped in GCC 4.8, and Microsoft's C++ compiler has supported it since Visual Studio 2019 version 16.9.

    usenix.orggcc.gnu.orglearn.microsoft.com

    Credit

    Konstantin Serebryany, Derek Bruening, Alexander Potapenko and Dmitry Vyukov, Google

  4. 2015

    Findings moved into code review

    Google's Tricorder shows the results of many analyzers as comments in code review, the step where a colleague reads a change before it goes in. Before it, the command-line version of FindBugs, the analyzer that had drawn the most use at Google, was run by only 35 developers in 2014, 20 of them once. Reviewers can click "NOT USEFUL", and an analyzer whose not-useful rate stays above 10% is put on probation and can be switched off. At Facebook, which open-sourced its Infer analyzer in June 2015, the team first ran Infer nightly over the whole Android codebase and assigned 20 to 30 issues by hand: almost none were fixed, although the team put its false-positive rate below 20%. Run on each change when the engineer submitted it for review, the same analysis saw more than 70% of its reports fixed. The Infer authors say it is also used at Amazon, Spotify and Mozilla.

    LessonThe same warnings get fixed when they arrive inside the change a person is making, and ignored when they arrive later as a list.

    static.googleusercontent.comengineering.fb.comdiscovery.ucl.ac.uk

    Credit

    Caitlin Sadowski, Jeffrey van Gogh, Ciera Jaspan, Emma Söderberg and Collin Winter at Google (Tricorder); Dino Distefano, Manuel Fähndrich, Francesco Logozzo, Peter O'Hearn and the Infer team at Facebook

  5. 2016

    Tests that cannot be trusted

    A flaky test passes and fails on the same code. John Micco of Google reported that about 1.5% of all test runs at Google gave a flaky result, that almost 16% of its tests showed some flakiness, and that about 84% of the changes from pass to fail it observed involved a flaky test. He wrote that developers sometimes dismissed a real failure as flaky. An outside study of 201 fixes to flaky tests in 51 open-source projects found the three most common causes were waiting on asynchronous work, concurrency and dependence on the order in which tests run.

    LessonA test that sometimes fails for no reason teaches people to ignore the failures that are real.

    testing.googleblog.commir.cs.illinois.edu

    Credit

    John Micco, Google; Qingzhou Luo, Farah Hariri, Lamyaa Eloussi and Darko Marinov, University of Illinois

Watch for

  1. Each maker wins its test

    AI code reviewers comment on pull requests, proposed changes to a codebase, the way a human reviewer would. Greptile tested five of them on 50 bug-introducing changes from 5 open-source repositories and put itself first, catching 82% of the bugs, against 58% for Cursor Bugbot, 54% for GitHub Copilot and 44% for CodeRabbit; false alarms did not count against the score. Augment Code ranked its own reviewer first on another public set, with Bugbot at an F-score (a combined measure of catching bugs and avoiding false alarms) of 49%, Greptile 45%, CodeRabbit 39%, Claude Code 31% and Copilot 25%. GitHub published ReviewBench, 219 public pull requests from 187 repositories with findings checked by people, ran every rival on it itself, and put Copilot first with an F-score of 40.1%. On the online table of Martian's Code Review Bench, which Martian presents as independent and which scores tools by how developers respond to their comments, Copilot ranked fourth.

    Achieved whena tester with no product in this market and no payment from a vendor runs the tools on the same changes, publishes every comment, and gets the same ranking on a second run.

    greptile.comaugmentcode.comthenewstack.ioraw.githubusercontent.com

    Claimed by

    Greptile, July 2025; Augment Code, December 2025; GitHub, October 2026

  2. Tests that repair themselves

    Playwright, Microsoft's open-source browser testing tool, added three AI agents in version 1.56. The healer "executes the test suite and automatically repairs failing tests": it replays the failing steps, looks at the page, proposes a patch such as a new locator (the rule a test uses to find a button or field) or a longer wait, and reruns the test until it passes or a limit stops it. Microsoft gives no figures for how often the repairs are right.

    Achieved whenan outside test reports how often Playwright's healer correctly repairs tests broken by harmless page changes, and how often it makes a test pass over a real bug.

    playwright.dev

    Claimed by

    Microsoft, October 2025

  3. A much faster type checker

    A type checker reads code without running it and reports where a value of the wrong kind is used. Astral released ty, a type checker for Python written in Rust, as a beta. It says that without caching ty is "consistently between 10x and 60x faster than mypy and Pyright", two widely used checkers, and that after an edit to a central file in the PyTorch codebase ty updated its findings in 4.7 ms, against 386 ms for Pyright and 2.38 s for Pyrefly.

    Achieved whensomeone outside Astral times ty against mypy and Pyright on a published list of projects and hardware, and compares the errors each one reports.

    astral.sh

    Claimed by

    Astral, December 2025

Open

  1. 01

    AI reviewers beside static analyzers

    A public test that gives the same 100 code changes to AI code reviewers (CodeRabbit, GitHub Copilot code review, Cursor Bugbot, Greptile) and to rule-based static analyzers (Infer, Semgrep, CodeQL). Half the changes are known to introduce a bug and half have no known bug, so false alarms can be counted as well as catches. Every comment from every tool is published.

    The brief
    Credit · unclaimed
  2. 02

    Test suites under sanitizers

    A public table of the 100 most-starred C projects and 100 most-starred C++ projects on GitHub, showing for each whether its own test suite builds and passes with AddressSanitizer and UndefinedBehaviorSanitizer switched on. A sanitizer is a check the compiler adds to a program that stops it at the first memory error or undefined operation. Each report is sorted into real bug or false alarm, and real bugs are reported to the maintainers.

    The brief
    Credit · unclaimed
  3. 03

    A false-heal test for Playwright

    A test bench that measures Playwright's healer agent. Take an open-source web app with Playwright tests and plant 50 harmless changes (a renamed button, a moved field) and 50 real bugs (a wrong total, a save that silently fails). Run the healer on every test that fails and count three outcomes: repaired correctly, gave up, or made the test pass over a real bug.

    The brief
    Credit · unclaimed

Open 01 · the brief

AI reviewers beside static analyzers

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public test that gives the same 100 code changes to AI code reviewers (CodeRabbit, GitHub Copilot code review, Cursor Bugbot, Greptile) and to rule-based static analyzers (Infer, Semgrep, CodeQL). Half the changes are known to introduce a bug and half have no known bug, so false alarms can be counted as well as catches. Every comment from every tool is published.

Why it is still open

The leaderboards found rank AI reviewers only. Greptile and Augment Code tested their own products against rivals, GitHub ran every rival on its own ReviewBench, and Martian's offline and online tables, as The New Stack reports them, list only AI tools; each put a different tool first. None sets rule-based analyzers beside the AI tools as a baseline, or includes changes known to be clean, so that every comment on them is a false alarm. The one independent study found, of 31,073 CodeRabbit comments on 10,191 pull requests in 239 GitHub repositories, measured how developers reacted (36.4% of comments accepted, 56.3% rejected), not how many bugs were caught or missed.

thenewstack.iogreptile.comarxiv.org

Built with what exists

Defects4J, a collection of 854 reproducible bugs from open-source Java projects, each with its fix, from which a change that reintroduces the bug can be made; forks of those projects on GitHub; free or trial plans of each AI reviewer (see the Note "Findings that cite notes" for the kit Cursor builds its agents with, and the Note "Effort, not model" for GitHub Copilot's current setup); Infer, Semgrep and CodeQL, which all analyze Java.

  1. First step

    Take 10 Defects4J bugs, open each bug-introducing change as a pull request on a fork, and run one AI reviewer and Infer on all ten.

  2. Done looks like

    50 bug-introducing changes and 50 clean ones, each reviewed twice by every tool. Published per tool: bugs caught, comments on clean changes, and how often the two runs agree, with all comments and scripts. A second person reruns 10 changes and gets the same counts within 2.

Open 02 · the brief

Test suites under sanitizers

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public table of the 100 most-starred C projects and 100 most-starred C++ projects on GitHub, showing for each whether its own test suite builds and passes with AddressSanitizer and UndefinedBehaviorSanitizer switched on. A sanitizer is a check the compiler adds to a program that stops it at the first memory error or undefined operation. Each report is sorted into real bug or false alarm, and real bugs are reported to the maintainers.

Why it is still open

A 2018 survey of the top 100 C and top 100 C++ projects on GitHub found AddressSanitizer in the build or test scripts of only 24 and 19 of them. Scripts exist to run the test suites of Debian packages under AddressSanitizer, but their author publishes no results and notes that most of those suites test only basic functionality. A search found no published per-project table.

arxiv.orggithub.com

Built with what exists

Clang or GCC with the flags -fsanitize=address and -fsanitize=undefined; Docker to fix each build's setup; the GitHub API to choose projects by stars; an AI coding agent such as Claude Code to adapt build scripts and sort reports.

  1. First step

    Take 10 of the most-starred C projects, add the two flags to their builds, run their tests and record every report.

  2. Done looks like

    All 200 projects listed with build result, test result and every sanitizer report sorted. At least 20 confirmed bugs reported upstream with links, and a count of how many were fixed after 90 days.

Open 03 · the brief

A false-heal test for Playwright

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A test bench that measures Playwright's healer agent. Take an open-source web app with Playwright tests and plant 50 harmless changes (a renamed button, a moved field) and 50 real bugs (a wrong total, a save that silently fails). Run the healer on every test that fails and count three outcomes: repaired correctly, gave up, or made the test pass over a real bug.

Why it is still open

Microsoft documents what the healer does but publishes no accuracy figures. A Currents blog post of August 2026 points out that the healer's own instructions let it change assertions, the lines that state what a test expects, with a passing test as its goal, and builds an example in which a loosened repair passes while a real bug stays; it measures nothing. A search found no published count of how often Playwright's healer hides a planted bug.

playwright.devcurrents.dev

Built with what exists

Playwright 1.56 or later and its test agents, run through a coding agent such as Claude Code or GitHub Copilot (see the Note "Effort, not model"); a small open-source web app run locally, such as a TodoMVC example, and a larger one that already has a Playwright suite; git to keep each planted change on its own branch.

  1. First step

    Plant five renamed buttons and five real bugs in a TodoMVC app, write ten Playwright tests, and run the healer on each failure.

  2. Done looks like

    100 planted changes, each healed three times. Published: share repaired correctly, share where the healer hid a planted bug, and every patch it proposed.