Industries / Technology / Software and SaaS / Developer tools
AI coding
AI models that write, change, test and review code, from suggestions inside a code editor to agents that work through a task on their own and hand back a finished change.
7
Milestones achieved, from 2021 to 2026.
3
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
2021
Code graded by running it
OpenAI researchers released HumanEval, a set of programming problems in which each problem is a function description with tests, and a model passes only if its code passes the tests. Codex, a model trained on public code from GitHub, solved 28.8% of the problems, GPT-3 solved 0%, and sampling 100 answers per problem raised Codex to 70.2%. "A distinct production version of Codex" powered GitHub Copilot. In 2023 Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang and Lingming Zhang added 80 times more tests to what they call "the popular HumanEval benchmark". The new tests caught wrong code the originals passed, cut scores by up to 28.9%, and changed the order of some models.
LessonA test set with too few tests passes wrong code and can rank the tools it measures in the wrong order.
Credit
Mark Chen and colleagues, OpenAI; the EvalPlus authors for the stronger tests
2022
Contest problems solved
Competitive programming problems give a task in plain English and need a working algorithm, not just a translation of instructions into code. DeepMind's AlphaCode generated large numbers of candidate programs and kept those that behaved correctly on the problem's examples. In simulated entries to recent contests on Codeforces with more than 5,000 participants, it ranked on average in the top 54.3%. The paper was published in Science on 8 December 2022.
Credit
Yujia Li and colleagues, DeepMind
2024
Better code, shakier delivery
DORA, the software delivery research program run by Google Cloud, reported in its 2024 survey that a 25% increase in AI adoption went with gains of 7.5% in documentation quality, 3.4% in code quality and 3.1% in code review speed. As AI adoption increased, it also estimated that delivery throughput fell 1.5% and delivery stability fell 7.2%. 39% of respondents reported little or no trust in AI-generated code.
LessonWriting code faster does not by itself deliver software better, and DORA tied delivery gains to basics such as small batches of change and strong testing.
Credit
DORA, Google Cloud
2025
A month with an agent
Cognition launched Devin in March 2024 as "the first AI software engineer", saying it resolved 13.86% of SWE-bench issues against a previous best of 1.96%. In January 2025 three researchers at Answer.AI, Hamel Husain, Isaac Flath and Johno Whitaker, reported a month of giving it real work. Of 20 tasks, 3 succeeded, 14 failed and 3 were inconclusive, and they could not predict which tasks would work. Tasks similar to the early successes failed in complex, time-consuming ways.
LessonA launch demo and a benchmark score do not show how a tool performs on a team's own work over weeks.
Credit
Answer.AI
2025
Packages that do not exist
Code written by AI models often tells the user to install libraries, called packages, from public registries. Joseph Spracklen and colleagues at the University of Texas at San Antonio, the University of Oklahoma and Virginia Tech generated 576,000 code samples in Python and JavaScript with 16 models. On average at least 5.2% of the packages named by commercial models, and 21.7% of those named by open models, did not exist, 205,474 different invented names in all. An attacker who registers such a name can wait for people to install it, an attack since called slopsquatting. The paper appeared at USENIX Security 2025. A 2026 replication by Aleksandr Churilov on five newer models, checked against the live PyPI and npm registries, found 3.6% to 5.0% of package mentions invented, about 1% of invented names registered by someone within six months, and one registered with malware.
LessonCheck that every package an AI tool names exists, and is the one meant, before installing it.
Credit
Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath and Murtuza Jadliwala; Aleksandr Churilov for the replication
2025
Agents send pull requests
A pull request is a proposed change that a project's maintainers can accept or reject. By mid-2025 coding agents were opening them on GitHub on their own. Hao Li, Haoxiang Zhang and Ahmed E. Hassan of Queen's University, Canada, published AIDev, an open dataset of more than 456,000 pull requests by five agents, OpenAI Codex, Devin, GitHub Copilot, Cursor and Claude Code, in 61,000 repositories. The agents' pull requests were accepted less often than people's, by 15 to 40 percentage points: 64% for Codex, 49% for Devin and 35% for Copilot. On documentation changes Codex and Claude Code were accepted more often than people.
Credit
Hao Li, Haoxiang Zhang and Ahmed E. Hassan
2026
A browser by agents, judged
In January 2026 Cursor said it ran hundreds of coding agents at once, for close to a week, on one goal: a web browser built from scratch. It said they wrote over a million lines of code across 1,000 files, and its chief executive, Michael Truell, gave about three million. The Register reported that some developers could compile the code only after bug fixes, and quoted Jason Gorman of the consultancy Codemanship noting that the project's automated build was still failing. Software Improvement Group, a firm that rates code quality, scored its maintainability at 1.3 out of 5 stars, in the bottom 5% of the systems it measures, while noting that Cursor presented it as an experiment.
LessonThe number of lines an agent writes says nothing about whether the software works; standard tests and quality checks do.
Credit
Wilson Lin, Cursor; The Register; Software Improvement Group
Watch for
A compiler from 16 agents
Anthropic's Nicholas Carlini set 16 copies of Claude Opus 4.6 to write a C compiler in Rust with no internet access. Over nearly 2,000 Claude Code sessions and just under $20,000 in API charges, they produced 100,000 lines that, the post says, build a bootable Linux 6.9 on x86, ARM and RISC-V. The post lists limits: the compiler called on GCC for 16-bit boot code and for its assembler and linker, and its output ran slower than GCC's with all optimizations off. An outside developer, writing on a personal blog, compiled test programs with it and found them correct against GCC, but did not build a kernel. Another, writing as Harshanu on 8 February 2026, compiled every C file of Linux 6.9 with it without an error, but the link step, which joins the compiled pieces into one program, failed with about 40,784 undefined references, so no kernel came out (see the Note "Code no person read").
Achieved whensomeone outside Anthropic builds and boots Linux with it on all three architectures and publishes the steps and logs.
Claimed by
Anthropic, February 2026
Near the top, half price
GitHub made Copilot Fusion generally available. It picks and combines AI models for each request, and the user chooses only an effort level. GitHub says that on the Artificial Analysis Coding Agent Index its top setting, Fusion Max, "scored 60, within two points of Claude Code with Opus 5.5 Max", at about half the average API cost per task and about 40% less time. Artificial Analysis lists $5.59 per task against $11.42. The Note "Effort, not model" reports that GitHub commissioned and paid for that test.
Achieved whena party with no commercial tie to GitHub reruns the comparison on tasks it publishes.
Claimed by
GitHub, October 2026
Checks its work unasked
Thinking Machines Lab says it trained Inkling 2 Small by reinforcement learning, training by reward, on hundreds of thousands of long coding tasks, with a grader that judged how the model approached each problem and not only whether its code passed. It says the result "verifies its work without being asked". It also says early versions sometimes fabricated results and hid what they had done. It gives no count of how often the model checks its work (see the Note "Checked without being asked").
Achieved whenan outside test counts how often the model tests its own changes unprompted on a published set of tasks, beside other models.
Claimed by
Thinking Machines Lab, November 2026
Open
01
Boot Linux with agents' compiler
An independent rerun of the main claims for Claude's C Compiler. Build Linux 6.9 with it for x86-64, AArch64 and RISC-V, boot each kernel in an emulator, run the same kernel tests on it and on a kernel built with GCC, and rerun the PostgreSQL regression tests its README says pass.
The briefCredit · unclaimed02
Score the agent-built browser
A run of the Web Platform Tests, the shared test suite that browser makers use, against FastRender, the browser Cursor's agents wrote. Publish pass rates for each area beside Servo's published rates, starting with the areas FastRender's README lists as "Complete", such as HTML5 parsing and CSS Flexbox.
The briefCredit · unclaimed
Open 01 · the brief
Boot Linux with agents' compiler
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
An independent rerun of the main claims for Claude's C Compiler. Build Linux 6.9 with it for x86-64, AArch64 and RISC-V, boot each kernel in an emulator, run the same kernel tests on it and on a kernel built with GCC, and rerun the PostgreSQL regression tests its README says pass.
Why it is still open
The kernel claim rests on Anthropic's own account. Of the two outside tests found, one compiled small programs, and the other compiled every C file of Linux 6.9 but failed at the link step with about 40,784 errors, three days after Anthropic's post. A search found no third-party report of booting a kernel built with it. The README now says it uses its own assembler and linker, where the February post said GCC's were used, so which parts were used needs recording too.
Built with what exists
the claudes-c-compiler repository and Rust's cargo build tool; the Linux 6.9 source; QEMU, which emulates each processor; GCC and its cross-compilers as the reference; the kernel's own self-tests; PostgreSQL's source and regression suite; a coding agent such as GitHub Copilot (see the Note "Effort, not model") to write the build scripts.
First step
Build the compiler on a Linux machine, compile a default x86-64 kernel configuration with it, and try to boot the result in QEMU.
Done looks like
For each of the three architectures, a script that builds and boots the kernel from a clean machine, with build and boot logs, build time, and the number of kernel self-tests passed beside a GCC-built kernel. PostgreSQL's result is given as tests passed out of 237. A second person reruns the scripts and gets the same counts.
Open 02 · the brief
Score the agent-built browser
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A run of the Web Platform Tests, the shared test suite that browser makers use, against FastRender, the browser Cursor's agents wrote. Publish pass rates for each area beside Servo's published rates, starting with the areas FastRender's README lists as "Complete", such as HTML5 parsing and CSS Flexbox.
Why it is still open
The outside reviews opened looked at whether the code builds and how maintainable it is. Software Improvement Group says it "didn't try to re-implement browser conformance tests". FastRender's repository holds only a subset of the Web Platform Tests, and no published score on the full suite was found.
softwareimprovementgroup.comraw.githubusercontent.comservo.org
Built with what exists
the Web Platform Tests repository and its test runner; FastRender's Rust source; a small adapter that lets the runner hand pages to FastRender and collect its rendered images; Servo's published results for comparison; a coding agent such as Inkling 2 Small (see the Note "Checked without being asked") to write the adapter.
First step
Run the CSS Flexbox tests through FastRender's own test harness and compare the pass rate with the README's "Complete".
Done looks like
Pass rates for the whole CSS suite and for each area FastRender lists as complete, with run logs and rendered images published. A second run on another machine lands within 1 percentage point.