Industries / Technology / Software and SaaS / Cybersecurity
Finding and fixing weaknesses
Finding the flaws in software that is already in use, deciding which to fix first, and getting the fixes installed.
7
Milestones achieved, from 1995 to 2026.
2
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
1995
A free scanner for administrators
On 5 April 1995 Dan Farmer and Wietse Venema released SATAN (Security Administrator Tool for Analyzing Networks), a free program that checked networked Unix computers for well-known holes, such as file systems shared with any computer and old versions of the sendmail mail program. It reported each problem without exploiting it and explained how to fix it. Sun Microsystems sent its customers a security bulletin the same day with the list of patches to install. SAGE, the system administrators' association, warned that the danger lay in how easy SATAN was to use, since anyone could point it at any network.
LessonOnce a free tool can find a known hole in minutes, every unpatched machine is easy to find, for defenders and attackers alike.
Credit
Dan Farmer and Wietse Venema
1995
Paid to find bugs
In October 1995 Netscape Communications announced a "Bugs Bounty": $1,000 to the first person to find a major security bug in the new test version of its web browser, as the Wall Street Journal reported on 11 October. Paying outsiders for flaw reports is now called a bug bounty. By 2026 GitHub and many other organizations ran bounties through platforms such as HackerOne and Bugcrowd.
Credit
Netscape Communications
2003
Fixes on a fixed day
Until 2003 Microsoft released security fixes whenever they were ready. On 15 October 2003 it released its first monthly bundle: seven bulletins covering seven flaws, five of them rated critical. From then on regular fixes came on the second Tuesday of each month, so that administrators could plan testing and restarts; urgent fixes could still come at any time. Microsoft said the timing of fixes had been its customers' "number one" request.
Credit
Microsoft
2021
Fix what attackers use
Teams long ranked flaws by their CVSS score (Common Vulnerability Scoring System), a 0-to-10 severity rating kept since 2005 by FIRST, the global forum of incident response teams. On 3 November 2021 the US Cybersecurity and Infrastructure Security Agency (CISA) ordered federal civilian agencies to fix the flaws in its new catalog of Known Exploited Vulnerabilities (KEV) by set deadlines: two weeks for most, six months for flaws named before 2021. CISA's reasons: the CVSS base score "does not account for if the vulnerability is actually being used," and fewer than 4% of the more than 160,000 recorded flaws had been publicly exploited. On 10 June 2026 a new directive, BOD 26-04, replaced the flat deadlines with an order of priority based on four questions: does the system face the internet, is the flaw in the catalog, can an attack be automated, and how much control does it give.
LessonA severity score says how bad a flaw could be, not whether anyone is using it.
Credit
CISA; FIRST
2025
A near miss for CVE
In April 2025 MITRE, the nonprofit that runs the CVE program, the shared list that gives each known flaw one ID, warned that its US government contract would end on 16 April. A MITRE spokesperson said that if it lapsed, "no new CVEs would be added to the program." Hours before the deadline CISA extended the $57.8 million contract by 11 months, to March 2026. The same day, members of the CVE board announced a nonprofit CVE Foundation to make the program less dependent on one government sponsor. In March 2026 CSO Online reported that CISA and MITRE had renegotiated the contract to remove its end date and, citing unnamed sources, that the program now had a protected line in CISA's budget.
LessonA list the whole world relies on can hang on one contract with one government.
Credit
CISA; MITRE; the founders of the CVE Foundation
2026
A bounty ended by junk
On 26 January 2026 Daniel Stenberg, lead developer of curl, a widely used free tool for moving data over networks, announced that curl's bug bounty would stop on 31 January. Since 2019 it had confirmed 87 flaws and paid more than $100,000. Once, over 15% of reports had been real flaws; in 2025, as reports written with AI tools poured in, fewer than 5% were. curl kept taking reports without payment, and in February 2026 moved back to HackerOne's reporting system. In April Stenberg wrote that junk reports were no longer the problem; the growing number of valid ones was. Nextcloud also ended its bounty over low-effort AI reports, GitHub's Jarom Brown said in May 2026.
LessonA cash reward draws cheap machine-written reports, and each one costs volunteers time to disprove.
Credit
Daniel Stenberg and the curl security team
2026
NIST stops enriching every flaw
The US National Vulnerability Database (NVD), run by the National Institute of Standards and Technology (NIST), adds detail to each CVE record. The most used detail is the list of affected products in a standard naming scheme called CPE, which scanners use to match flaws to installed software. NIST's processing slowed in spring 2024, submissions rose 32% that year, and a backlog grew. On 15 April 2026 NIST said submissions had risen 263% between 2020 and 2025, and that although it enriched nearly 42,000 records in 2025, 45% more than in any earlier year, it could not keep up. It now enriches only records that meet priority criteria; backlog records published before March 2026 were marked "Not Scheduled". Two weeks later it corrected about 4,500 severity scores, 19% of those in the newest CVSS version, that a calculation error had set too high.
LessonChecking every flaw by hand stops working once reports grow faster than the team.
Credit
NIST
Watch for
AI tops a bounty leaderboard
XBOW, a company that sells an autonomous penetration tester (software that probes systems for flaws with the owner's permission), says its system reached first place on HackerOne's US leaderboard. It submitted nearly 1,060 reports to public and private bug bounty programs. At the time of writing, by XBOW's count, 130 had been resolved and 303 accepted as real, while 208 were duplicates and 209 were judged informative only. XBOW's staff reviewed reports before they went out, as HackerOne's rules on automated tools require.
Achieved whenHackerOne or the programs involved publish how many of the system's reports led to fixes, or an outside party audits a sample of them.
Claimed by
XBOW, June 2025
AI foils an exploit
Google says its Big Sleep agent, steered by tips from Google Threat Intelligence, found a flaw in the SQLite database (CVE-2025-6965) that was "known only to threat actors and was at risk of being exploited". Google wrote: "We believe this is the first time an AI agent has been used to directly foil efforts to exploit a vulnerability in the wild."
Achieved whenan outside party, such as the SQLite developers or another threat-intelligence firm, confirms that attackers were preparing to use the flaw before it was fixed.
Claimed by
Google, July 2025
Open
01
Product data for the backlog
An open pipeline that drafts the missing affected-product data, CPE names with version ranges, for CVE records that NIST has marked "Not Scheduled". An AI model reads the CVE text and the vendor's advisory and drafts the entry; a person checks each draft before it is published as open data that scanners can load.
The briefCredit · unclaimed02
Revert rates for AI fixes
A public table that follows security fixes written with AI help and merged into open-source projects in 2025 and 2026. For each, record whether it was later reverted, amended, or followed by a new CVE saying the fix was incomplete. Beside them, the same measures for fixes written by people in the same projects over the same period.
The briefCredit · unclaimed
Open 01 · the brief
Product data for the backlog
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
An open pipeline that drafts the missing affected-product data, CPE names with version ranges, for CVE records that NIST has marked "Not Scheduled". An AI model reads the CVE text and the vendor's advisory and drafts the entry; a person checks each draft before it is published as open data that scanners can load.
Why it is still open
NIST no longer enriches most records. VulnCheck, a vulnerability-data company, publishes machine-made product data free to registered users and says it covered 76.95% of the CVEs published in 2024, against 41.35% for NIST. It reports how many records it covers and how fast, not how often its entries are wrong, and its data sits behind a sign-in rather than under an open license. A Cloud Security Alliance analysis dated 25 April 2026, which says it was written with AI assistance and has not had the organization's formal review, found that the share of CVEs with affected-product data fell from about 95% in 2022 to 57% in 2026. It found that CISA's own enrichment work does not supply this field and that "no commercial or open-source third party publishes CPE applicability at scale" into the shared record, and it estimated that 25,000 to 30,000 CVEs a year now lack it. It proposed a 1,000-record pilot; searches in November 2026 found no published results. NIST said in March 2025 that it was exploring machine learning for such tasks.
Built with what exists
the CVE List in JSON (the CVEProject/cvelistV5 repository on GitHub); NIST's CPE dictionary and NVD API; CISA's Vulnrichment repository; a large language model such as Claude or Gemini; Python; OWASP Dependency-Check, an open scanner that matches software to CPE names, to test that the output works.
First step
Take 100 "Not Scheduled" records for WordPress plugins, a large group whose records often lack product data, and fill them in twice, once by hand and once by model, timing both and comparing results.
Done looks like
1,000 backlog records with checked product data, published under an open license. On 200 records that NIST did enrich, the drafts are scored against NIST's product names and version ranges, and the error rate is published with every mismatch listed; VulnCheck's entries for the same records are scored the same way where its terms allow. Dependency-Check flags a known vulnerable test install using the new data.
Open 02 · the brief
Revert rates for AI fixes
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public table that follows security fixes written with AI help and merged into open-source projects in 2025 and 2026. For each, record whether it was later reverted, amended, or followed by a new CVE saying the fix was incomplete. Beside them, the same measures for fixes written by people in the same projects over the same period.
Why it is still open
Lab tests show that AI-written fixes often look right and are not. In August 2026 1Password's Off-by-1 Labs reported that of 6,080 patches from two frontier models for six flaws, 26.0% fixed the flaw without a material change to the program's behavior. Such studies stop at the lab. The same week, the AppSec Santa newsletter asked whether anyone had published a revert rate for AI-generated security fixes and wrote: "I could not find one." Meanwhile AI models find flaws in growing numbers (see the Note "Cyber defenders first"), and Google says its CodeMender agent alone has sent 72 fixes to open-source projects.
Built with what exists
git history and the GitHub API; OSV.dev, the open vulnerability database, and the credit fields of CVE records, which name the finder; the patches and tools Off-by-1 Labs released as FLAWED on GitHub; Python and pandas; an AI coding agent to read commit discussions, with a person confirming each status.
First step
In five large C projects, list the security fixes merged since January 2025 whose commit message or advisory credits an AI tool, and check each for a later revert or follow-up fix.
Done looks like
300 AI-written fixes and 300 human-written fixes from the same projects, each followed for at least 90 days after merge, each status linked to the commit or advisory it came from. A second person rechecks 30 rows chosen at random and agrees with at least 28.