Industries / Technology / Software and SaaS
Releasing
Getting a finished change from the programmers' shared copy of the code to the people who use it: building it, checking it, and switching it on, for a few users first or for everyone at once.
6
Milestones achieved, from 2000 to 2024.
2
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
2000
Integration many times a day
Martin Fowler and Matthew Foemmel described continuous integration, a practice from Extreme Programming, as it ran on a large project at ThoughtWorks. Each programmer merges work into the shared code at least once a day, and a fully automated build, tests included, "runs many times a day". Mistakes in how the pieces fit together show up within hours instead of at the end of a project.
Credit
Martin Fowler and Matthew Foemmel, ThoughtWorks; the practice came from Extreme Programming
2008
One store for phone apps
On 10 July 2008 Apple opened the App Store through iTunes 7.7, with more than 500 apps, over 125 of them free, that downloaded straight to an iPhone over the cellular network or Wi-Fi. Four days later Apple counted more than 10 million downloads and more than 800 apps. Releasing a phone program came to mean publishing it through the platform's store.
Credit
Apple
2009
Ten deploys a day
At the Velocity conference in June 2009, John Allspaw and Paul Hammond said that on a normal day Flickr, the photo site, was deployed to its servers about 10 times. Their slides name the means: automated servers, shared version control, a one-step build and deploy, feature flags (switches in the code that turn a feature on or off without a new release), shared metrics, and chat robots that post every build and deploy. Data Center Knowledge reported that the talk "generated the most buzz" at the conference.
Credit
John Allspaw and Paul Hammond, Flickr, then part of Yahoo
2012
An old switch wakes up
Knight Capital, a US share-trading firm, installed new order-routing code on its eight routing servers in stages from 27 July 2012. A technician did not copy it to one of them, and no second person checked. The new code reused a flag, a setting that once switched on "Power Peg", a function unused since 2003 but never deleted. On 1 August the flag woke the old code on the eighth server, which in about 45 minutes made 4 million trades in 154 stocks, for more than 397 million shares. Knight lost more than $460 million, and the SEC later fined it $12 million.
LessonDelete code that is no longer used instead of reusing its switches, and check that every server received a release.
Credit
US Securities and Exchange Commission, order of 16 October 2013, which sets out the sequence
2014
Delivery put into numbers
DORA (DevOps Research and Assessment) began a yearly survey study in 2014 that tied software delivery to how well companies performed. It started with four measures: how often a team deploys, how long a change takes from being committed to running in production, how long service takes to recover, and what share of changes fail. By DORA's own history these became the industry's standard benchmark in 2016 and 2017, and in 2024 it moved to five measures.
Credit
DORA, a research program now run by Google Cloud
2024
One update, 8.5 million crashes
On 19 July 2024 CrowdStrike sent a content update for its Falcon security software, Channel File 291, to Windows computers. The update expected 21 input fields where the software supplied 20. The mismatch passed CrowdStrike's own checker, and reading the missing field crashed each machine. Microsoft estimated that 8.5 million Windows devices were affected, under 1% of all Windows machines. CrowdStrike's root-cause report says such updates will now go out in stages: canary testing on a small group first, then wider rings, with rollback if problems appear.
LessonContent and settings pushed to millions of machines need the same staged rollout as program code.
Credit
Microsoft, for the count; CrowdStrike, root-cause analysis of 6 August 2024
Watch for
Risk scores replace code freezes
Meta says Diff Risk Score, a version of its Llama AI model tuned to estimate how likely a code change is to cause a production incident, "helped us eliminate major code freezes". A code freeze is a period, around a big event or a holiday, when no changes are allowed in. During one 2024 partner event, Meta says, it landed more than 10,000 code changes "that previously could not have landed during a freeze", "with minimal production impact". It gives no incident counts.
Achieved whensomeone outside Meta publishes incident rates for the changes the model let through and the ones it held back, or reproduces the method on another company's or an open-source project's history.
Claimed by
Meta, August 2025
Rollback in under three minutes
Vercel, a web hosting company, says its Vercel Agent has run on Vercel's own production deployments for months. In its example, the agent traces a burst of errors on a checkout page to a deploy made four minutes earlier, recommends rolling it back, and with the on-call engineer's sign-off rolls back and opens a pull request with a fix. Vercel gives the time from alert to mitigated as "less than three minutes", and does not say whether the example was a real incident.
Achieved whenan outside party measures the time from alert to rollback, and the rollbacks it made without need, over a stated number of real or planted bad deploys.
Claimed by
Vercel, July 2026
Open
01
A test for canary gates
A public test bench for canary analysis. That is the check that runs while a new release serves a small share of users, say 10%, and decides whether to send it to everyone or roll it back. Plant 30 bad releases in an open-source demo shop (crashes, slow pages, failed payments, a memory leak that shows only after 20 minutes) and 30 good ones. Run every release through a rule-based gate and an AI gate, and publish what each one caught and missed.
The briefCredit · unclaimed02
Lock times for real migrations
A table of how long each recent database migration in popular open-source web apps blocks the database, measured on a copy filled to realistic size, and whether it can be undone. A migration is a script that changes the shape of the database, such as adding a column, and it runs as part of a release. While some migrations run, the database locks a table and the app stalls.
The briefCredit · unclaimed
Open 01 · the brief
A test for canary gates
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public test bench for canary analysis. That is the check that runs while a new release serves a small share of users, say 10%, and decides whether to send it to everyone or roll it back. Plant 30 bad releases in an open-source demo shop (crashes, slow pages, failed payments, a memory leak that shows only after 20 minutes) and 30 good ones. Run every release through a rule-based gate and an AI gate, and publish what each one caught and missed.
Why it is still open
AI gates now exist. The Argo Rollouts metric-ai plugin hands canary analysis to an AI agent that "decides whether a release is healthy or not", and its documentation reports no test of how often it is right. The closest research benchmark, SRE-Marathon (University of Toronto, September 2026), plants faults that arrive through code and configuration changes, but it scores the repair after a change has gone out, not the go-or-roll-back decision during a canary. The Meta and Vercel figures above are their own.
Built with what exists
the OpenTelemetry Demo, an open-source online shop made of many small services, whose built-in feature flags already cause failures such as payment errors, a memory leak and slow images; a Kubernetes cluster (kind on a laptop, or a small cloud cluster); Argo Rollouts with Prometheus metrics for the rule-based gate; the argoproj-labs metric-ai plugin with a current model such as Claude or GPT for the AI gate; NVIDIA OpenShell to keep the agent inside the test cluster (see the Note "Stopped in milliseconds").
First step
Install the OpenTelemetry Demo and Argo Rollouts on a laptop cluster, release a payment service that fails 5% of charges, and see whether a rule-based gate stops it.
Done looks like
60 releases, 30 bad and 30 good, each run three times through each gate. Published for each gate: bad releases caught, good releases rolled back without need, and minutes from the start of the rollout to the decision. Scripts and logs are released so anyone can rerun them.
Open 02 · the brief
Lock times for real migrations
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A table of how long each recent database migration in popular open-source web apps blocks the database, measured on a copy filled to realistic size, and whether it can be undone. A migration is a script that changes the shape of the database, such as adding a column, and it runs as part of a release. While some migrations run, the database locks a table and the app stalls.
Why it is still open
Linters such as squawk and strong_migrations flag risky migrations by reading their code; they do not time them. The one empirical study found of these blocking changes covered a single company's Django system, counting 1,499 schema changes over six years (Willie Lawrence and Eiji Adachi, 2021). A 2021 scan of 666 Django projects by the vendor Code Review Doctor found that 22% had migrations that could not be rolled back, but it read the code and did not measure lock times.
Built with what exists
PostgreSQL in Docker; each app's own migration commands (Rails db:migrate and db:rollback, Django migrate); PostgreSQL's pg_locks view to record how long each lock is held; squawk and strong_migrations for comparison; a language model to write scripts that fill each table with made-up rows.
First step
Fill a Mastodon development database with 1 million posts and time last month's migrations, forward and back, while recording locks.
Done looks like
10 open-source apps built on Rails or Django, such as Mastodon and Discourse (Rails) and Zulip and NetBox (Django). Every migration merged in the past 12 months is run forward and back with 1 million rows in each large table. The published table gives each migration's longest lock in seconds, whether it reversed cleanly, and what the two linters said. A second person reruns 20 migrations picked at random and gets lock times within 20% of the first run.