Industries / Technology / Software and SaaS
Running it in production
Keeping software up and answering for the people who use it: the computers it runs on, the people on call, and what happens when it fails.
7
Milestones achieved, from 1961 to 2021.
2
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
1961
Many users, one computer
At MIT, Fernando Corbató, Marjorie Merwin (later Daggett) and Robert Daley began designing the Compatible Time-Sharing System (CTSS) in spring 1961 and first demonstrated it that November, on an IBM 709 with four typewriter terminals. Time-sharing lets many people use one central computer at the same time, each from a terminal and each with their own files. By 1963 CTSS was a stable, large system, "proof positive of the feasibility of time sharing", in the words of the IEEE Computer Society's commemorative history.
Credit
Fernando Corbató, Marjorie Merwin Daggett and Robert Daley, MIT Computation Center
2000
Software rented over the web
Salesforce, founded in 1999, began offering its customer-relationship management (CRM) software as an "on-demand application service" in February 2000. Customers used it through a web browser and paid by subscription instead of installing software themselves, and one system, which the company called multi-tenant, served all of them. Its 2004 stock-market filing said it had drawn "substantially all" of its revenue from these subscriptions. The model is now called software as a service, or SaaS.
Credit
Marc Benioff and the other founders of Salesforce
2003
Operations run by programmers
Benjamin Treynor Sloss joined Google in 2003 to run a "Production Team" of seven engineers and built it into site reliability engineering, or SRE. His definition: "what happens when you ask a software engineer to design an operations team." Google caps the operations work of its SREs at 50% of their time. Each service has an error budget, one minus its availability target: the amount of failure it may spend on shipping new features.
Credit
Benjamin Treynor Sloss, Google
2006
Servers rented by use
Amazon Web Services launched Amazon S3, "storage for the Internet", on 13 March 2006, and a test version of Amazon EC2, "resizable compute capacity in the cloud", on 24 August 2006. Customers paid only for the storage and computing they used, and could scale up or down as needed. A software firm could rent servers instead of buying them.
Credit
Amazon Web Services
2014
One scheduler for many machines
Google released Kubernetes as open source in June 2014, the month its public code repository was created. It places containers, programs packed with everything they need to run, onto a fleet of machines and restarts them when they fail. The project "traces its lineage directly from Borg", Google's internal cluster manager. In the Cloud Native Computing Foundation's survey of 750 people in late 2024, production use of Kubernetes reached 80%, up from 66% in 2023.
Credit
Google engineers; the project is now held by the Cloud Native Computing Foundation
2017
A typo stops S3
On 28 February 2017 an engineer on Amazon's S3 storage team, following an established playbook, ran a command to remove a few servers from a billing system in AWS's US-East-1 region. One input was entered wrongly, and far more servers were removed, including ones that two core S3 systems needed. Both had to restart, and S3 in that region could not serve requests until they did; the second finished at 1:54 PM Pacific time, 4 hours and 17 minutes after the command. Until 11:37 AM AWS could not update its own status page, because the page depended on S3. AWS changed the tool to remove capacity more slowly and never below a system's minimum.
LessonTools that remove capacity need built-in limits, and the page that reports an outage must not depend on the system that is down.
Credit
Amazon Web Services, post-event summary
2021
Locked out by its outage
On 4 October 2021 a command meant to check the capacity of Facebook's global backbone, the network linking its data centers, disconnected all of them. A bug in the audit tool let the command through. Facebook's DNS servers, which tell the rest of the internet where to find its sites, then withdrew their routes, so its services could not be found. Remote access and many internal tools were down too, so engineers went to the data centers in person, which took time because the buildings are hard to get into by design.
LessonThe tools and access needed to recover must keep working when the network they run over is gone.
Credit
Santosh Janardhan, Meta engineering post of 5 October 2021
Watch for
An SRE agent across Microsoft
Microsoft says its Azure SRE Agent, an AI agent that investigates incidents, looks for root causes and can act within limits a team sets, runs as more than 1,300 agents across Microsoft's own services. By Microsoft's figures it has mitigated more than 35,000 incidents and saved more than 20,000 engineering hours. Teams choose its freedom, "ranging from recommended actions to automated responses within defined guardrails."
Achieved whenan outside party reports its root-cause accuracy and time to mitigation on a stated set of incidents, compared with the same kind of incidents handled without it.
Claimed by
Microsoft, March 2026
Investigations 80% faster
AWS says its DevOps Agent, which reads logs, traces and recent code changes to find the cause of an incident and suggest a fix, gave preview customers and partners "up to 75% lower MTTR, 80% faster investigations, and 94% root cause accuracy". MTTR, mean time to resolve, is the average time from an incident's start to its fix. One customer, Western Governors University, reports MTTR down from about two hours to 28 minutes.
Achieved whenan outside party measures its root-cause accuracy on a public set of planted failures, or a customer publishes before-and-after incident data with its method.
Claimed by
AWS, March 2026
Open
01
Vendor SRE agents, tested
A public, side-by-side test of the AI incident agents that cloud companies sell, starting with Azure SRE Agent and AWS DevOps Agent, on the same planted failures. Their scores are set beside those already published for open agents on the same problems.
The briefCredit · unclaimed02
Failure drills for self-hosted apps
A public scorecard of how popular open-source web apps that people host themselves behave when a part fails. The failures: the database stops, the cache disappears, file storage slows down, a background worker dies, the disk fills up. For each app and failure, record whether it keeps serving pages, loses data, shows users a clear error, and recovers by itself when the part comes back.
The briefCredit · unclaimed
Open 01 · the brief
Vendor SRE agents, tested
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public, side-by-side test of the AI incident agents that cloud companies sell, starting with Azure SRE Agent and AWS DevOps Agent, on the same planted failures. Their scores are set beside those already published for open agents on the same problems.
Why it is still open
The products' headline figures come from their makers and customers. Research benchmarks with planted failures exist: SREGym (90 problems, 2026) tested Stratus, Claude Code and Codex, and SRE-Marathon (University of Toronto, September 2026) tested ten methods. Both list the AWS and Microsoft products among their references without scoring them. A June 2026 buyer's guide from Augment Code says "no independent RCA accuracy benchmark exists for this category", RCA meaning root-cause analysis.
Built with what exists
SREGym, which is open source and runs on Kubernetes; Amazon EKS and Azure AKS, the two clouds' managed Kubernetes services, so each vendor's agent can see the cluster it is asked about; each agent on its trial or pay-as-you-go plan; SREGym's own scoring.
First step
Run three SREGym problems on a small EKS cluster and connect the AWS DevOps Agent to it, noting what access it asks for.
Done looks like
At least two vendor agents run on the same 30 SREGym problems, three times each. Published: share of root causes found, share of failures fixed, and minutes per incident, beside the paper's figures for Claude Code and Codex, with all run logs released.
Open 02 · the brief
Failure drills for self-hosted apps
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public scorecard of how popular open-source web apps that people host themselves behave when a part fails. The failures: the database stops, the cache disappears, file storage slows down, a background worker dies, the disk fills up. For each app and failure, record whether it keeps serving pages, loses data, shows users a clear error, and recovers by itself when the part comes back.
Why it is still open
Free tools for causing failures on purpose, a practice called chaos engineering, have existed for years. A 2025 study of 1,275 GitHub repositories that use ten such tools mapped who adopts them and which faults they inject; it did not report how widely used apps behave under failure. Searches found no published failure results for Mastodon, Nextcloud or Discourse.
Built with what exists
each app's own Docker Compose or Kubernetes install; Chaos Mesh or Toxiproxy to cause the failures; a load script in k6 or Locust to play the users; a language model to help write each app's test scripts.
First step
Run Mastodon with Docker Compose, stop its Redis cache for five minutes under light load, and record what users see and whether any posts are lost.
Done looks like
10 apps and 5 failures, each run three times, published as a table with screen recordings and scripts. Each app's maintainers get their results before publication.