BRENNEN

Industries / Technology / Software and SaaS

Running it in production

Keeping software up and answering for the people who use it: the computers it runs on, the people on call, and what happens when it fails.

7

Milestones achieved, from 1961 to 2021.

2

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 1961

    Many users, one computer

    At MIT, Fernando Corbató, Marjorie Merwin (later Daggett) and Robert Daley began designing the Compatible Time-Sharing System (CTSS) in spring 1961 and first demonstrated it that November, on an IBM 709 with four typewriter terminals. Time-sharing lets many people use one central computer at the same time, each from a terminal and each with their own files. By 1963 CTSS was a stable, large system, "proof positive of the feasibility of time sharing", in the words of the IEEE Computer Society's commemorative history.

    multicians.org

    Credit

    Fernando Corbató, Marjorie Merwin Daggett and Robert Daley, MIT Computation Center

  2. 2000

    Software rented over the web

    Salesforce, founded in 1999, began offering its customer-relationship management (CRM) software as an "on-demand application service" in February 2000. Customers used it through a web browser and paid by subscription instead of installing software themselves, and one system, which the company called multi-tenant, served all of them. Its 2004 stock-market filing said it had drawn "substantially all" of its revenue from these subscriptions. The model is now called software as a service, or SaaS.

    sec.govsalesforce.com

    Credit

    Marc Benioff and the other founders of Salesforce

  3. 2003

    Operations run by programmers

    Benjamin Treynor Sloss joined Google in 2003 to run a "Production Team" of seven engineers and built it into site reliability engineering, or SRE. His definition: "what happens when you ask a software engineer to design an operations team." Google caps the operations work of its SREs at 50% of their time. Each service has an error budget, one minus its availability target: the amount of failure it may spend on shipping new features.

    sre.google

    Credit

    Benjamin Treynor Sloss, Google

  4. 2006

    Servers rented by use

    Amazon Web Services launched Amazon S3, "storage for the Internet", on 13 March 2006, and a test version of Amazon EC2, "resizable compute capacity in the cloud", on 24 August 2006. Customers paid only for the storage and computing they used, and could scale up or down as needed. A software firm could rent servers instead of buying them.

    aws.amazon.comaws.amazon.com

    Credit

    Amazon Web Services

  5. 2014

    One scheduler for many machines

    Google released Kubernetes as open source in June 2014, the month its public code repository was created. It places containers, programs packed with everything they need to run, onto a fleet of machines and restarts them when they fail. The project "traces its lineage directly from Borg", Google's internal cluster manager. In the Cloud Native Computing Foundation's survey of 750 people in late 2024, production use of Kubernetes reached 80%, up from 66% in 2023.

    api.github.comkubernetes.iocncf.io

    Credit

    Google engineers; the project is now held by the Cloud Native Computing Foundation

  6. 2017

    A typo stops S3

    On 28 February 2017 an engineer on Amazon's S3 storage team, following an established playbook, ran a command to remove a few servers from a billing system in AWS's US-East-1 region. One input was entered wrongly, and far more servers were removed, including ones that two core S3 systems needed. Both had to restart, and S3 in that region could not serve requests until they did; the second finished at 1:54 PM Pacific time, 4 hours and 17 minutes after the command. Until 11:37 AM AWS could not update its own status page, because the page depended on S3. AWS changed the tool to remove capacity more slowly and never below a system's minimum.

    LessonTools that remove capacity need built-in limits, and the page that reports an outage must not depend on the system that is down.

    aws.amazon.com

    Credit

    Amazon Web Services, post-event summary

  7. 2021

    Locked out by its outage

    On 4 October 2021 a command meant to check the capacity of Facebook's global backbone, the network linking its data centers, disconnected all of them. A bug in the audit tool let the command through. Facebook's DNS servers, which tell the rest of the internet where to find its sites, then withdrew their routes, so its services could not be found. Remote access and many internal tools were down too, so engineers went to the data centers in person, which took time because the buildings are hard to get into by design.

    LessonThe tools and access needed to recover must keep working when the network they run over is gone.

    engineering.fb.com

    Credit

    Santosh Janardhan, Meta engineering post of 5 October 2021

Watch for

  1. An SRE agent across Microsoft

    Microsoft says its Azure SRE Agent, an AI agent that investigates incidents, looks for root causes and can act within limits a team sets, runs as more than 1,300 agents across Microsoft's own services. By Microsoft's figures it has mitigated more than 35,000 incidents and saved more than 20,000 engineering hours. Teams choose its freedom, "ranging from recommended actions to automated responses within defined guardrails."

    Achieved whenan outside party reports its root-cause accuracy and time to mitigation on a stated set of incidents, compared with the same kind of incidents handled without it.

    techcommunity.microsoft.com

    Claimed by

    Microsoft, March 2026

  2. Investigations 80% faster

    AWS says its DevOps Agent, which reads logs, traces and recent code changes to find the cause of an incident and suggest a fix, gave preview customers and partners "up to 75% lower MTTR, 80% faster investigations, and 94% root cause accuracy". MTTR, mean time to resolve, is the average time from an incident's start to its fix. One customer, Western Governors University, reports MTTR down from about two hours to 28 minutes.

    Achieved whenan outside party measures its root-cause accuracy on a public set of planted failures, or a customer publishes before-and-after incident data with its method.

    aws.amazon.com

    Claimed by

    AWS, March 2026

Open

  1. 01

    Vendor SRE agents, tested

    A public, side-by-side test of the AI incident agents that cloud companies sell, starting with Azure SRE Agent and AWS DevOps Agent, on the same planted failures. Their scores are set beside those already published for open agents on the same problems.

    The brief
    Credit · unclaimed
  2. 02

    Failure drills for self-hosted apps

    A public scorecard of how popular open-source web apps that people host themselves behave when a part fails. The failures: the database stops, the cache disappears, file storage slows down, a background worker dies, the disk fills up. For each app and failure, record whether it keeps serving pages, loses data, shows users a clear error, and recovers by itself when the part comes back.

    The brief
    Credit · unclaimed

Open 01 · the brief

Vendor SRE agents, tested

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public, side-by-side test of the AI incident agents that cloud companies sell, starting with Azure SRE Agent and AWS DevOps Agent, on the same planted failures. Their scores are set beside those already published for open agents on the same problems.

Why it is still open

The products' headline figures come from their makers and customers. Research benchmarks with planted failures exist: SREGym (90 problems, 2026) tested Stratus, Claude Code and Codex, and SRE-Marathon (University of Toronto, September 2026) tested ten methods. Both list the AWS and Microsoft products among their references without scoring them. A June 2026 buyer's guide from Augment Code says "no independent RCA accuracy benchmark exists for this category", RCA meaning root-cause analysis.

arxiv.orgarxiv.orgaugmentcode.com

Built with what exists

SREGym, which is open source and runs on Kubernetes; Amazon EKS and Azure AKS, the two clouds' managed Kubernetes services, so each vendor's agent can see the cluster it is asked about; each agent on its trial or pay-as-you-go plan; SREGym's own scoring.

  1. First step

    Run three SREGym problems on a small EKS cluster and connect the AWS DevOps Agent to it, noting what access it asks for.

  2. Done looks like

    At least two vendor agents run on the same 30 SREGym problems, three times each. Published: share of root causes found, share of failures fixed, and minutes per incident, beside the paper's figures for Claude Code and Codex, with all run logs released.

Open 02 · the brief

Failure drills for self-hosted apps

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public scorecard of how popular open-source web apps that people host themselves behave when a part fails. The failures: the database stops, the cache disappears, file storage slows down, a background worker dies, the disk fills up. For each app and failure, record whether it keeps serving pages, loses data, shows users a clear error, and recovers by itself when the part comes back.

Why it is still open

Free tools for causing failures on purpose, a practice called chaos engineering, have existed for years. A 2025 study of 1,275 GitHub repositories that use ten such tools mapped who adopts them and which faults they inject; it did not report how widely used apps behave under failure. Searches found no published failure results for Mastodon, Nextcloud or Discourse.

arxiv.org

Built with what exists

each app's own Docker Compose or Kubernetes install; Chaos Mesh or Toxiproxy to cause the failures; a load script in k6 or Locust to play the users; a language model to help write each app's test scripts.

  1. First step

    Run Mastodon with Docker Compose, stop its Redis cache for five minutes under light load, and record what users see and whether any posts are lost.

  2. Done looks like

    10 apps and 5 failures, each run three times, published as a table with screen recordings and scripts. Each app's maintainers get their results before publication.