BRENNEN

Industries / Technology / Software and SaaS / Cloud and data centers

Operations and reliability

This is the daily work of keeping a data center running: maintenance and repair, watching for failing parts, and learning from outages.

7

Milestones achieved, from 1990s to 2024.

1

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 1990s

    Four tiers of backup

    The Uptime Institute, a US advisory firm, created its Tier classification "over 30 years ago", by its own account. Tier I is basic capacity. Tier II adds spare power and cooling parts. Tier III is "concurrently maintainable": any part can be serviced without shutting the site down. Tier IV is fault tolerant: computing carries on through any single equipment failure. Uptime says it has issued over 4,300 Tier certifications in more than 120 countries.

    uptimeinstitute.com

    Credit

    Uptime Institute

  2. 2007

    Disks fail more than rated

    Two studies at the USENIX FAST conference in February 2007 counted real drive failures. Bianca Schroeder and Garth Gibson of Carnegie Mellon University, with about 100,000 disks, found yearly replacement rates of 2 to 4% common and up to 13%, against at most 0.88% implied by makers' datasheets. A Google study of its own drives found that temperature and workload mattered much less than earlier reports said, and that the drives' self-monitoring readings (SMART) could not predict individual failures on their own.

    LessonPlan spares from field counts, not from datasheet lifetimes.

    usenix.orgresearch.google

    Credit

    Bianca Schroeder and Garth Gibson; Eduardo Pinheiro, Wolf-Dietrich Weber and Luiz André Barroso at Google

  3. 2013

    Drive failures made public

    Backblaze, a cloud storage company, has published failure rates for its hard drives since 2013, with a daily record of every working drive and its self-monitoring readings, free for anyone to download. In the second quarter of 2026 it counted 354,415 drives and an annualized failure rate of 1.73%.

    backblaze.com

    Credit

    Backblaze

  4. 2021

    Fire destroys a cloud building

    On 10 March 2021 fire destroyed OVHcloud's SBG2 data center in Strasbourg, France. The French accident investigator BEA-RI found that it began in two power rooms, where inverters had shown "unexplained" problems and sensors had read abnormal moisture in the hour before, but it did not settle the cause; power was still present in the burning building at 02:14, about 100 minutes after the 00:35 alarm. The Bas-Rhin fire service found the building had "no automatic fire extinguisher system" and no electrical cut-off.

    LessonA fast way to cut all power and automatic fire suppression matter as much as spare equipment.

    datacenterdynamics.comtheregister.com

    Credit

    OVHcloud (operator); BEA-RI and the Bas-Rhin fire service (investigations)

  5. 2021

    Processors that quietly miscalculate

    In 2021 Google and Facebook each reported processors in their fleets that sometimes give wrong answers without any error signal, which they call silent data corruption. Google named the faulty parts "mercurial cores". Facebook traced lost files to one processor core that always returned 0 for a particular calculation. Both said such chips have to be found by testing while in service.

    LessonFactory tests let some faulty chips through, so a large fleet needs testing in service.

    research.googleengineering.fb.com

    Credit

    Peter Hochschild, Amin Vahdat and others at Google; Harish Dattatraya Dixit and others at Facebook

  6. 2022

    Heatwave beats spare cooling

    On 19 July 2022, London's hottest day on record, a Google Cloud data center serving the europe-west2-a zone could not stay at a safe temperature after "a simultaneous failure of multiple, redundant cooling systems combined with the extraordinarily high outside temperatures". Google shut down part of the zone, and its automation wrongly steered internal traffic away from all three London zones. It promised an audit of cooling equipment and standards across its data centers.

    LessonSpare cooling units can fail together in extreme heat, and failover software needs testing as much as the machines.

    status.cloud.google.comtheregister.com

    Credit

    Google Cloud (operator and incident report)

  7. 2024

    GPUs fail in big runs

    Meta reported that in 54 days of training its Llama 3 405B model on up to about 16,000 H100 GPUs, the job stopped unexpectedly 419 times; faulty GPUs caused 30.1% of the stops and GPU memory 17.2%, yet training ran more than 90% of the time. An outside study of the Delta supercomputer at the University of Illinois, covering 11.7 million GPU hours over 2.5 years, found memory errors 3.2 times as frequent per GPU on H100s as on the older A100s and concluded that about 5% spare GPUs are needed.

    LessonAI clusters need spare GPUs and automatic restarts built into the plan.

    arxiv.orgarxiv.org

    Credit

    Meta's Llama team; Shengkun Cui, Ravishankar Iyer and others at the University of Illinois

Watch for

  1. Robots for data hall chores

    Staff told reporters that Meta is testing robot arms from ABB, Kinova and Watney Robotics to swap network cables and switch servers off and on, and already uses tugger robots to move racks at sites in Iowa and Virginia. Meta declined to comment on the tests to Ars Technica; a spokesperson told Wired, "We need more workers, not fewer." Google had shown video of robots handling hard drives and equipment at the Open Compute summit in October 2024.

    Achieved whenMeta or an outside party publishes how many hall tasks robots do in production and how often they succeed.

    futurism.comnovaramedia.comdatacenterdynamics.com

    Claimed by

    Meta staff speaking to Wired and Ars Technica, August and September 2026

Open

  1. 01

    A robot that swaps drives

    A low-cost robot arm with a camera that pulls a drive from the front bay of a server on a bench and fits a new one, then confirms the server sees it, with the design, code and training data published.

    The brief
    Credit · unclaimed
  2. 02

    An open register of outages

    A free, coded list of publicly reported data center and cloud outages since 2016 whose cause lay in the building: power, cooling, fire, water or physical damage. Each entry links to the operator's own incident report where one exists, is coded by two people, and records how long the outage lasted.

    The brief
    Credit · unclaimed

Open 01 · the brief

A robot that swaps drives

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A low-cost robot arm with a camera that pulls a drive from the front bay of a server on a bench and fits a new one, then confirms the server sees it, with the design, code and training data published.

Why it is still open

Alibaba Cloud said in November 2020 that robots in its data centers inspect and replace faulty hard disks in 4 minutes, but it gave no success rate and published no design. Meta's 2026 tests cover cables and power cycling, and Google showed video, not results. Hebbian Robotics published open planning skills for data hall work on the open-source OpenArm robot in July 2026; its demonstration covers patch cables and thermal scans, not drives. No public tally of drive swaps by a low-cost arm turned up.

theregister.comfuturism.comgithub.com

Built with what exists

The SO-101 open-design arm with Hugging Face's LeRobot software (see the Note "No download"); a wrist camera; Hebbian Robotics' planning skills for the task steps; a decommissioned 1U or 2U server with hot-swap drive bays, on a bench, not in a live hall.

  1. First step

    Record 50 remote-controlled drive swaps on one bench server with an SO-101 arm and train a first policy.

  2. Done looks like

    20 filmed attempts each at removing and fitting a drive, with a published tally of successes and the time per swap beside a technician's time.

Open 02 · the brief

An open register of outages

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A free, coded list of publicly reported data center and cloud outages since 2016 whose cause lay in the building: power, cooling, fire, water or physical damage. Each entry links to the operator's own incident report where one exists, is coded by two people, and records how long the outage lasted.

Why it is still open

The last academic study of this kind, by Haryadi Gunawi and colleagues at the University of Chicago, coded 597 outages from 2009 to 2015. Uptime Institute keeps a database of publicly reported outages, but its full report is for members and subscribers, so its finding that power causes the most outages with an impact cannot be checked from public data. The Cloud Uptime Archive records start, end and severity but no cause. Dan Luu's open list of postmortems has a hand-sorted section on hardware and power failures of 15 entries among 256, and postmortems.app adds categories to 243 of them from keyword rules and an AI model, across software failures of every kind.

ece.iastate.eduuptimeinstitute.comgithub.comgithub.comgithub.com

Built with what exists

Operators' status pages and incident reports; the existing open lists as a starting set; news archives; an AI model to draft a cause code for each report, with two people checking every code; a public spreadsheet or code repository.

  1. First step

    Code the power, cooling and fire outages in the three largest cloud providers' own published incident reports for 2022 and 2023.

  2. Done looks like

    At least 300 outages with a physical cause from 2016 to 2026, each with a source link, a cause code and a duration, the agreement rate between the two human coders published, and the share of each cause set beside Uptime's.