BRENNEN

Industries / Technology / Telecommunications / Networking equipment

Data center networks

This work designs the switches, cables and layouts that connect the servers, and now the AI chips, inside a data center.

6

Milestones achieved, from 1999 to 2025.

2

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 1999

    A fabric for servers

    InfiniBand is a network built for short delays between servers and storage, whose adapters move data straight between machines' memory without the processor copying it, which is called remote direct memory access (RDMA). The InfiniBand Trade Association (IBTA) formed in August 1999 from two rival efforts, Next Generation I/O, led by Intel, and Future I/O, led by IBM, Compaq and Hewlett-Packard, and had more than 180 member companies by October 2000, when version 1.0 of its specification was due. It was designed as a switched network rather than a shared bus such as PCI, the slot connector inside a server.

    Lessonbuilt to replace the bus inside servers, it lasted as the fabric between them; the IBTA now presents it as the interconnect of large supercomputers and AI clusters.

    electronicdesign.cominfinibandta.org

    Credit

    the IBTA; Intel; IBM, Compaq and Hewlett-Packard

  2. 2008

    Big networks from small switches

    Mohammad Al-Fares, Alexander Loukissas and Amin Vahdat of the University of California, San Diego showed at SIGCOMM 2008, a peer-reviewed networking conference, how to wire many cheap, identical Ethernet switches into a fat-tree, a layered layout that lets clusters of tens of thousands of machines use their full bandwidth, with no change to the servers' network cards. They argued that such switches "may deliver more performance at less cost" than the high-end switches then used.

    ccr.sigcomm.org

    Credit

    Mohammad Al-Fares, Alexander Loukissas and Amin Vahdat, UC San Diego

  3. 2016

    BGP alone runs the hall

    BGP (Border Gateway Protocol) is the protocol networks use to tell each other which addresses they can reach. In August 2016 Petr Lapukhov of Facebook, Ariff Premji of Arista Networks and editor Jon Mitchell published RFC 7938 through the IETF, the internet's standards body: an account of running data centers of more than 100,000 servers on a Clos network, the many-small-switches layout, with BGP as the only routing protocol. Facebook had described one such fabric in November 2014 at its Altoona data center: identical pods of 48 server racks, each pod served by four fabric switches, all running standard BGP.

    Lessonthe design aimed, in the RFC's words, at "operational simplicity and network stability so that a small group of people can effectively support a significantly sized network.".

    rfc-editor.orgengineering.fb.com

    Credit

    Petr Lapukhov, Ariff Premji and Jon Mitchell; Facebook

  4. 2016

    Lossless Ethernet's hidden traps

    RDMA over Converged Ethernet (RoCE), first specified by the IBTA in 2010, carries InfiniBand-style memory transfers over ordinary Ethernet but needs a network that does not drop packets. It gets one from priority flow control (PFC): a switch whose buffer is filling tells the device sending to it to pause. At SIGCOMM 2016 Chuanxiong Guo and six Microsoft colleagues reported on a year and a half of RoCEv2 in Microsoft's data centers. Pause frames chasing each other around four switches formed a deadlock that did not clear "even if we restart all the servers." A network card with a bug sent pause frames without end, and a single such card "may block the entire network from transmitting," which they saw several times. They added watchdogs on switches and cards that limit the damage to hundreds of milliseconds.

    Lessonmaking Ethernet lossless turns dropped packets into stalled ones, and stalls can spread across the network.

    microsoft.cominfinibandta.org

    Credit

    Chuanxiong Guo, Haitao Wu, Zhong Deng, Gaurav Soni, Jianxi Ye, Jitendra Padhye and Marina Lipshteyn, Microsoft

  5. 2024

    AI training over Ethernet

    Meta reported at SIGCOMM 2024 that it uses RoCEv2 for most of its AI training capacity, on a separate back-end network that links GPUs in two-layer Clos "AI Zones," with an extra layer joining zones within a building. Training traffic has few, very large flows, so the usual hashing spread it unevenly; spreading each transfer over more connections improved AllReduce, the step where every GPU combines its results with all the others, by up to 40%. At 400G Meta found firmware bugs in DCQCN, the usual RoCE congestion control, turned it off, and ran for more than a year on PFC alone, with traffic admitted by the receiving GPU through its NCCL communication library.

    Lessonthe network and the GPUs' communication library had to be designed together.

    engineering.fb.com

    Credit

    Adi Gangidi, James Hongyi Zeng and colleagues, Meta

  6. 2025

    A GPU maker sells switches

    NVIDIA bought Mellanox, the main maker of InfiniBand equipment, for $7 billion; the deal closed on 27 April 2020. IDC, a market research firm, counted NVIDIA's Ethernet switch revenue, all of it from data centers, at $1.5 billion in the fourth quarter of 2025, up 192.6% from a year earlier and 15.2% of data center switch sales, behind Arista's 19%. Data center Ethernet switch revenue grew 53.5% in 2025 to $32.5 billion, which IDC ties to networks built for AI. In the second quarter of 2026 IDC ranked NVIDIA first in data center Ethernet switching, with 20.4% of revenue.

    nvidianews.nvidia.comidc.comidc.com

    Credit

    NVIDIA and Mellanox; figures by IDC

Watch for

  1. Ethernet inside the rack

    A scale-up network joins the AI chips in one rack, or a few racks, closely enough that they can share memory, a job NVIDIA does with its own NVLink. Broadcom says its Tomahawk Ultra switch chip, shipping from July 2025, switches 51.2 Tbit/s with 250 ns latency and cuts Ethernet's header overhead from 46 bytes to 10. Used with Broadcom's Scale-Up Ethernet (SUE) framework, Broadcom says, chip-to-chip latency including the switch stays under 400 ns.

    Achieved whena customer or independent lab publishes measured chip-to-chip latency and throughput for a working scale-up system built on it.

    evertiq.comtheregister.com

    Claimed by

    Broadcom, July 2025

  2. Open scale-up switches

    UALink (Ultra Accelerator Link) is an open scale-up standard; its consortium released the 200G 1.0 specification on 8 April 2025 and version 2.0 on 7 April 2026. In January 2026 The Register reported that AMD's first UALink racks would carry the protocol over Ethernet because purpose-built UALink switches were not ready. Upscale AI, a start-up, said that month that its SkyHammer chip would link up to 1,024 devices and be available in the fourth quarter of 2026; in October 2026 it gave SkyHammer's capacity as 115.2 Tbit/s, and The Next Platform noted that it runs UALink over Ethernet and ESUN, an Ethernet-based scale-up effort, so that "under the covers" it is an Ethernet switch chip.

    Achieved whenan open-standard scale-up switch ships in systems and a customer or independent lab reports measured results with accelerators from more than one maker.

    ualinkconsortium.orghpcwire.comtheregister.comnextplatform.com

    Claimed by

    Upscale AI, January and October 2026

Open

  1. 01

    RoCE failures on a bench

    A public kit that reproduces, on a small bench of second-hand switches, the RoCE failures Microsoft reported in 2016: a pause-frame storm from one network card, and a deadlock of pause frames looping through four switches. It then measures the switch watchdogs meant to stop both and publishes the configurations, traffic scripts and timings for each switch operating system tested.

    The brief
    Credit · unclaimed
  2. 02

    Scale-Up Ethernet on FPGAs

    An open-source hardware version of Broadcom's Scale-Up Ethernet framing and credit-based flow control (the receiver grants the sender credits for buffer space, so nothing is dropped) on two FPGA cards joined by a 100G cable, with measured latency and throughput. An FPGA is a chip whose circuits can be rewritten in software.

    The brief
    Credit · unclaimed

Open 01 · the brief

RoCE failures on a bench

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public kit that reproduces, on a small bench of second-hand switches, the RoCE failures Microsoft reported in 2016: a pause-frame storm from one network card, and a deadlock of pause frames looping through four switches. It then measures the switch watchdogs meant to stop both and publishes the configurations, traffic scripts and timings for each switch operating system tested.

Why it is still open

Microsoft described the failures in its own network and released no reproduction. SONiC, the open switch operating system, has a PFC watchdog test plan that generates storms with a script on a helper switch and checks pass or fail inside its own test framework; it publishes no timings across switch makers. The open ns3-rdma project simulates RoCE rather than running real switches. GitHub searches on 20 November 2026 for "PFC deadlock", "pause storm RoCE" and "RoCE testbed congestion" returned no repositories.

github.commicrosoft.com

Built with what exists

Four to six used 100G switches with PFC and a PFC watchdog, for example NVIDIA (Mellanox) Spectrum SN2700 switches running SONiC; NVIDIA ConnectX-4 or later network cards in ordinary servers; rdma-core and perftest to generate RDMA traffic; SONiC's pfc_gen.py script to send pause frames; Wireshark for captures.

  1. First step

    Connect two servers through one switch, send a storm with pfc_gen.py from a third host, and record how long the watchdog takes at its default and its shortest settings.

  2. Done looks like

    Each failure reproduced at least 10 times on at least three switch operating systems, with the time from storm start to watchdog action and the time to recovery published for every run, and a second lab repeating one scenario from the published kit.

Open 02 · the brief

Scale-Up Ethernet on FPGAs

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

An open-source hardware version of Broadcom's Scale-Up Ethernet framing and credit-based flow control (the receiver grants the sender credits for buffer space, so nothing is dropped) on two FPGA cards joined by a 100G cable, with measured latency and throughput. An FPGA is a chip whose circuits can be rewritten in software.

Why it is still open

Broadcom lists the SUE framework specification on its public documents site, but the open work found is software. SUE-Sim, from the Chinese Academy of Sciences, models SUE in the ns-3 network simulator; open UALink projects are a C++ behavioural model (rosslwheeler/ualink) and a controller its author calls still under development. GitHub searches on 20 November 2026.

github.comgithub.comdocs.broadcom.com

Built with what exists

Two AMD Alveo U55C or U250 cards; Coyote, ETH Zurich's open FPGA shell, whose 100G networking already runs RoCEv2 (it needs AMD's licence for the UltraScale+ 100G Ethernet subsystem); Vitis HLS; SUE-Sim to predict results; coding agents can draft the HLS blocks, each checked against SUE-Sim's output (see the Note "Code no person read", draft).

  1. First step

    Run SUE-Sim for a two-node link and record its predicted latency, then build a loopback test on one card that sends SUE-framed packets through Coyote's network path.

  2. Done looks like

    A public repository in which the two cards exchange SUE frames at line rate for one hour with no lost frames under induced congestion, with one-way latency measured by hardware timestamps and compared against SUE-Sim's prediction for the same setup.