Industries / Technology / Software and SaaS / Developer tools
Debugging and monitoring
Seeing what a running program actually does, from stopping it to inspect it line by line to collecting logs, measurements and traces from programs in use, so that faults can be found and fixed.
7
Milestones achieved, from 1960s to 2022.
2
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
1960s
Debugging while it runs
DDT, the DEC Debugging Tape, let users of Digital Equipment Corporation's PDP-1 computer find errors in a program while the program was in use. The Computer History Museum says it was based on FLIT, an earlier debugging program written by MIT professor Jack Dennis, and was translated for the PDP-1 by a group of MIT students including Alan Kotok. The museum calls it the first program for the PDP-1 created at MIT and donated to DEC.
Credit
Jack Dennis (FLIT); Alan Kotok and other MIT students (DDT)
2004
Probes in live systems
Bryan Cantrill, Michael Shapiro and Adam Leventhal of Sun Microsystems presented DTrace at the 2004 USENIX Annual Technical Conference. It lets an engineer switch on measurement points, called probes, inside a running operating system and its programs, and it has "zero probe effect" when they are off. Even small systems offered about 30,000 probe points in the kernel alone. Performance problems could now be studied on production machines instead of on copies. USENIX gave the three its 2008 Software Tools User Group award and noted that DTrace had been ported to other operating systems.
Credit
Bryan Cantrill, Michael Shapiro and Adam Leventhal, Sun Microsystems
2010
One request, followed everywhere
A web service can pass one user's request through many programs on many machines, so the log of any one machine shows only part of what happened. Google's Dapper follows each request through every service it touches, a method called distributed tracing. To keep the cost low it records only a sample of requests and instruments a small number of shared libraries, not every program. Google's 2010 paper reported on more than two years of use. Others built the same kind of tool in the open: Uber released Jaeger, a distributed tracing platform, as open source in 2016 and gave it to the Cloud Native Computing Foundation (CNCF), where it became one of the first graduated projects.
Credit
Benjamin H. Sigelman and colleagues, Google; Uber for Jaeger
2012
Metrics for anyone
Prometheus, a monitoring and alerting toolkit, collects numbers from running programs, such as requests per second, and stores each as a time series, a value with the time it was recorded. It was built at SoundCloud, a music-streaming company, starting in 2012. It was contributed to the CNCF in May 2016 as its second hosted project after Kubernetes, and in August 2018 it became the foundation's second graduated project. At graduation the CNCF counted more than 1,000 contributors and named users including DigitalOcean and Uber.
Credit
SoundCloud; the Prometheus community
2017
Record once, replay exactly
A fault that shows up only now and then is hard to catch in a debugger, because the next run may behave differently. rr records a program's run on Linux and replays it exactly, as often as needed, including backwards, which its authors call reverse execution. It runs entirely outside the operating system kernel, on stock hardware, compilers and operating systems. The 2017 USENIX paper says it records real-world low-parallelism workloads with low overhead and is "seeing significant use in practice". The project lists Firefox, Chrome, QEMU and LibreOffice among the programs it can record.
Credit
Robert O'Callahan, Chris Jones, Nathan Froyd (Mozilla), Kyle Huey, Albert Noll and Nimrod Partush
2019
Two standards become one
Before a program can send traces and metrics, its code must be fitted with calls that produce them, called instrumentation. By 2019 there were two open projects for this, OpenTracing and OpenCensus, and they were not compatible. In May 2019 they merged into OpenTelemetry, a CNCF project, with seed committees drawn from Google, LightStep, Microsoft and Uber. Its leads wrote that "the biggest problem with either project has been the fact that there were two of them."
LessonRival standards split the work of instrumenting shared libraries, and a single standard lets each library be instrumented once for every tool.
Credit
Ben Sigelman (OpenTracing) and Morgan McLean (OpenCensus, Google), and the CNCF
2022
A $65 million monitoring bill
Monitoring services charge for the data they take in, and that grows with traffic. On a May 2023 earnings call, Datadog's finance chief said a "large upfront bill" from a client in the first quarter of 2022 had not recurred, and an analyst put it at about $65 million. Gergely Orosz, who writes the Pragmatic Engineer newsletter, confirmed with current and former Coinbase engineers that Coinbase had paid Datadog $65 million for 2021. Coinbase built an in-house replacement on Grafana, Prometheus and ClickHouse and ran it alongside for months, then stayed with Datadog at what an engineer called reasonable costs.
LessonWhat a team collects about its programs needs a budget from the start, because the bill rises with the data.
Credit
Gergely Orosz, The Pragmatic Engineer, for the reporting
Watch for
An AI debugger's own score
Sentry says Seer, an AI agent that reads an error's stack trace, traces, logs and the project's commit history, then finds the cause and drafts a fix as a pull request, "identified root causes with 94.5% accuracy" over more than 38,000 issues it analyzed since its beta. The post does not say how accuracy was judged.
Achieved whenan outside party scores Seer's stated causes against the fixes developers actually merged, on a published set of real errors.
Claimed by
Sentry, June 2025
Traces with no code changes
On 30 October 2026 the OpenTelemetry project released version 1.0.0 of OBI, OpenTelemetry eBPF Instrumentation. eBPF is a Linux feature that runs small, checked programs inside the operating system kernel. OBI uses it to capture traces and metrics from running programs written in Java,.NET, Go, Python, Ruby, Node.js, C, C++ and Rust, with no change to their code. The documentation says its probes work "with minimal overhead". The code began as Grafana Labs' Beyla, which Grafana announced it was donating in May 2025. Grafana's documentation for Beyla gives its own figures: with every service of the OpenTelemetry Demo, a sample online shop, instrumented at about 75 requests a second, Beyla used about 0.5% of the processor and 75 MB of memory, and its probes added about 500 nanoseconds to each request.
Achieved whenan outside party publishes measured overhead and span coverage for OBI against code-level instrumentation on the same application.
opentelemetry.ioopentelemetry.iografana.comarchive.grafana.com
Claimed by
the OpenTelemetry project, October 2026
Open
01
eBPF against SDKs, measured
A public table comparing OBI with the OpenTelemetry SDKs, the libraries that instrument a program from inside its code, on one realistic application. For each service and setup, measure the extra CPU, memory and response time, and how many of the spans (timed steps of a trace) the SDK produces that OBI also produces, with the same parent and the same key details.
The briefCredit · unclaimed02
One app's telemetry bill
A measured, public account of how much log, metric and trace data one standard application produces per million user requests, and what several monitoring services actually bill for it, at three settings for the share of traces kept.
The briefCredit · unclaimed
Open 01 · the brief
eBPF against SDKs, measured
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public table comparing OBI with the OpenTelemetry SDKs, the libraries that instrument a program from inside its code, on one realistic application. For each service and setup, measure the extra CPU, memory and response time, and how many of the spans (timed steps of a trace) the SDK produces that OBI also produces, with the same parent and the same key details.
Why it is still open
MooBench, the Kieker group's benchmark of monitoring overhead, covers tracing agents for Java and Python, OpenTelemetry's among them, and no eBPF tool. Grafana measured Beyla, the code OBI grew from, on the OpenTelemetry Demo, but it counted Beyla's own processor and memory use, not what the instrumented services lost, and made no comparison with the SDKs; Grafana Labs' December 2025 post on combining eBPF with SDKs gives no measured figures. OBI reached 1.0 only on 30 October 2026.
raw.githubusercontent.comarchive.grafana.comgrafana.comopentelemetry.io
Built with what exists
the OpenTelemetry Demo, an open-source online shop made of services in several languages, with its own load generator; OBI 1.0; each language's OpenTelemetry SDK and zero-code agent; MooBench's method for repeated runs; Linux cgroup accounting for CPU and memory; a coding agent to write the run scripts.
First step
Run the Demo on one Linux machine, instrument one service with its SDK and then with OBI, and compare CPU use and span counts over two 30-minute runs.
Done looks like
Services in at least four languages, each run three times for 30 minutes at a fixed load under three setups: no instrumentation, SDK, and OBI. Published for each: added CPU as a share of one core, added memory in MB, added median and 99th-percentile response time in ms, and the share of SDK spans that OBI matched. A second team reruns two services and gets figures within 10%.
Open 02 · the brief
One app's telemetry bill
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A measured, public account of how much log, metric and trace data one standard application produces per million user requests, and what several monitoring services actually bill for it, at three settings for the share of traces kept.
Why it is still open
The one test found that runs the OpenTelemetry Demo against paid services was done by a vendor, OpenObserve, against one rival, Datadog. It reports $174 a day for Datadog against $3 for itself and publishes its dataset. A 2026 study of log retention costs used synthetic logs only, and priced storage time, not traces or metrics.
Built with what exists
the OpenTelemetry Demo and its load generator; one OpenTelemetry Collector that copies every signal to several backends at once and counts the bytes it sends; free tiers or trials of hosted services such as Grafana Cloud, New Relic and Datadog, each with a spending limit set; a self-hosted Prometheus, Grafana and ClickHouse stack as the low-cost reference.
First step
Run the Demo for 24 hours into a Collector that writes to local files, and measure the bytes of each signal per 1,000 requests.
Done looks like
Seven days at a fixed load under three trace sampling settings (100%, 10% and 1% of traces kept), sent to at least four backends. Published: gigabytes per million requests for each signal, each backend's billed usage and cost as shown on its own usage page, and the raw data. A second team reruns one day and gets volumes within 10%.