BRENNEN

Industries / Health / Healthcare

Diagnosis

Working out what is wrong with a person from what they report, an examination, and tests on blood, tissue and the heart's electrical signals.

7

Milestones achieved, from 1816 to 2021.

2

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 1816

    Listening through a tube

    R. T. H. Laennec, a physician at the Necker Hospital in Paris, invented the stethoscope in 1816 and matched what he heard in patients' chests with what he later found at autopsy. His book on the method, De l'Auscultation Médiate, followed in 1819.

    europepmc.org

    Credit

    R. T. H. Laennec

  2. 1901

    Recording the heart's current

    Willem Einthoven built the string galvanometer, a very sensitive meter that drew the heart's small electric currents as a trace, the electrocardiogram (ECG). The first machines were massive and could not be moved. He received the 1924 Nobel Prize "for his discovery of the mechanism of the electrocardiogram".

    europepmc.orgeuropepmc.org

    Credit

    Willem Einthoven

  3. 1979

    A program matches the experts

    MYCIN was a computer program that recommended antibiotics by applying a set of rules. In a blinded test on 10 cases of meningitis, eight outside experts rated its choices acceptable 65% of the time, against 42.5% to 62.5% for five faculty specialists. It never became an established clinical system. A 2023 review gives the reason as the limited number of rules that could be fitted into clinical work.

    LessonMatching experts on a test does not get a tool used in care.

    europepmc.orgeuropepmc.orgeuropepmc.org

    Credit

    V. L. Yu, B. G. Buchanan and colleagues, who ran the test

  4. 2017

    Slides read on a screen

    The US Food and Drug Administration (FDA) allowed Philips to market a system that scans glass slides of tissue so that pathologists can make a diagnosis on screen instead of at a microscope. The FDA said it was the first time it had permitted this, after a study of about 2,000 cases. Digital slides made large open contests possible: in the PANDA challenge, 1,290 developers trained programs on 10,616 prostate biopsies, and several of the programs reached the level of pathologists on independent sets from the US and Europe. Their agreement with expert pathologists scored 0.862 and 0.868, on a scale where 1 is perfect agreement.

    fda.goveuropepmc.org

    Credit

    Philips; W. Bulten, M. Eklund and the PANDA consortium

  5. 2018

    Fraud in a blood test

    The US Securities and Exchange Commission (SEC) charged Theranos, its founder Elizabeth Holmes and its former president Ramesh Balwani with a fraud that raised more than $700 million. The company had said its portable analyzer ran comprehensive tests from a few drops of blood from a finger. By the SEC's account it could run only a small number of tests, and most patient tests were run on other makers' machines.

    LessonA lab test needs checking by outsiders before patients rely on it.

    sec.gov

    Credit

    US Securities and Exchange Commission

  6. 2021

    A sepsis alert tested outside

    Sepsis is the body's runaway response to infection. Researchers at Michigan Medicine tested the Epic Sepsis Model, a proprietary alert used in hundreds of US hospitals, on 38,455 hospital stays. It scored 0.63 on a scale where 0.5 is chance and 1 is perfect, missed 67% of sepsis cases, and raised alerts on 18% of all stays.

    LessonA vendor's model must be tested in each hospital that uses it.

    europepmc.org

    Credit

    A. Wong, K. Singh and colleagues

  7. 2021

    AI-read ECGs in a trial

    An AI model that reads an ordinary ECG can flag a weak heart pump, called low ejection fraction. In a randomized trial with 120 primary care teams and 22,641 adults, teams shown the AI result diagnosed low ejection fraction within 90 days in 2.1% of patients, against 1.6% for teams in usual care.

    europepmc.org

    Credit

    X. Yao, P. A. Noseworthy and colleagues

Watch for

  1. AI works up hard cases

    Microsoft's MAI-DxO runs several AI models as a virtual panel of doctors that asks questions, orders tests and names a diagnosis. On 304 case records from the New England Journal of Medicine (NEJM), it was correct in 83.2% of cases, against 36.1% for generalist physicians allowed reference books and search, and 22.3% for physicians without them. On 325 emergency department cases from MIMIC-IV, a research database of real hospital records, it scored 80.3% against 46.8% for physicians. All 15 authors work for Microsoft, and NEJM cases lean toward hard, rare problems.

    Achieved whena group outside Microsoft repeats the result on cases it chose, or a study in working clinics shows it.

    microsoft.aiunite.ai

    Claimed by

    Microsoft AI, June 2025 (preprint) and October 2026 (Science)

  2. An AI takes the history

    Google's AMIE, a chat-based AI, interviewed 100 adults by text before their urgent primary care visits at Beth Israel Deaconess Medical Center in Boston, between April and November 2025, with a physician watching each chat. No chat had to be stopped for safety. By the study's count, the final diagnosis was among AMIE's top seven possibilities in 90% of cases and was its first choice in 56%. Blinded reviewers found no significant difference between its diagnosis lists and those of the primary care doctors, and rated the doctors' treatment plans more practical and cheaper. Alphabet, Google's parent company, funded the study.

    Achieved whena controlled trial run outside Google shows that patients interviewed by AMIE are diagnosed as accurately and safely as patients who are not.

    research.googleunite.ai

    Claimed by

    Google and Beth Israel Deaconess Medical Center, March 2026 (preprint) and October 2026 (The Lancet)

Open

  1. 01

    An outside rerun of MAI-DxO

    At least 100 open diagnostic cases made from openly licensed case reports, written up in the format of Microsoft's Sequential Diagnosis Benchmark (SDBench), in which an AI must ask for each fact and test before it names a diagnosis. Every case comes from an article published after the training cutoff of each model tested. Run the benchmark on at least three models, including one with downloadable weights, and publish accuracy and test costs beside Microsoft's.

    The brief
    Credit · unclaimed
  2. 02

    A lab-result answer key

    A public set of at least 500 made-up lab reports, such as blood counts and metabolic panels, laid out the way patient portals show them. Each has an answer key: which values fall outside the reference range, which cross a critical limit that calls for same-day contact with a clinician, and which diagnoses the numbers do not support. Then score popular chatbots on what they tell a patient who pastes the report in.

    The brief
    Credit · unclaimed

Open 01 · the brief

An outside rerun of MAI-DxO

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

At least 100 open diagnostic cases made from openly licensed case reports, written up in the format of Microsoft's Sequential Diagnosis Benchmark (SDBench), in which an AI must ask for each fact and test before it names a diagnosis. Every case comes from an article published after the training cutoff of each model tested. Run the benchmark on at least three models, including one with downloadable weights, and publish accuracy and test costs beside Microsoft's.

Why it is still open

Microsoft's results come from its own staff. Its code is public under the MIT license, but its NEJM case set must be licensed from NEJM, its MIMIC set needs credentialed access, and the only open set is five made-up cases that its makers say should not be used to report performance. I found no outside run published since the code appeared in October 2026.

github.commicrosoft.github.io

Built with what exists

the SDBench code; case reports from the PubMed Central Open Access Subset, which can be retrieved through PMC's BioC API or OAI-PMH service, keeping only articles whose license allows reuse; hosted models such as Claude, Gemini or GPT; open-weight models such as Google's MedGemma 27B or Mistral Large 4 (see the Note "A trillion open parameters"). The advance that makes this a small-team job is language models that can play the patient record, revealing only the facts asked for, as SDBench's "gatekeeper" does.

pmc.ncbi.nlm.nih.gov

  1. First step

    Pull ten 2026 case reports under CC BY licenses from PMC, convert them to SDBench's case format, and run the bundled baseline agent on them.

  2. Done looks like

    100 or more cases, each with its source article and license. A clinician checks 20 cases and confirms that the case file holds no facts the article does not. Results for at least three models are published with the cases and code, and an outside person reruns 10 cases and gets the same correct-or-wrong verdict on at least 9.

Open 02 · the brief

A lab-result answer key

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public set of at least 500 made-up lab reports, such as blood counts and metabolic panels, laid out the way patient portals show them. Each has an answer key: which values fall outside the reference range, which cross a critical limit that calls for same-day contact with a clinician, and which diagnoses the numbers do not support. Then score popular chatbots on what they tell a patient who pastes the report in.

Why it is still open

Under the US 21st Century Cures Act Final Rule, patients get results as soon as they are ready, often before a clinician has seen them. In a 2023 survey of 8,139 patients, 95.7% preferred that, and 16.5% of those with abnormal results said it made them more worried. The best-known test of chatbots on lab questions used 53 questions taken from Yahoo! Answers, and medical experts reviewed the answers to only 7. A search found no public test that scores whether chatbots flag critical values to patients.

europepmc.orgeuropepmc.org

Built with what exists

Synthea, MITRE's open-source generator of made-up patient records, or a short script that draws values from published reference ranges; LOINC codes to name each test; a hospital laboratory's published critical-value list; chatbots through their public apps or APIs; a scoring script.

synthetichealth.github.io

  1. First step

    Write 50 basic metabolic panels from published reference ranges, 10 of them with a critical potassium or sodium value, and test one chatbot.

  2. Done looks like

    500 reports, 100 of them with at least one critical value. Two people with clinical laboratory training agree on the key for at least 95% of items. At least five chatbots are scored on two numbers: the share of critical values flagged with advice to contact a clinician that day, and the share of answers that name a diagnosis the numbers do not support. Data, prompts and scores are published and rerun after each major model release.