BRENNEN

Industries / Health / Healthcare / Telehealth

Messaging and triage

Messaging and triage covers the written messages that pass between patients and their care team, and the advice lines, apps and chatbots that decide how soon someone needs care, short of the emergency number.

7

Milestones achieved, from 1998 to 2026.

2

To watch for: announced, not yet confirmed outside.

2

Open deliverables. Nobody has shipped them.

The road

AchievedWatch forOpen

Achieved

  1. 1998

    A nurse on the phone

    NHS Direct, a national telephone helpline run by nurses, was announced in December 1997 and opened in March 1998 at three first sites: Preston and Chorley, Milton Keynes, and Northumbria. In its first year it took about 68,500 calls from a population of 1.3 million, and the changes it brought in demand for ambulances and emergency departments were small and not statistically significant. The National Audit Office later reported the service fully available across England and Wales from November 2000, with 29 adverse events in three years, "fewer than one for every 220,000 calls".

    pmc.ncbi.nlm.nih.govnao.org.uk

    Credit

    the National Health Service in England; J. Munro, J. Nicholl, A. O'Cathain and E. Knowles (evaluation); National Audit Office

  2. 2013

    An advice line adds calls

    NHS 111 is a free 24-hour telephone line in England for health problems that are not emergencies, answered by trained call handlers who are not clinicians, with nurses to back them up. It began at four pilot sites in 2010. An evaluation published in 2013 found that it did not reduce calls to 999, the emergency number: in the pilot areas monthly emergency ambulance activity rose by 2.9% against control areas, and emergency department attendance did not change.

    Lessonan advice line that sends uncertain callers onward can add to the demand for urgent care instead of taking pressure off it.

    pmc.ncbi.nlm.nih.gov

    Credit

    J. Turner, A. O'Cathain, E. Knowles and J. Nicholl, University of Sheffield

  3. 2015

    Symptom checkers put to test

    A symptom checker is a website or app that asks about symptoms and suggests a diagnosis and how urgently to seek care. Researchers entered 45 standardized patient cases into 23 free symptom checkers. The correct diagnosis came first 34% of the time, and triage advice was right 57% of the time: 80% for emergencies, but 33% for cases that needed only self-care.

    Lessontools built never to miss an emergency send many people who could stay home to a doctor.

    pmc.ncbi.nlm.nih.gov

    Credit

    Hannah L. Semigran, Jeffrey A. Linder, Courtney Gidengil and Ateev Mehrotra

  4. 2023

    Charging for patient messages

    A patient portal is the secure website or app where patients read their records and write to their care team. In national data from the Epic health record system covering 280,712 US physicians, patients' requests for medical advice by message rose in March 2020 and stayed up; primary care doctors received a mean of 16 such messages a week. UCSF Health in San Francisco let clinicians bill some messages as e-visits, a paid online consultation. Billed e-visits rose from 50.6 to 235.7 a week, still only 1.4% of message threads, and messages fell by about 4,594 a week, from about 59,600.

    Lessona fee touched few messages and cut volume only slightly, so billing alone does not solve a full inbox.

    pmc.ncbi.nlm.nih.govpmc.ncbi.nlm.nih.gov

    Credit

    A. Jay Holmgren, Julia Adler-Milstein and colleagues, UCSF; Holmgren, Nate C. Apathy and Christine A. Sinsky (national data)

  5. 2024

    AI drafts in the inbox

    In 2023 three licensed clinicians compared 195 answers to questions posted on Reddit's r/AskDocs forum and preferred a chatbot's answers to the volunteer doctors' in 78.6% of their ratings, though the authors noted they did not know how chatbots would do in a clinic. In July and August 2023 Stanford Health Care gave 162 clinicians in primary care and gastroenterology a tool inside the Epic record that drafts replies to portal messages with GPT-4. They used a draft about 20% of the time; time spent on the inbox did not change, but scores for task load fell from 61.31 to 47.26 and for work exhaustion from 1.95 to 1.62.

    pmc.ncbi.nlm.nih.govpmc.ncbi.nlm.nih.gov

    Credit

    John W. Ayers and colleagues (2023 study); Patricia Garcia, Stephen P. Ma, Shreya Shah and colleagues, Stanford (pilot)

  6. 2024

    Drafts that misjudge urgency

    Six cancer specialists at Mass General Brigham reviewed GPT-4 drafts that answered 100 made-up patient questions about cancer symptoms. In 11 of 156 reviews (7.1%), sending the draft unedited would have risked severe harm, and in one (0.6%) death; most of these drafts misjudged how urgent the situation was or failed to say what to do about it.

    Lessonthe dangerous mistake in a drafted reply is usually the wrong level of urgency, not a wrong medical fact.

    pmc.ncbi.nlm.nih.gov

    Credit

    Shan Chen, Danielle S. Bitterman and colleagues, Brigham and Women's Hospital and Dana-Farber Cancer Institute

  7. 2026

    A health chatbot stress-tested

    OpenAI launched ChatGPT Health, a consumer health tool, in January 2026. Researchers gave it 60 cases written by clinicians, under 16 test conditions (960 responses). It sent 52% of true emergencies, among them diabetic ketoacidosis and impending respiratory failure, to evaluation within 24 to 48 hours instead of to the emergency department, while it triaged classic emergencies such as stroke correctly. A reanalysis by another team found that, on four emergency cases, under-triage ranged from 17% to 50% depending only on how the answer was asked for, and argued the headline figure does not hold across models.

    Lessonthe format of a triage test can change the measured failure rate threefold, so a result means little without the format beside it.

    europepmc.orgarxiv.org

    Credit

    Ashwin Ramaswamy, Girish Nadkarni and colleagues, Icahn School of Medicine at Mount Sinai (Nature Medicine); David Fraile Navarro, Jialei Sheng, Farah Magrabi and Enrico Coiera (reanalysis)

Watch for

  1. Drafts in each doctor's voice

    Dartmouth researchers tested AI models (Claude, Gemini, ChatGPT, Llama, Aloe and Qwen) on about 146,000 past portal conversations between 10,105 patients and their primary care doctors at Dartmouth Health. They report that matching the drafts to each physician's way of writing improved accuracy by 33% and cut the editing needed by 26%; drafts still ran long, skipped follow-up questions or added irrelevant or wrong medical details. The test was offline: no draft was sent to a patient.

    Achieved whena study in live use, by a group outside Dartmouth, measures clinicians' real editing time and how many drafts would have been unsafe to send.

    home.dartmouth.edu

    Claimed by

    Sarah Preum, Parker Seegmiller, Tim Burdick and colleagues, Dartmouth, at the ACL 2026 conference, July 2026

  2. Small models sort by urgency

    PMR-Bench is a test set of 1,569 patient portal messages, paired with health record data, in which a model must say which of two messages is more urgent (over 2,000 test pairs). Its authors report that two smaller open models they trained, UrgentSFT-8B and UrgentReward-8B, sort an inbox about 15 and 16 points better than the same-size models off the shelf.

    Achieved whenan outside team runs the models on another health system's messages and publishes how many urgent messages they move to the top compared with the current order.

    arxiv.org

    Claimed by

    Joseph Gatto, Sarah M. Preum and colleagues, January 2026

Open

  1. 01

    A written red-flag test

    A public set of at least 300 made-up patient messages, written by clinicians as portal messages and online-consultation forms, in which about one in five hides a warning sign that needs same-day care among routine requests (a refill request that mentions new chest pain, a billing question from someone newly confused). Run the set through the shipped tools that draft replies or sort requests, in their makers' test environments and with written permission, and publish how often each tells the patient to get urgent care or flags the message to staff.

    The brief
    Credit · unclaimed
  2. 02

    A standing triage scoreboard

    A public scoreboard, rerun every three months, that sends the same clinician-written cases to consumer symptom checkers and AI health chat tools, each in several formats (pick a category, answer freely, and a short back-and-forth), and reports how often each tool sends emergencies to a later appointment and sends self-care cases to a doctor.

    The brief
    Credit · unclaimed

Open 01 · the brief

A written red-flag test

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public set of at least 300 made-up patient messages, written by clinicians as portal messages and online-consultation forms, in which about one in five hides a warning sign that needs same-day care among routine requests (a refill request that mentions new chest pain, a billing question from someone newly confused). Run the set through the shipped tools that draft replies or sort requests, in their makers' test environments and with written permission, and publish how often each tells the patient to get urgent care or flags the message to staff.

Why it is still open

Published safety tests use raw models, not the tools clinicians use. The 2024 Mass General Brigham test used GPT-4 directly. A 2025 evaluation of Epic's In Basket drafts scored fluency, coherence and completeness, not whether urgent messages were escalated. PMR-Bench (January 2026) and Columbia's AcuityBench (May 2026) score models on urgency labels, not the replies products send. Meanwhile, since 1 October 2025 GP practices in England must keep online request tools open all day, and the BMA, the doctors' union, says there is "no practical way" to keep urgent requests out.

pmc.ncbi.nlm.nih.govdoi.orgarxiv.orgbma.org.uk

Built with what exists

Messages written by two clinicians, with variants drafted by an AI model such as Claude or Gemini and then checked by the clinicians; synthetic patient records from Synthea, an open-source generator of made-up patients; a sorting model such as Gemini Decision 1 as an open baseline (see the Note "A thousand questions, one pass"); two clinicians grading each reply blind to the tool.

  1. First step

    Write 50 messages and have two clinicians agree, without seeing each other's answers, which ones need same-day action.

  2. Done looks like

    At least 300 messages, 60 of them with red flags, run through at least 3 shipped tools; for each tool, the share of red-flag messages that get an urgent instruction or a staff flag, with a 95% confidence interval, and the false-alarm rate on routine messages; messages, grading sheet and results published.

Open 02 · the brief

A standing triage scoreboard

Credit · unclaimed

The first to publish is credited on this line, and the deliverable moves from Open to Achieved.

What to build

A public scoreboard, rerun every three months, that sends the same clinician-written cases to consumer symptom checkers and AI health chat tools, each in several formats (pick a category, answer freely, and a short back-and-forth), and reports how often each tool sends emergencies to a later appointment and sends self-care cases to a doctor.

Why it is still open

Tests of these tools are one-off studies that use different methods, so the numbers cannot be compared: Semigran's 2015 audit, the 2026 ChatGPT Health study, and the reanalysis that showed the result moving with the answer format. The authors of SCARF, a 2026 reporting framework for these tests, write that a high-quality framework for standardizing them "is still lacking" and describe no shared leaderboard; searches of GitHub and Hugging Face found none.

pmc.ncbi.nlm.nih.govarxiv.orgeuropepmc.org

Built with what exists

The 60 cases released with the ChatGPT Health study and new cases chosen by the SCARF checklist; the symptomcheckR R package for scoring; a browser agent such as Browser Use to enter cases into web-based checkers the same way each time (see the Note "Deleting 80% to start over"); two clinicians to settle disagreements.

  1. First step

    Rerun the 60 released cases on three tools in two formats and publish the transcripts and scores.

  2. Done looks like

    Four quarterly rounds covering at least 8 tools and at least 100 cases in 3 formats each, with under-triage of emergencies and over-triage of self-care cases reported with 95% confidence intervals, and every transcript published.