Industries / Health / Healthcare / Telehealth
Messaging and triage
Messaging and triage covers the written messages that pass between patients and their care team, and the advice lines, apps and chatbots that decide how soon someone needs care, short of the emergency number.
7
Milestones achieved, from 1998 to 2026.
2
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
1998
A nurse on the phone
NHS Direct, a national telephone helpline run by nurses, was announced in December 1997 and opened in March 1998 at three first sites: Preston and Chorley, Milton Keynes, and Northumbria. In its first year it took about 68,500 calls from a population of 1.3 million, and the changes it brought in demand for ambulances and emergency departments were small and not statistically significant. The National Audit Office later reported the service fully available across England and Wales from November 2000, with 29 adverse events in three years, "fewer than one for every 220,000 calls".
Credit
the National Health Service in England; J. Munro, J. Nicholl, A. O'Cathain and E. Knowles (evaluation); National Audit Office
2013
An advice line adds calls
NHS 111 is a free 24-hour telephone line in England for health problems that are not emergencies, answered by trained call handlers who are not clinicians, with nurses to back them up. It began at four pilot sites in 2010. An evaluation published in 2013 found that it did not reduce calls to 999, the emergency number: in the pilot areas monthly emergency ambulance activity rose by 2.9% against control areas, and emergency department attendance did not change.
Lessonan advice line that sends uncertain callers onward can add to the demand for urgent care instead of taking pressure off it.
Credit
J. Turner, A. O'Cathain, E. Knowles and J. Nicholl, University of Sheffield
2015
Symptom checkers put to test
A symptom checker is a website or app that asks about symptoms and suggests a diagnosis and how urgently to seek care. Researchers entered 45 standardized patient cases into 23 free symptom checkers. The correct diagnosis came first 34% of the time, and triage advice was right 57% of the time: 80% for emergencies, but 33% for cases that needed only self-care.
Lessontools built never to miss an emergency send many people who could stay home to a doctor.
Credit
Hannah L. Semigran, Jeffrey A. Linder, Courtney Gidengil and Ateev Mehrotra
2023
Charging for patient messages
A patient portal is the secure website or app where patients read their records and write to their care team. In national data from the Epic health record system covering 280,712 US physicians, patients' requests for medical advice by message rose in March 2020 and stayed up; primary care doctors received a mean of 16 such messages a week. UCSF Health in San Francisco let clinicians bill some messages as e-visits, a paid online consultation. Billed e-visits rose from 50.6 to 235.7 a week, still only 1.4% of message threads, and messages fell by about 4,594 a week, from about 59,600.
Lessona fee touched few messages and cut volume only slightly, so billing alone does not solve a full inbox.
Credit
A. Jay Holmgren, Julia Adler-Milstein and colleagues, UCSF; Holmgren, Nate C. Apathy and Christine A. Sinsky (national data)
2024
AI drafts in the inbox
In 2023 three licensed clinicians compared 195 answers to questions posted on Reddit's r/AskDocs forum and preferred a chatbot's answers to the volunteer doctors' in 78.6% of their ratings, though the authors noted they did not know how chatbots would do in a clinic. In July and August 2023 Stanford Health Care gave 162 clinicians in primary care and gastroenterology a tool inside the Epic record that drafts replies to portal messages with GPT-4. They used a draft about 20% of the time; time spent on the inbox did not change, but scores for task load fell from 61.31 to 47.26 and for work exhaustion from 1.95 to 1.62.
Credit
John W. Ayers and colleagues (2023 study); Patricia Garcia, Stephen P. Ma, Shreya Shah and colleagues, Stanford (pilot)
2024
Drafts that misjudge urgency
Six cancer specialists at Mass General Brigham reviewed GPT-4 drafts that answered 100 made-up patient questions about cancer symptoms. In 11 of 156 reviews (7.1%), sending the draft unedited would have risked severe harm, and in one (0.6%) death; most of these drafts misjudged how urgent the situation was or failed to say what to do about it.
Lessonthe dangerous mistake in a drafted reply is usually the wrong level of urgency, not a wrong medical fact.
Credit
Shan Chen, Danielle S. Bitterman and colleagues, Brigham and Women's Hospital and Dana-Farber Cancer Institute
2026
A health chatbot stress-tested
OpenAI launched ChatGPT Health, a consumer health tool, in January 2026. Researchers gave it 60 cases written by clinicians, under 16 test conditions (960 responses). It sent 52% of true emergencies, among them diabetic ketoacidosis and impending respiratory failure, to evaluation within 24 to 48 hours instead of to the emergency department, while it triaged classic emergencies such as stroke correctly. A reanalysis by another team found that, on four emergency cases, under-triage ranged from 17% to 50% depending only on how the answer was asked for, and argued the headline figure does not hold across models.
Lessonthe format of a triage test can change the measured failure rate threefold, so a result means little without the format beside it.
Credit
Ashwin Ramaswamy, Girish Nadkarni and colleagues, Icahn School of Medicine at Mount Sinai (Nature Medicine); David Fraile Navarro, Jialei Sheng, Farah Magrabi and Enrico Coiera (reanalysis)
Watch for
Drafts in each doctor's voice
Dartmouth researchers tested AI models (Claude, Gemini, ChatGPT, Llama, Aloe and Qwen) on about 146,000 past portal conversations between 10,105 patients and their primary care doctors at Dartmouth Health. They report that matching the drafts to each physician's way of writing improved accuracy by 33% and cut the editing needed by 26%; drafts still ran long, skipped follow-up questions or added irrelevant or wrong medical details. The test was offline: no draft was sent to a patient.
Achieved whena study in live use, by a group outside Dartmouth, measures clinicians' real editing time and how many drafts would have been unsafe to send.
Claimed by
Sarah Preum, Parker Seegmiller, Tim Burdick and colleagues, Dartmouth, at the ACL 2026 conference, July 2026
Small models sort by urgency
PMR-Bench is a test set of 1,569 patient portal messages, paired with health record data, in which a model must say which of two messages is more urgent (over 2,000 test pairs). Its authors report that two smaller open models they trained, UrgentSFT-8B and UrgentReward-8B, sort an inbox about 15 and 16 points better than the same-size models off the shelf.
Achieved whenan outside team runs the models on another health system's messages and publishes how many urgent messages they move to the top compared with the current order.
Claimed by
Joseph Gatto, Sarah M. Preum and colleagues, January 2026
Open
01
A written red-flag test
A public set of at least 300 made-up patient messages, written by clinicians as portal messages and online-consultation forms, in which about one in five hides a warning sign that needs same-day care among routine requests (a refill request that mentions new chest pain, a billing question from someone newly confused). Run the set through the shipped tools that draft replies or sort requests, in their makers' test environments and with written permission, and publish how often each tells the patient to get urgent care or flags the message to staff.
The briefCredit · unclaimed02
A standing triage scoreboard
A public scoreboard, rerun every three months, that sends the same clinician-written cases to consumer symptom checkers and AI health chat tools, each in several formats (pick a category, answer freely, and a short back-and-forth), and reports how often each tool sends emergencies to a later appointment and sends self-care cases to a doctor.
The briefCredit · unclaimed
Open 01 · the brief
A written red-flag test
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public set of at least 300 made-up patient messages, written by clinicians as portal messages and online-consultation forms, in which about one in five hides a warning sign that needs same-day care among routine requests (a refill request that mentions new chest pain, a billing question from someone newly confused). Run the set through the shipped tools that draft replies or sort requests, in their makers' test environments and with written permission, and publish how often each tells the patient to get urgent care or flags the message to staff.
Why it is still open
Published safety tests use raw models, not the tools clinicians use. The 2024 Mass General Brigham test used GPT-4 directly. A 2025 evaluation of Epic's In Basket drafts scored fluency, coherence and completeness, not whether urgent messages were escalated. PMR-Bench (January 2026) and Columbia's AcuityBench (May 2026) score models on urgency labels, not the replies products send. Meanwhile, since 1 October 2025 GP practices in England must keep online request tools open all day, and the BMA, the doctors' union, says there is "no practical way" to keep urgent requests out.
Built with what exists
Messages written by two clinicians, with variants drafted by an AI model such as Claude or Gemini and then checked by the clinicians; synthetic patient records from Synthea, an open-source generator of made-up patients; a sorting model such as Gemini Decision 1 as an open baseline (see the Note "A thousand questions, one pass"); two clinicians grading each reply blind to the tool.
First step
Write 50 messages and have two clinicians agree, without seeing each other's answers, which ones need same-day action.
Done looks like
At least 300 messages, 60 of them with red flags, run through at least 3 shipped tools; for each tool, the share of red-flag messages that get an urgent instruction or a staff flag, with a 95% confidence interval, and the false-alarm rate on routine messages; messages, grading sheet and results published.
Open 02 · the brief
A standing triage scoreboard
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public scoreboard, rerun every three months, that sends the same clinician-written cases to consumer symptom checkers and AI health chat tools, each in several formats (pick a category, answer freely, and a short back-and-forth), and reports how often each tool sends emergencies to a later appointment and sends self-care cases to a doctor.
Why it is still open
Tests of these tools are one-off studies that use different methods, so the numbers cannot be compared: Semigran's 2015 audit, the 2026 ChatGPT Health study, and the reanalysis that showed the result moving with the answer format. The authors of SCARF, a 2026 reporting framework for these tests, write that a high-quality framework for standardizing them "is still lacking" and describe no shared leaderboard; searches of GitHub and Hugging Face found none.
Built with what exists
The 60 cases released with the ChatGPT Health study and new cases chosen by the SCARF checklist; the symptomcheckR R package for scoring; a browser agent such as Browser Use to enter cases into web-based checkers the same way each time (see the Note "Deleting 80% to start over"); two clinicians to settle disagreements.
First step
Rerun the 60 released cases on three tools in two formats and publish the transcripts and scores.
Done looks like
Four quarterly rounds covering at least 8 tools and at least 100 cases in 3 formats each, with under-triage of emergencies and over-triage of self-care cases reported with 95% confidence intervals, and every transcript published.