Industries / Health / Healthcare / HealthTech
Decision support for clinicians
Decision support is software that gives doctors, nurses and pharmacists advice while they care for a patient: reminders, warnings about doses, allergies and drug interactions when an order is entered, risk scores built into the record, and reference and AI tools that answer clinical questions.
7
Milestones achieved, from 1972 to 2026.
3
To watch for: announced, not yet confirmed outside.
3
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
1972
A computer diagnoses abdominal pain
In Leeds, doctors entered each patient's findings in coded form, and a computer applied Bayes' rule, a formula that turns data on past patients into a probability for each possible diagnosis. In 304 patients with sudden abdominal pain, its leading diagnosis was right in 91.8% of cases, against 79.6% for the most senior doctor who saw each patient. In a 1986 study across eight centers with 16,737 patients, comparing periods before and after the system arrived, correct first diagnoses rose from 45.6% to 65.3%, and perforated appendixes fell from 23.7% to 11.5% of appendicitis cases. De Dombal later wrote that the gains came from the computer acting as a focus for teaching, not from superior artificial intelligence; in a 1991 study of 12,506 patients, structured forms plus a teaching program did at least as well as live computer advice.
LessonMuch of the benefit came from recording findings in a fixed, structured way, with or without the computer.
Credit
F. T. de Dombal and colleagues, University of Leeds
1976
Reminders double follow-through
Clement McDonald, who built the Regenstrief Medical Record, one of the first electronic health records, had a computer check patient records against 390 rules, mostly about conditions that drugs treat, such as high blood pressure, or cause, such as liver damage, and remind doctors when an action was due. In a study of nine physicians, alternating periods with and without reminders, doctors acted on 51% of 327 events when reminded and on 22% of 385 events when not; their level of training made no significant difference. McDonald concluded that many errors come from the limits of human information processing, not from failings that training can fix. The US National Library of Medicine says the study "introduced the first use of decision support in randomized trials of health information systems."
LessonCapable doctors still miss routine steps, and a prompt at the right moment catches many of them.
Credit
Clement J. McDonald
1998
Order errors cut by half
In computerized physician order entry (CPOE), doctors type orders into a computer, which checks them for problems such as allergies, overdoses and interactions. At Brigham and Women's Hospital in Boston, serious medication errors that went uncaught fell 55% after it was introduced, from 10.7 to 4.86 per 1,000 patient-days, and uncaught errors that could have caused harm fell 84%; preventable drug injuries fell 17%, a change too small to rule out chance. The same year LDS Hospital in Salt Lake City, whose HELP computer system had run since 1967, reported that an antibiotic advice program in a 12-bed intensive care unit cut excess doses from 405 to 87, and orders for drugs patients were allergic to from 146 to 35, compared with the two years before.
Credit
D. W. Bates, L. L. Leape and colleagues, Brigham and Women's Hospital; R. S. Evans, D. C. Classen and colleagues, LDS Hospital
2005
Order software creates new errors
At one teaching hospital, University of Pennsylvania researchers surveyed 261 house staff, the doctors in training who entered most orders (88% of the system's users), and ran focus groups, interviews and observation. They found that a widely used order entry system made 22 kinds of medication error more likely, among them screens that split a patient's drug list so it could not be seen whole, pharmacy stock lists mistaken for dosing guides, and separate functions that allowed double doses; three quarters of the house staff reported seeing each risk, which the authors read as weekly or more often. The same year, a review of 100 controlled trials found that decision support improved what clinicians did in 64% of studies, but improved patient outcomes in only 7 of the 52 that measured them, and that systems succeeded in 74% of studies when the study authors had built them, against 28% when they had not.
LessonOrder software removes some errors and adds others, and studies by a system's own builders report better results than studies by others.
Credit
R. Koppel, B. L. Strom and colleagues, University of Pennsylvania; A. X. Garg, R. B. Haynes and colleagues
2006
Most drug alerts overridden
A review of 17 studies by hospital pharmacists and researchers at Erasmus University Medical Center in Rotterdam found that clinicians overrode drug safety alerts in order entry systems, clicking past them, in 49% to 96% of cases. Some overrides were justified, but the authors traced many to alerts that fired when they did not matter for the patient, unclear wording and interruptions to work, which lead doctors to ignore or misread alerts.
LessonAn alert that fires too often stops being read, so a system has to be judged on what it leaves quiet as well as on what it flags.
Credit
H. van der Sijs, J. Aarts, A. Vulto and M. Berg
2010
Test orders for live systems
In a test designed by researchers at the University of Utah and Brigham and Women's Hospital and run by the Leapfrog Group, which publishes US hospital safety ratings, hospital staff load made-up patients into their live order system, a clinician enters orders modeled on real errors that harmed or killed patients, and the hospital reports which orders drew a warning. In the first published results, from 62 hospitals, the systems flagged 53% of orders that would have killed a patient and 10% to 82% of those that would have caused serious harm. By 2018, 1,812 hospitals took the test and the average score had risen from 53.9% in 2009 to 65.6%: simple checks such as drug allergies reached 85.6%, checks that need lab results, age or diagnosis only 46.1%, and the choice of vendor explained just 9.9% of the differences between hospitals. A 2020 study found that hospitals which also alerted on harmless "nuisance" test orders scored 3 percentage points higher, a sign to its authors that some gain points by over-alerting; Leapfrog's 2026 standard asks systems to catch at least 60% of common, serious prescribing errors.
LessonThe same software catches different errors depending on how each hospital sets it up, so it has to be tested as installed.
europepmc.orgpmc.ncbi.nlm.nih.goveuropepmc.orgratings.leapfroggroup.org
Credit
J. Metzger, D. C. Classen, D. W. Bates and colleagues; the Leapfrog Group
2026
AI advice tested in Kenya
At 16 Penda Health primary care clinics in Nairobi and Kiambu counties, 103 clinical officers, the clinicians who saw the patients, were randomly split: half had AI Consult, a tool built on OpenAI's GPT-4o that reads visit notes in the clinic's record as they are written and shows a green, yellow or red signal, and half did not. Among 9,691 patients enrolled from April to July 2025, treatment failure within 14 days, such as death or symptoms that did not resolve, was 2.2% with the tool and 2.0% without; after adjustment the odds were 23% lower with the tool, a result not statistically significant. Expert reviewers of 2,000 visits found appropriate diagnoses and treatment plans more often with the tool (odds about 1.7 times higher), and of 1,000 red alerts reviewed, 4.2% were judged unsafe or somewhat unsafe. An earlier study in the same network, funded by OpenAI and posted in July 2025, had reported 16% fewer diagnostic errors and 13% fewer treatment errors over 39,849 visits.
LessonBetter notes and plans did not show up as fewer treatment failures in a trial of about 10,000 patients, and a co-author estimated that about 139,000 would be needed to detect a meaningful difference.
Credit
A. Agweyu (KEMRI-Wellcome Trust Research Programme), B. A. Mateen (University of Birmingham and PATH) and colleagues, with Penda Health staff; funded by the Gates Foundation, with no role for OpenAI in the trial
Watch for
OpenEvidence ahead on real queries
OpenEvidence is an AI answer tool for clinicians; a JAMA study co-written with three people from the company counted 703,265 monthly active US clinician users in June 2026. In a June 2026 preprint, 149 physicians in 36 states compared blinded answers to 620 real questions sent to OpenEvidence by its users, and rated OpenEvidence above Claude Opus 4.8, Gemini 3.1 Pro and GPT-5.5 on all five measures, including accuracy, by margins of 25 to 39 percentage points. None of the authors works for the company, but the paper states that OpenEvidence co-developed the data collection plan, carried it out and paid the physicians. An NYU Langone test in Nature Medicine the same month, with no company role, found the opposite with earlier models: on 100 real questions from NYU doctors, 12 blinded clinicians rated OpenEvidence 3.24 out of 4, below three general models (3.52 to 3.62) and level with Google Search's automatic AI Overview (3.27).
Achieved whena test with no vendor involvement, on a public question set not drawn from one tool's own users, finds OpenEvidence ahead.
Claimed by
J. Feng, A. B. Jena and colleagues, with OpenEvidence, June 2026
UpToDate Expert AI checks itself
Wolters Kluwer says the AI version of its UpToDate clinical reference gave clinically aligned information on 99.9% of more than 15,000 criteria across 1,669 test questions, and that two unnamed general AI models left out critical information 15% more often. The reviewers were the company's own physician editors and AI experts, and no outside party is named. In NYU's independent test, run in November 2025 and February 2026, UpToDate Expert AI declined to answer 19% of 100 real doctors' questions, against 1% to 3% for the general models and OpenEvidence, and scored 88.4% on 500 exam questions, against 90.2% to 97.4% for the general models.
Achieved whenan outside group scores the tool on a published question set and reports results close to the company's.
Claimed by
Wolters Kluwer, May 2026
AI checks for pharmacy orders
Singapore's National University Health System (NUHS) says MedVerify, one of four tools in its new National Cluster AI in Pharmacy platform, reviews medication orders for dosing, contraindications (reasons a drug should not be given) and drug interactions, and was about 95% accurate in pre-validation testing. NUHS pharmacists review about 6,600 orders a day; the platform aims to save up to 850 pharmacist hours a week, with rollout across the cluster planned from the end of 2027. No method, sample size or error counts were published.
Achieved whenNUHS or an outside group publishes MedVerify's catches and misses on live orders against pharmacists' own review, including how often it flags safe orders.
Claimed by
NUHS, August 2026
Open
01
An open order-check test
A public set of at least 200 test cases in the style of the Leapfrog test, using made-up patients only. Each case is a synthetic patient (allergies, kidney function, age, current drugs) plus one draft order, labeled "should alert" or "should not alert" by two pharmacists. Cases cover the test's 10 published categories: drug-allergy, drug-drug, wrong route, single dose, duplicate therapy, drug-lab result, daily dose, drug-age, drug-diagnosis and a missing companion order, and at least 40% are safe orders, so that needless alerts are counted. Each case is packaged as a FHIR record bundle sent through CDS Hooks, the HL7 standard by which a record system calls an outside advice service, at the "order-sign" moment when a clinician signs orders. The same cases then run against any order-checking software or AI model, and the scores are published.
The briefCredit · unclaimed02
AI-written guideline rules, checked
Give AI models the text of the 12 recommendations in the US CDC's 2022 guideline for prescribing opioids for pain, and ask each to write them as CQL (Clinical Quality Language), the HL7 language for decision rules that a computer runs against FHIR records. Run the AI-written rules and an existing hand-written version on the same made-up patients, list every patient for whom they give different advice, and have a clinical informatics reviewer decide which version is right in each case. No real patient records are used.
The briefCredit · unclaimed03
A standing dose-answer check
A public set of at least 200 written dosing questions of the kind clinicians ask AI answer tools: dose changes for poor kidney function, weight-based doses for children, and switching a patient from one drug to another. No real patients are involved. Each answer in the key cites the section of an FDA label or guideline it rests on and is agreed by two pharmacists. Every three months, put the same questions to named clinician tools, such as OpenEvidence and UpToDate Expert AI, and to general AI models; score whether each dose is right and whether the cited source says what the answer claims; publish the scores with the dates of each round.
The briefCredit · unclaimed
Open 01 · the brief
An open order-check test
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public set of at least 200 test cases in the style of the Leapfrog test, using made-up patients only. Each case is a synthetic patient (allergies, kidney function, age, current drugs) plus one draft order, labeled "should alert" or "should not alert" by two pharmacists. Cases cover the test's 10 published categories: drug-allergy, drug-drug, wrong route, single dose, duplicate therapy, drug-lab result, daily dose, drug-age, drug-diagnosis and a missing companion order, and at least 40% are safe orders, so that needless alerts are counted. Each case is packaged as a FHIR record bundle sent through CDS Hooks, the HL7 standard by which a record system calls an outside advice service, at the "order-sign" moment when a clinician signs orders. The same cases then run against any order-checking software or AI model, and the scores are published.
Why it is still open
Leapfrog's tool is run by hospitals as part of its annual survey, and Leapfrog calls it "the only known tool of its type". PsiBench, a June 2026 preprint whose authors include David Classen, an author of the Leapfrog test's published results, scored 40 AI models on 492 order scenarios, but did not release the scenarios, to protect the benchmark, and tested no conventional order-checking software. RxSafeBench tests AI models on simulated consultations with multiple-choice answers, not on orders. Open-source record systems have no public test at all: OpenEMR's interaction check still answers "No interactions found" because the US National Library of Medicine switched off its free interaction service on 2 January 2024, a gap users reported in April 2026.
ratings.leapfroggroup.orgpmc.ncbi.nlm.nih.govmedrxiv.orgarxiv.orgcommunity.open-emr.orglhncbc.nlm.nih.gov
Built with what exists
Synthea, MITRE's open-source generator of made-up patient records in FHIR R4; the CDS Hooks order-sign hook; the 15 high-priority interacting drug pairs that an expert panel said every record system should flag and FDA drug labels from openFDA as the basis for the expected answers; OpenEMR and AHRQ's open-source CQL Services, which answers CDS Hooks calls with rules written in CQL, as conventional engines; AI models through their APIs, including an open-weight model such as Mistral Large 4 (see the Note "A trillion open parameters").
synthetichealth.github.iocds-hooks.orgeuropepmc.orgopen.fda.govgithub.com
First step
Write 20 cases, two per category, on Synthea patients, including the 15 high-priority pairs, and run them against a local OpenEMR install and one AI model.
Done looks like
At least 200 cases, each labeled by two pharmacists with their agreement reported; at least 3 conventional engines and 5 AI models scored in every category on two numbers, the share of harmful orders flagged and the share of safe orders passed without an alert; cases, code and scores published under CC BY 4.0 and rerun when an engine changes.
Open 02 · the brief
AI-written guideline rules, checked
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
Give AI models the text of the 12 recommendations in the US CDC's 2022 guideline for prescribing opioids for pain, and ask each to write them as CQL (Clinical Quality Language), the HL7 language for decision rules that a computer runs against FHIR records. Run the AI-written rules and an existing hand-written version on the same made-up patients, list every patient for whom they give different advice, and have a clinical informatics reviewer decide which version is right in each case. No real patient records are used.
Why it is still open
The hand-written version exists: an implementation guide released under CC0 with CQL for all 12 recommendations and a test suite. The nearest AI work, PathWISE (May 2026), used AI agents to turn five NHS cancer pathway flowcharts into CQL and reported that all of it compiled and covered every path, but it did not compare its answers with hand-written rules on patients. GuidelineForge, a NeurIPS 2026 workshop paper by Kaavya Kumar and Timothy Torubarov, turns open-access guidelines into CQL in which every rule cites the sentence it came from; its abstract reports no comparison with hand-written rules and its page links no code. A Taiwan team that hand-built 39 CQL libraries for national quality indicators (JMIR Medical Informatics, October 2026) lists patient-level testing as still to be done. AHRQ's CDS Connect repository of shared rules went offline on 28 April 2025. A search found no study that scores AI-written CQL against a hand-written version patient by patient.
Built with what exists
the guideline text in CDC's MMWR; the CC0 opioid CQL and test data from cqframework; cql-execution, an open-source JavaScript engine that runs CQL, and the CQL-to-ELM translator that checks it compiles; AHRQ's CQL-Testing-Framework for writing test cases; Synthea patients, with pain diagnoses and opioid prescriptions added by script where needed; AI models through their APIs.
First step
Run one hand-written recommendation from the guide on 100 Synthea patients with cql-execution, then ask one model to write the same rule from the guideline text and compare the two outputs patient by patient.
Done looks like
For all 12 recommendations and at least 3 AI models, CQL that compiles and runs on at least 1,000 synthetic patients; agreement with the hand-written rules reported for each recommendation; every disagreement classed as an AI error, a hand-coding error or unclear guideline text; all code and results public.
Open 03 · the brief
A standing dose-answer check
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public set of at least 200 written dosing questions of the kind clinicians ask AI answer tools: dose changes for poor kidney function, weight-based doses for children, and switching a patient from one drug to another. No real patients are involved. Each answer in the key cites the section of an FDA label or guideline it rests on and is agreed by two pharmacists. Every three months, put the same questions to named clinician tools, such as OpenEvidence and UpToDate Expert AI, and to general AI models; score whether each dose is right and whether the cited source says what the answer claims; publish the scores with the dates of each round.
Why it is still open
Outside checks so far are small or one-off. Ipema and colleagues at the University of Illinois Chicago (JACCP, June 2026) put 30 drug questions to five chatbots once, and all scored below pharmacy faculty. Bergsbaken and colleagues at UW-Madison (JACCP, July 2026) gave two examples: OpenEvidence proposed a carvedilol dose two to four times higher than the equivalent of the metoprolol dose asked about, and about six months later it was still wrong on a second question, with changed sources. NYU's test used general questions, not doses, its set of real questions is not public, and it notes the clinical tools have no public API, so they had to be queried by hand in a browser. FDARxBench and DoseBench test general models on drug-label questions, not the clinician tools, and DoseBench covers only acetaminophen and ibuprofen. Use is already wide: OpenEvidence alone had 703,265 monthly active US clinician users in June 2026.
europepmc.orgpharmacy.wisc.edupmc.ncbi.nlm.nih.govarxiv.orgarxiv.orgeuropepmc.org
Built with what exists
FDA drug labels from openFDA; two paid pharmacist reviewers; clinician accounts or the makers' permission for each tool, queried by hand as in the NYU test; a small extraction model such as Meta Muse Ember to pull dose statements out of label text for the pharmacists to check (see the Note "The jobs agents repeat"); general AI models through their APIs; a shared scoring sheet.
First step
Write 25 questions from the labels of 10 drugs whose doses change with kidney function, have two pharmacists agree the key, and run them on two clinician tools and one general model.
Done looks like
At least 200 questions, with a key agreed by two pharmacists and their agreement reported; at least 4 named tools scored in each of 4 quarterly rounds; for each tool, the share of correct doses, the share of answers whose cited source supports them, and the change since the last round; questions, key and scores published under CC BY 4.0.