Why read the evidence, not the brochure
AI tools for healthcare are sold with strong claims. Some of those claims are backed by careful studies and some are not. For a clinic owner or practice manager, the useful skill is not knowing every tool. It is knowing how to tell a result that has been tested from one that has only been announced.
This ebook walks through four areas where published evidence now exists: screening, hospital alerts, clinical notes and patient communication. It then covers the data-protection duties that apply to all of them. In each area we say what the sources report, how strong the study design is, and where the result stops. This document is general information about published evidence. It is not medical, legal or regulatory advice.
Everything here comes from the sources listed at the end. We did not run our own studies. Where we give an opinion, we mark it as ours.
One theme runs through all of it. A good result in a trial is a result about a particular product, used in a particular way, by particular staff. It becomes a result for your clinic only if your tool, workflow and patients are similar enough, and the only way to find out is to measure it locally. That is why each section ends with limits as well as findings, and why the last section is about what the evidence does not tell you.
How to read clinical AI evidence
Not all evidence weighs the same. A short ladder helps when you read a claim:
| Type of evidence | What it can tell you | What it cannot |
|---|---|---|
| Vendor claim or press release | What the supplier says the product does | Whether it is true, or whether it holds in your setting |
| Regulatory clearance | A regulator judged the device met the route it applied under | That patients do better. A US 510(k) clearance means substantial equivalence to an existing device, per CIDRAP |
| Study on stored data | Whether a model can find patterns in past records | How it behaves with live staff and patients |
| Independent external validation | Whether the tool holds up outside its developer | Whether acting on it improves outcomes |
| Randomized trial in routine practice | Whether the tool changes an outcome compared with current practice | Whether it works in other places, for other groups, or with other products |
Two more questions apply to every row. Who ran the study, and who paid for it? And which outcome was measured: a convenient number, such as alerts raised, or one that matters, such as cancers missed or time spent per note?
Screening: AI-supported mammography reading
The strongest design in this ebook is the MASAI trial. According to Lund University, nearly 106,000 women in Sweden’s national screening program were split into two groups: standard screening with two radiologists reading each mammogram, or AI-supported screening.
In February 2025 the team reported that AI-supported screening detected 29% more cancers (338 against 262) and that radiologist screen-reading workload fell by 44%. A January 2026 follow-up in The Lancet, as summarized by Textbook of Digital Health, reported interval cancers, the cancers found between screening rounds, at 1.55 per 1,000 women with AI support against 1.76 in the control arm, a 12% reduction.
The AI did two jobs, per Lund University’s description: it scored each exam so that lower-risk exams went to one radiologist and higher-risk exams to two, and it highlighted suspicious areas. A radiologist still read every exam.
Limits the authors and sources state: the trial ran in one national program that already used double reading, and the benefit depends on the specific AI system and reading protocol. Interval cancers are a sign of missed cancers, not a measure of deaths, and the sources we read do not report mortality. This is evidence for one system in one program, not for AI mammography in general.
Hospital alerts: AI sepsis early warning
Sepsis is a life-threatening reaction to infection, and CIDRAP reports that each hour of delayed treatment reduces survival by 8%. In May 2026, Healthcare Dive reported that an AI sepsis early-warning system received FDA 510(k) clearance. The system reads electronic health record data and raises an alert that a clinician must confirm.
The main published evidence, as CIDRAP describes it, is a 2022 study of more than 764,000 patient encounters at five US hospitals, in which sepsis patients were 18% less likely to die in hospital when clinicians acted on the alerts in time. The timing claim of detection 2 to 48 hours faster than traditional methods is the company’s own.
Read it carefully. The study compares patients whose alerts were confirmed in time with those whose alerts were not, which does not by itself show that the alerts caused the difference. That is our reading of the design as Healthcare Dive describes it. Clearance by the 510(k) route means substantial equivalence to an existing device, not an outcomes study.
A different, widely used sepsis model was checked independently. A University of Michigan validation in JAMA Internal Medicine across 38,455 hospitalizations found 33% sensitivity and 12% positive predictive value. It did not detect sepsis in 67% of patients with sepsis, yet it generated alerts on 18% of all patients. The lesson is that alerts vary widely in quality, and a false-alarm rate matters as much as a headline benefit.
Most clinics will never run a sepsis model, but the same questions apply to any AI flag in a clinical workflow, such as a missed-appointment prediction or a follow-up prompt. Has it been validated on patients like ours? How many alerts are false? Who acts on each alert, and how fast? If a supplier cannot answer those three questions with numbers, you do not yet have enough to decide.
Clinical notes: ambient AI scribes
An ambient scribe records a visit with the patient’s consent, drafts a note, and leaves the physician to review, edit and sign. UCLA Health announced a randomized trial in NEJM AI that enrolled 238 physicians across 14 specialties between November 2024 and January 2025, comparing two scribe tools with usual documentation.
Per UCLA, one tool saved 41 seconds per note, a 9.5% improvement from 4:30 to 3:49. The other showed no statistically significant reduction against the control group. Both tools showed roughly 7% better burnout scores than controls.
UCLA also reports that notes occasionally contained clinically significant inaccuracies, such as omissions and pronoun errors, and that one mild patient safety event occurred. The trial ran at one academic medical center for about two months, and UCLA says longer, multi-institution studies are needed. The sensible reading is a modest and uneven time saving that needs careful human review of every note.
Recording a consultation also raises consent and data-protection questions, covered in the data section below. If you are considering a scribe, our own suggestion, not a finding from the studies, is to measure time per note and error types in your own clinic for a few weeks before and after, rather than relying on another site’s number.
Patient communication: what the assistant may and may not say
Routine front-desk questions about hours, prices, insurance and availability repeat constantly and can be answered from a clinic’s own approved information. Our whitepaper on AI patient communication for clinics covers this in depth. Unlike the three areas above, we have no outcome trial to cite here, so what follows is design guidance and our own suggestion, not a finding from the studies:
- Keep the assistant administrative: answer from approved information, book, reschedule, and hand clinical questions to staff with the conversation attached.
- Write three lists: always allowed topics, topics allowed only with fixed wording (such as urgent symptoms), and never allowed topics (diagnosis, treatment, dosage, result interpretation).
- Test the boundary on purpose with questions about a rash, a dose or a scan result. The right answer is a polite refusal and a handoff.
- Tell patients they are talking to an automated system, and decide what is stored, for how long, and who can read it.
Start with one channel and a short list of topics, and have staff review a sample of conversations each week before widening scope.
Protecting patient data
Every tool above handles sensitive information. In the United States, an AI vendor that creates, receives, maintains or transmits protected health information is generally a business associate. Under 45 CFR 164.502(e), a covered entity must obtain satisfactory assurance that the business associate will appropriately safeguard the information, documented in a written contract. The Security Rule’s technical safeguards cover access control, audit controls, integrity, authentication and transmission security. Some are labelled addressable, which means you assess and document a decision, not that they are optional.
In the United Kingdom, the ICO lists data concerning health as special category data, and its AI and data protection guidance is organized around accountability, lawfulness, transparency, accuracy, fairness, security and individual rights. NHS England’s DTAC is an assessment framework used when assuring digital health technologies at procurement. Transfers to a supplier outside the UK are restricted transfers under ICO guidance and need a transfer mechanism. Confirm the details for your own situation with your advisers. Our AI governance and data privacy checklist gives a general starting point.
What the evidence does not tell you
Across the four areas, the same limits recur:
- One tool, one setting. Each positive result belongs to a named system in a named program or hospital.
- Proxy outcomes. Interval cancers, alerts confirmed in time and seconds saved per note are useful signs, not proof of better health.
- Short follow-up. The scribe trial lasted about two months.
- Errors still occur. Notes contained omissions, and one sepsis model raised many alerts while missing most cases.
- People remain essential. Radiologists read every exam, clinicians confirm sepsis alerts, and physicians sign notes.
The pattern is consistent. The tools that have been tested are decision aids that sit inside a human workflow, and the benefits reported depend on people using them well. None of the sources we read supports the idea of an AI tool acting alone on clinical decisions.
Our own suggestion, not a finding from the studies: treat each AI feature as a pilot with a written success test, a named owner and a review date. If you want help scoping one, see our custom AI agent work or book a call.
Sources
- Lund University, AI-supported breast cancer screening: new results suggest even higher accuracy, February 4, 2025
- Lund University, AI support in breast cancer screening: fewer missed cancer cases, 2026
- Textbook of Digital Health, AI-supported mammography cuts interval cancers by 12% in first randomised trial of 100,000 women, January 29, 2026
- Lang et al., MASAI interval-cancer results, The Lancet, 2026, doi:10.1016/S0140-6736(25)02464-X (figures as summarized by the Lund University and Textbook of Digital Health pages above; we did not read the full paper)
- Healthcare Dive, Bayesian Health gets FDA nod for AI sepsis detection tool, May 12, 2026
- CIDRAP, FDA clears first AI-based early warning system for sepsis
- Wong et al., External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients, JAMA Internal Medicine, 2021
- UCLA Health, UCLA study finds AI scribes may reduce documentation time and improve physician well-being, November 26, 2025
- Lukac et al., Ambient AI Scribes in Clinical Practice: A Randomized Trial, NEJM AI, 2025 (figures as reported by UCLA Health; we did not read the full paper)
- Legal Information Institute, 45 CFR 164.502, Uses and disclosures of protected health information: general rules
- Legal Information Institute, 45 CFR 164.312, Technical safeguards
- ICO, What is special category data? (updated 9 April 2024)
- ICO, Guidance on AI and data protection (last updated 15 March 2023)
- ICO, International transfers, a guide (updated 15 January 2026)
- NHS England, Digital Technology Assessment Criteria (DTAC)
Frequently asked questions
Does the evidence show that AI works in healthcare?
It shows that specific tools, tested in specific settings, produced specific results. A Swedish randomized trial of about 106,000 women reported fewer interval cancers with AI-supported mammography reading, and a randomized scribe trial reported a modest time saving for one of two tools. Neither result can be assumed for other tools or settings.
Which AI uses have the strongest evidence?
Among the studies summarized here, the two randomized trials are the strongest designs: the MASAI mammography trial and a UCLA randomized trial of ambient scribes. The sepsis alert evidence is largely reported by the developers, and the patient-communication guidance rests on design principles rather than outcome trials.
What should a small clinic take from this?
Start with administrative tasks, ask any supplier for independent evidence and error rates, keep a person responsible for every clinical decision, and run a small supervised pilot before widening scope.
Is this medical or legal advice?
No. It summarizes published evidence and its limits. Requirements differ by country and practice type, so confirm legal and regulatory duties with your own advisers.
What you get:
- How to rank clinical AI evidence, from vendor claim to randomized trial
- What the MASAI mammography trial reported and where it stops
- What reports say about an AI sepsis alert and an independent check of another
- What a randomized scribe trial found on time saved and note errors
- Scope rules for a patient-communication assistant and a data-protection primer