Who this guide is for
This guide is for clinic owners, practice managers and operations leads who have been offered an AI product and need a clear way to decide. It does not rank products. It gives a method you can apply to any of them. This document is general information about published evidence. It is not medical, legal or regulatory advice.
The method draws on published studies and regulator guidance, listed at the end. Where a step is our own suggestion rather than a finding from the studies, we say so.
Step 1: Define the job in one sentence
Write down what the tool must do, for whom, and how you will know it worked. “Reduce time spent writing notes” is a job. “Improve patient care” is not.
Then place the job on a risk scale. Administrative tasks, such as answering opening-hours questions or booking, carry less risk than tools that influence a clinical decision, such as flagging a patient as high risk. The more a tool shapes a clinical decision, the more evidence you should demand.
Step 2: Ask for evidence, then grade it
Ask the supplier for every study and report behind its claims. Then grade what you receive:
- Who ran it? A study run by the developer deserves a second look. In the sepsis alert example, the 18% lower in-hospital death figure was reported by the developers and their partners, as CIDRAP and Healthcare Dive describe it.
- What was compared? The strongest designs compare the tool with current practice. The MASAI mammography trial randomly assigned about 106,000 women to AI-supported or standard double reading.
- What was measured? Prefer outcomes that matter. MASAI looked at interval cancers, not only cancers detected.
- Where and for how long? The scribe trial ran at one academic medical center for about two months, and UCLA says longer, multi-institution studies are needed.
- Is it independent? Ask whether anyone outside the developer has tested the product on patients like yours.
If the supplier can only offer a brochure or a customer testimonial, record that as a finding.
Step 3: Check the numbers behind the headline
A single accuracy figure hides trade-offs. Ask for these, in writing:
| Ask for | Why it matters |
|---|---|
| Sensitivity (cases found) | A low value means real cases are missed |
| Positive predictive value (alerts that are right) | A low value means most alerts are false |
| Alert volume per 100 patients | Shows the workload the tool creates for your staff |
| Results by patient group | The sources we read did not report how results vary by age, density or other populations, so ask |
The independent check of one sepsis model shows why. Across 38,455 hospitalizations it showed 33% sensitivity and 12% positive predictive value, per the JAMA Internal Medicine validation.
Step 4: Check regulation and assurance
Ask what regulatory status the product holds in your country and what that status does and does not mean. A US 510(k) clearance, for example, means the FDA judged the device substantially equivalent to an existing one, according to CIDRAP.
In the UK, NHS England’s Digital Technology Assessment Criteria are used when assuring digital health technologies at procurement. Ask whether the supplier has completed it and ask to see the result. Regulatory duties differ by country and product type, so confirm yours with your own advisers.
Step 5: Check data protection before any patient data moves
In the US, a vendor that creates, receives, maintains or transmits protected health information is generally a business associate, and 45 CFR 164.502(e) requires satisfactory assurance, documented in a written contract, before disclosure. If a vendor wants patient data but will not sign, treat that as a stop sign. In the UK, data concerning health is special category data, and a supplier outside the UK usually means a restricted transfer under ICO guidance.
Ask where data is stored, whether it trains the vendor’s models, who can read conversations, how long it is kept, and how a breach would be reported. For the technical controls, see the Security Rule safeguards. A longer checklist is in our AI governance and data privacy whitepaper.
Step 6: Define human oversight and scope limits
Decide in writing who reviews the tool’s output and who is accountable. In the scribe trial, UCLA reports occasional clinically significant inaccuracies in notes, such as omissions and pronoun errors, so each note needs real review by the physician. In the MASAI trial a radiologist read every exam. The sepsis system requires clinician confirmation.
For patient-facing assistants, write what it may answer, what needs fixed wording, and what it must never do. Our patient communication whitepaper explains this boundary.
Step 7: Run a supervised pilot with a pass mark
Our own suggestion, not a finding from the studies:
- Pick one channel, one team or one clinic site.
- Write the pass mark before you start, such as a target time saving and a maximum tolerated error rate.
- Test with fictional or test data first where possible.
- Have staff review a sample of outputs every week and log every error.
- Set a review date. Widen scope only if the log supports it, and stop if it does not.
The question checklist
Take these to any supplier meeting. A plain answer in writing is a good sign. Vague or missing answers are a finding.
| Area | Question |
|---|---|
| Job | What exact task does the tool perform, and what does it not do? |
| Evidence | Which studies support it, who ran them, and were they compared with current practice? |
| Evidence | Has anyone independent tested it on patients like ours? |
| Accuracy | What are sensitivity, positive predictive value and alert volume? |
| Errors | What types of error have been seen, and how are they caught? |
| Regulation | What status does it hold here, and what does that status cover? |
| Data | Will you sign the required data agreement? Where is data stored? Is it used to train models? |
| Access | Who at the vendor can read patient conversations, and is access logged? |
| Oversight | Who reviews outputs, and can a human take over at any point? |
| Exit | Can we export and delete our data if we leave? |
If you would like a second opinion on a product you are considering, you can book a call or read about our custom AI agent work.
Sources
- Lund University, AI-supported breast cancer screening: new results suggest even higher accuracy, February 4, 2025
- Textbook of Digital Health, AI-supported mammography cuts interval cancers by 12% in first randomised trial of 100,000 women, January 29, 2026
- CIDRAP, FDA clears first AI-based early warning system for sepsis
- Healthcare Dive, Bayesian Health gets FDA nod for AI sepsis detection tool, May 12, 2026
- Wong et al., External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients, JAMA Internal Medicine, 2021
- UCLA Health, UCLA study finds AI scribes may reduce documentation time and improve physician well-being, November 26, 2025
- Lukac et al., Ambient AI Scribes in Clinical Practice: A Randomized Trial, NEJM AI, 2025 (figures as reported by UCLA Health; we did not read the full paper)
- Legal Information Institute, 45 CFR 164.502, Uses and disclosures of protected health information: general rules
- Legal Information Institute, 45 CFR 164.312, Technical safeguards
- ICO, What is special category data? (updated 9 April 2024)
- ICO, International transfers, a guide (updated 15 January 2026)
- NHS England, Digital Technology Assessment Criteria (DTAC)
Frequently asked questions
What is the first question to ask a healthcare AI supplier?
Ask what exact job the tool does and what evidence shows it does that job in a setting like yours. A general claim about accuracy is not enough. Ask for the study, who ran it, and which outcome it measured.
Is regulatory clearance enough to buy?
No. CIDRAP explains that a US 510(k) clearance means the FDA judged a device substantially equivalent to an existing one, so clearance alone is not an outcomes study. Treat it as one input and still ask for outcome and false-alarm data.
Why do false alarms matter so much?
An independent validation of one widely used sepsis model found it did not detect sepsis in 67% of patients with sepsis while alerting on 18% of all patients. High alert volume with low accuracy wastes staff time and can lead people to ignore alerts.
How long should a pilot run?
There is no single correct length. The randomized scribe trial summarized by UCLA ran about two months and its authors called for longer studies, so a pilot should run long enough to see error patterns and set review dates before you start.
What you get:
- A seven-step method from defining the job to a supervised pilot
- How to grade the evidence a supplier hands you
- The accuracy figures to request beyond a headline number
- Data protection and regulatory checks for US and UK settings
- A ten-row question checklist to take into supplier meetings