Before a routine thyroid nodule visit, a clinician asks an artificial intelligence assistant to summarize the chart. The summary is fluent and specific: “The right thyroid nodule had benign cytology in 2024 and has remained stable on follow-up ultrasound since then.”

Nothing sounds alarming. The problem is that the patient never had a follow-up ultrasound, so stability cannot be determined from the record. The model inferred, invented, or misplaced a key clinical fact, then presented it as if it were documented. This is a hallucination, a recognized risk of large language models (LLMs) that underscores the need to verify their outputs in clinical settings.

LLMs and other artificial intelligence (AI) tools are increasingly being explored for clinical tasks such as drafting documentation, summarizing records, preparing patient communication, supporting clinical reasoning, and retrieving or organizing medical information. Although interest in clinical LLMs is growing, evidence from real-world clinical settings remains relatively limited. A recent systematic review found that much of the clinical LLM literature still does not use real-world patient data, with relatively few studies conducting prospective evaluations.¹

What are hallucinations?

A hallucination is AI-generated content that sounds plausible but is false, fabricated, or unsupported by the information provided to or retrieved by the model. For clinicians, the practical concern is when the model presents a claim as fact that is not supported by the material it was given or by the source it cites^2,3. However, not every incomplete or misleading output is a hallucination. An output may also be incomplete because relevant information was not made available to the model; this is a context or retrieval failure rather than a hallucination.

Clinically important hallucinations and related failures may include:

• Fabrication: inventing a fact, such as a follow-up ultrasound that never happened.

• Misattribution: assigning information to the wrong patient, date, clinician, or source.

• False sourcing: citing a guideline, paper, or recommendation that does not exist or does not support the claim.

• Unsupported inference: turning a possibility into a documented fact, such as “tolerated medication,” when the chart only says it was prescribed.

• Wrong application: applying a generally correct recommendation to the wrong patient context.

Why hallucinations happen and why they persist

LLMs are AI systems trained to generate human-like text from patterns in large amounts of written material. LLMs do not review a chart with a clinician’s knowledge, accountability, and patient-specific judgment. They generate text by estimating likely sequences of tokens, meaning words or parts of words, based on the prompt, available context, and patterns learned during training².

That is why they can sound fluent and still be wrong. If the chart includes a thyroid nodule and prior benign cytology, the model may add that the nodule was “stable on follow-up ultrasound” because that phrase commonly fits the pattern of a reassuring thyroid nodule summary. But if no follow-up ultrasound exists, stability cannot be determined. The sentence sounds clinically reasonable, but the claim is unsupported.

Many approaches can reduce hallucinations: clearer prompts, better context, retrieval from trusted sources, clinical tuning, and stronger product design. But because LLMs are built to generate plausible language rather than verify truth on their own, hallucinations are likely to remain a persistent issue^2,3.

How clinicians can identify and reduce hallucinations

Clinicians cannot eliminate hallucinations, but they can make them less likely and easier to catch. A practical approach has three parts: reduce risk through better context and prompting; detect remaining errors through verification; and escalate when an important clinical question cannot be confidently resolved.

Better prompting can reduce errors, but it does not make the output true. Clinically important AI-generated information should be verified before it is used to guide care, with the level of verification reflecting the uncertainty and potential consequences of error. These practices focus on what clinicians can do at the point of use, but they are only one part of safe clinical AI. Healthcare organizations and developers also remain responsible for ensuring that clinical AI systems are appropriately evaluated, monitored, and governed.

Reducing risk before generation

One lever clinicians can control is how they prompt the model. Effective prompts specify the task, provide the relevant context, set boundaries, and make uncertainty or missing information easier to see. Four practical approaches can help reduce unsupported output.

1. Provide the relevant context. Better context means giving the model the information necessary for the task, not simply writing a longer prompt.

Insufficient: “Summarize this patient’s thyroid nodule history.”

Better context: “Using the thyroid ultrasound reports, cytology results, prior endocrine notes, documented symptoms, and relevant risk factors, prepare a pre-visit summary of this patient’s thyroid nodule history. Separate documented findings from interpretation and clearly identify information that is missing or uncertain.”

2. Ask the model to expose uncertainty and evidence against the leading interpretation.

Insufficient: “Interpret this thyroid nodule history.”

Better: “Interpret this thyroid nodule history. Identify conflicting information, unresolved questions, and any findings that argue against the leading interpretation.”

This instruction may make uncertainty and conflicting information more visible, allowing the clinician to identify an incomplete or overly confident clinical narrative more readily.

3. Ask what information is missing before requesting a recommendation.

Insufficient: “What should I do next?”

Better: “What additional information would you need before recommending the next step for this thyroid nodule?”

Clinicians already think this way. The prompt just makes the model say so out loud.

4. Ask for verifiable sources, not just an explanation.

Insufficient: “Recommend management options.”

Better: “List possible management options. For each, identify the relevant guideline or authoritative source, describe the population it applies to, and state important contraindications or uncertainties. If you cannot identify a source, say so.”

Asking for a source is not the same as checking it. The clinician still has to confirm that the source exists, is current, and actually supports the claim.

Detect errors with Trace, Check, and Fit

We propose Trace, Check, and Fit as a practical clinician-facing framework for evaluating clinically important AI-generated information. Prompting may improve the structure and completeness of an AI-generated response, but it does not establish that the information is accurate and does not replace verification.

Trace: Can the claim be linked to a specific location in the medical record or another identifiable source?

Check: Does that source actually support the claim, and is the information current and reliable?

Fit: Does the information or recommendation apply to this patient and this clinical situation?

In the thyroid example, “stable on follow-up ultrasound” fails at Trace because no follow-up ultrasound exists in the record. It also fails Check because stability cannot be concluded from a single imaging study. Fit asks a different question: Even after the facts are corrected, does the next step make sense for this patient, given the nodule’s ultrasound features, symptoms, risk factors, prior biopsy history, comorbidities, preferences, and ability to complete follow-up?

Escalate: When the stakes require a person

Trace, Check, and Fit may not resolve every clinical question. In those situations, escalation may be necessary. Escalation means seeking independent verification using an established clinical protocol or seeking input from another qualified person, such as a pharmacist, specialist, or second clinician.

Independent review is particularly important when the case falls outside familiar practice, the available information remains incomplete or conflicting, or the potential harm of an incorrect decision is substantial. Clinician review is an important safeguard, but it is not infallible. In a randomized trial, even physicians trained in AI literacy demonstrated automation bias when presented with erroneous AI recommendations⁴.

This does not mean that every AI-supported decision requires independent review, but it does support using verification that is independent of the original AI output when the stakes are high, uncertainty remains, or the clinician cannot confidently resolve the question.

Hallucinations are unlikely to disappear completely from LLMs. Clinicians therefore need practical safeguards that distinguish documented facts from plausible-sounding inventions. Better prompting, systematic verification through Trace, Check, and Fit, and escalation when uncertainty remains can help prevent unsupported AI output from becoming clinical action.

Juan P. Brito, MD and Misk Al Zahidy, MS are with the Care and AI Laboratory, Knowledge and Evaluation Research Unit with the Division of Endocrinology, Diabetes, Metabolism, and Nutrition at Mayo Clinic in Rochester, Minnesota.

References

1. Chen SF, Alyakin A, Seas A, et al. LLM-assisted systematic review of large language models in clinical medicine. Nature Medicine. 2026;32:1152–1159.

2. Roustan D, Bastardot F. The Clinicians’ Guide to Large Language Models: A General Perspective With a Focus on Hallucinations. Interact J Med Res. 2025;14:e59823. doi:10.2196/59823.

3. Basu S, Huynh B. Mitigating hallucinations in healthcare AI: a systematic review of evidence-based strategies. BMC Health Serv Res. 2026. doi:10.1186/s12913-026-14851-1.

4. Qazi IA, Ali A, Khawaja AU, et al. Automation bias in large language model–assisted diagnostic reasoning among physicians trained in AI literacy: a randomized clinical trial. NEJM AI. 2026;3(5). doi:10.1056/AIoa2501001.