LLMs differ on thyroid nodule biopsy
Artificial intelligence models recommended biopsy in 46% to 74% of responses to clinical vignettes describing incidentally discovered thyroid nodules, compared with 58% among clinicians.
The study, published in The Journal of Clinical Endocrinology & Metabolism, evaluated large language models (LLMs) under clinical equipoise—uncertainty associated with a lack of consensus on the best clinical approach.
“Large language models differed meaningfully in biopsy recommendation patterns, with potential population-level consequences,” the authors wrote.
Vignette development
An endocrinologist developed 100 clinical vignettes, and five thyroid specialists reviewed them for clarity and plausibility. Each described a solitary or dominant nodule measuring 0.5 to 4 cm, with ultrasound features corresponding to American College of Radiology Thyroid Imaging Reporting and Data Reporting System (ACR TI-RADS) scores ranging from 0 to 12. Patient descriptions varied by age, sex, race, medical history, and treatment preferences. Ultrasound images were not included.
The researchers recruited 94 clinicians through professional networks. All reported treating at least 10 patients with thyroid nodules or thyroid cancer during the preceding year, with 53 (56%) having treated more than 100. Each clinician evaluated five randomly selected vignettes, choosing biopsy, regular ultrasound follow-up, or no follow-up.
For the primary analysis, each vignette was presented to GPT-5 five times, producing 500 responses. Additional analyses applied the primary prompt to Claude Sonnet 3.5, 4, and 4.5; DeepSeek R1; and Llama 3.2-90B. The primary analysis did not direct clinicians or models to follow a specific guideline. Responses were categorized as biopsy or no biopsy. A rule-based implementation of ACR TI-RADS served as a provisional benchmark.
Comparing models and clinicians
Clinicians recommended biopsy in 273 of 470 responses (58%), compared with 352 of 500 GPT-5 responses (70%). Across models, biopsy recommendations ranged from 46% for Claude Sonnet 3.5 to 74% for Llama 3.2-90B.
Prompt wording also affected GPT-5 recommendations. Instructions to follow American Thyroid Association guidelines produced biopsy recommendations in 74 of 100 responses (74%), compared with 60 of 100 (60%) when instructed to minimize unnecessary screening.
Clinicians and models were more likely to recommend biopsy for larger nodules and those with higher ACR TI-RADS risk scores. Differences across models and prompts appeared greater for nodules of intermediate size and risk, although the authors considered this observation hypothesis-generating because of limited precision and multiple comparisons.
Overall biopsy recommendation rates ranged from 52% to 61% across clinician subgroups defined by annual thyroid case volume and endocrinology fellowship training. Possible differences between fellowship-trained endocrinologists and other clinicians for large, low-risk nodules were considered hypothesis-generating.
Clinicians were less likely to recommend biopsy for older patients. Analyses of clinician, GPT-5, and Claude Sonnet 4.5 responses found no statistically significant association between race or ethnicity and biopsy recommendations.
Rates of agreement
Agreement was measured with Krippendorff alpha, which accounts for chance agreement among multiple raters. Agreement among clinicians was moderate (α = 0.435), as was agreement between clinicians and GPT-5 (α = 0.453) or Claude Sonnet 4.5 (α = 0.409).
Across five repeated queries using the primary prompt, GPT-5 gave differing biopsy recommendations in 11 of 100 vignettes (11%). Clinicians disagreed with one another in 50 of 100 vignettes (50%).
Examining discordant responses
An exploratory analysis focused on 18 vignettes with at least five unanimous clinician responses: 10 favoring biopsy and 8 opposing it. GPT-5 agreed with clinician consensus using the primary prompt in all 18 vignettes. Claude Sonnet 4.5 recommended against biopsy in 4 of the 10 cases favoring biopsy and recommended biopsy in 2 of the 8 cases opposing it. Llama 3.2-90B recommended biopsy in all eight cases in which clinicians unanimously opposed it.
Across the models evaluated in this exploratory analysis, 23 responses differed from clinician consensus. Each was reviewed by three endocrinologists, who unanimously classified eight responses as seriously flawed. Five were Llama 3.2-90B responses recommending biopsy despite nodules not meeting size or appearance criteria; three came from Claude Sonnet models.
Limitations and implications
The authors emphasized that brief vignettes without ultrasound images could not address variability in image interpretation. Richer clinical information might also have increased clinician agreement.
Results do not apply to newer models, other prompts, or models fine-tuned for thyroid management. Clinicians did not evaluate the same vignette repeatedly, preventing assessment of within-clinician variability. Models were required to provide one recommendation, although routine use would likely allow multiple management options.
The authors proposed that professional societies define acceptable ranges of practice and endorse standardized benchmark datasets and reporting measures. They also recommended that developers report model versions and prompt configurations and that health care institutions or independent groups audit deployed systems.
“Due in part to the unavailability of a gold standard, this study should not be taken as demonstrating adequate quality of general-purpose LLMs to manage thyroid nodules,” the authors wrote.
The authors reported no financial or special interest conflicts. James H. Flory, MD, MSCE, is employed by Regeneron Pharmaceuticals; the authors stated that this work does not overlap with his responsibilities there.
AACE Endocrine AI is published by Conexiant under a license arrangement with the American Association of Clinical Endocrinology, Inc. (AACE®). The ideas and opinions expressed in AACE Endocrine AI do not necessarily reflect those of Conexiant or AACE. For more information, see Policies.