A no-code artificial intelligence workflow developed an externally validated thyroid nodule ultrasound malignancy classifier using natural language instructions, demonstrating a potential route for clinicians without programming expertise to conduct medical AI research, according to a retrospective computational diagnostic study published in The Journal of Clinical Endocrinology & Metabolism. 

“Many physicians and clinician scientists have ideas for how AI could improve care, but they may not have formal training in programming or machine learning,” said AACE Endocrine AI Editor-in-Chief Johnson Thomas, MD, FSCE, FEAA, who conducted the study with Nikita Pozdeyev, MD, PhD, of the University of Colorado Anschutz in Aurora. “The study provides a proof of concept that sophisticated AI research may become much more accessible to physicians who understand the clinical problem but do not have traditional programming expertise.” 

An investigator-specified, no-code workflow 

The authors used Hugging Face ML-Intern, an AI agent that followed an investigator-specified analysis plan from natural language instructions and logged its decisions, configurations, and metrics. They noted that the workflow did not replace the clinical and statistical judgment required to design a study.  

Development used TN5000, an open-source dataset of 5,000 cropped thyroid nodule ultrasound images from a single institution in China. The predefined splits contained 3,500 training images, 500 validation images, and 1,000 test images. Overall, 3,574 images represented malignant nodules, and 1,426 represented benign nodules, with fine-needle aspiration cytology as the reference standard. 

The dataset was partitioned at the patient level to prevent leakage between splits, and an automated audit found no exact duplicate images or conflicting labels. 

The agent conducted a 14-trial hyperparameter search using validation area under the receiver operating characteristic curve (AUROC) to select the best-performing configuration. The final classifier used a ResNet-18 architecture and ultrasound images alone, without manually selected sonographic features. Calibration and threshold selection used only the validation set, with the threshold chosen to maximize specificity while maintaining at least 95% sensitivity. After the model and threshold were locked, the internal test set was evaluated once. 

The internal test AUROC was 0.94. At the locked probability threshold of 0.711, sensitivity was 90%, specificity was 81%, positive predictive value was 93%, and negative predictive value was 75%. 

External validation reveals subtype differences 

External validation included 232 thyroid nodules—64 malignant and 168 benign—from University of Colorado biobank participants. Eligible nodules measured at least 10 mm and had diagnoses confirmed by surgical histopathology or fine-needle aspiration cytology. Each nodule contributed transverse and longitudinal ultrasound images. 

On external validation, the model achieved an AUROC of 0.90. Because the development threshold was specific to its source cohort, the primary external analysis used a default probability cutoff of 0.50. At that cutoff, sensitivity was 92%, specificity was 68%, positive predictive value was 52%, and negative predictive value was 96%. The model correctly classified 59 of 64 malignant nodules and 114 of 168 benign nodules. 

Detection varied by cancer subtype. The classifier detected 43 of 44 papillary carcinomas, 10 of 12 follicular-variant papillary carcinomas, 4 of 5 medullary carcinomas, 1 of 2 follicular carcinomas, and the single oncocytic carcinoma. Missed cancers clustered among follicular-pattern tumors. The authors characterized these subtype findings as hypothesis-generating because the nonpapillary subgroups were small. 

Higher discrimination, less accurate risk estimates 

The authors compared the agentic model with a previously published Big Transfer ResNet-50 classifier evaluated on the same external cohort. The agentic model achieved an AUROC of 0.90 vs 0.83 for the benchmark, but the benchmark produced better-calibrated absolute risk estimates. The agentic model’s mean predicted malignancy risk was 52% vs an observed prevalence of 28%. Its Brier score was 0.187 vs 0.143 for the benchmark.  

Both models showed positive net benefit across clinically relevant thresholds in decision-curve analysis. However, the authors limited their comparative claim to discrimination rather than overall clinical utility and emphasized the need for local recalibration before interpreting predictions as absolute risks. 

Prospective evaluation needed 

The authors noted several limitations, including retrospective evaluation, single-center training data, a modest external cohort, small cancer-subtype subgroups, and differences in populations, scanners, and malignancy prevalence between datasets. External performance reflected nodules already selected for tissue sampling, limiting generalizability to an unselected clinic population. Cine clips were not evaluated, and demographic metadata were insufficient for performance analyses across age, sex, or racial and ethnic groups. Operator selection of the nodule region could also affect predictions. 

The authors released the model, code, and experiment log for noncommercial research and encouraged researchers to use the model as a baseline comparator and apply the framework to larger datasets. 

“External validation, calibration, prospective testing, transparency, and careful assessment of failure modes remain essential,” Dr. Thomas said. “But I think the immediate impact is less about putting this particular model into clinical practice and more about changing who can participate in developing medical AI.”  

The study received no specific grant from public, commercial, or nonprofit funding agencies. Dr. Thomas disclosed holding patents involving machine learning–based thyroid nodule risk stratification algorithms and reported nothing to disclose specifically regarding the study. Dr. Pozdeyev disclosed a patent combining AI-based and genetic risk evaluation for thyroid cancer diagnosis.