Agentic AI system may improve rare disease diagnosis 

Objective:

To evaluate the diagnostic accuracy of the DeepRare AI system for rare diseases, specifically comparing its performance against existing diagnostic tools and experienced physicians.

Key Findings:

• DeepRare achieved 57% Recall@1 and 65% Recall@3 on phenotype-based tasks, outperforming the second-best method (Reasoning LLM) by 24% and 19%, respectively, across 6,401 cases.
• In comparisons using human phenotype ontology and genetic data, DeepRare had a Recall@1 of 69.1% compared to Exomiser's 55.9% in 168 cases.
• The system maintained performance across heterogeneous datasets, achieving 29% Recall@1 in the MIMIC-IV dataset.
• DeepRare demonstrated higher diagnostic accuracy than five experienced physicians, with 64% Recall@1 vs 55% for physicians across 163 cases.
• Failure analysis revealed reasoning weighting errors (41%) and phenotypic mimic diagnosis (39%) as common causes of incorrect diagnoses.

Interpretation:

DeepRare shows promise as a valuable decision support tool for non-specialist physicians in diagnosing rare diseases, potentially improving diagnostic accuracy and efficiency across various medical specialties.

Limitations:

• Incomplete integration of available data sources.
• Difficulty distinguishing conditions with similar clinical features.
• Patient interaction features not fully validated.
• Intended primarily for patients already suspected of having a rare disease.

Conclusion:

DeepRare's advanced diagnostic capabilities could significantly aid in the timely and accurate diagnosis of rare diseases, though further validation and integration are needed to enhance its clinical utility.