Multimodal artificial intelligence models that integrate multiple sources of clinical data are becoming increasingly common in medicine, but explainability methods have lagged behind model development, according to a scoping review published in npj Digital Medicine.

Across 82 studies, most multimodal artificial intelligence (AI) models used post hoc techniques that explain individual data inputs after a prediction is made rather than methods that reveal how information from multiple data sources contributes to the model's reasoning, according to the review authors.  

They examined explainable artificial intelligence (XAI) methods used in multimodal AI models for disease detection. Multimodal models combine information from more than one data source—such as medical images, electronic health records, genomics, structured clinical information, or clinical text—to generate predictions. Although these models may improve predictive performance, their increasing complexity can make it more difficult for physicians to understand how predictions are generated, raising concerns about accountability, trustworthiness, safety, and clinical adoption. 

Researchers led by a team at the University of Sydney in Australia conducted a scoping review to characterize the explainability methods currently used in multimodal medical AI, evaluate how those methods are assessed, and identify gaps that could limit clinical translation. Rather than evaluating diagnostic performance of specific models, they focused on how multimodal systems explain their predictions and whether those explanations can be verified or reproduced. 

The investigators searched Scopus, PubMed, Web of Science, and IEEE Xplore for English-language, peer-reviewed studies published through October 2025. Of 2,521 records identified, 82 studies met the inclusion criteria after screening according to PRISMA Extension for Scoping Reviews guidance.  

Models most commonly integrated either two (33 studies) or three (35 studies) data modalities. Magnetic resonance imaging–based configurations were the most common, appearing in 38 studies and often combined with clinical, genomic, pathologic, positron emission tomography, or other imaging data. Neurology accounted for the largest proportion of clinical applications, whereas metabolic and endocrine disorders were represented by only one study. 

To organize the field, the authors categorized explainability approaches into four groups based on the depth of multimodal interaction and the type of reasoning produced. The largest group comprised Independent Feature Attribution approaches (37 studies), which assign importance scores to individual inputs using methods such as SHAP, Gradient-weighted Class Activation Mapping (Grad-CAM), or Local Interpretable Model-agnostic Explanations (LIME).  

A second category, Cross-Modal Alignment (30 studies), used attention mechanisms to visualize relationships between different data types. Less common were Concept-Based Reasoning methods (13 studies), which map raw inputs to clinically meaningful concepts before generating a prediction, and Holistic Multimodal Reasoning approaches (two studies), which generated integrated explanations across modalities, including natural-language reasoning and visual grounding. 

According to the authors, the predominance of independent feature attribution reflects a divergence between the rapid development of multimodal AI architectures and slower progress in explainability methods. Although these techniques can identify influential features within individual data types, they generally do not explain how different modalities interact to reach a diagnostic conclusion. Emerging approaches, including concept-based reasoning and holistic multimodal reasoning, aim to address that limitation by embedding transparency directly into model architecture rather than relying solely on post hoc explanations generated after prediction. 

Only 27 of the 82 included studies quantitatively compared explainability between unimodal and multimodal AI approaches. Among those studies, 18 reported improved localization or explanation performance with multimodal methods, whereas nine found that interpretability remained comparable, stagnated, or involved a trade-off despite greater model complexity. 

The review highlighted one example in which adding radiology reports to chest radiographs improved agreement between model attention maps and expert-annotated pathology by more than 11% compared with an imaging-only approach. Another study found that a large multimodal model using visual grounding generated more precise explanations by linking textual justifications directly to image regions.  

Across the review, post hoc explainability methods predominated, with 51 of 82 studies applying techniques after a model reached a decision. The authors classified the remaining 31 as using explanation methods derived from model architecture, including attention mechanisms, prototype networks, rule-based or counterfactual frameworks, and other architecture-specific designs.  

Evaluation methods differed considerably across studies. The authors reported qualitative assessment in 52 studies and quantitative metrics in 33, with 10 studies using both approaches. Qualitative evaluations typically involved inspection of heat maps or highlighted features, whereas quantitative assessments used measures such as Intersection over Union, Dice coefficient, or faithfulness. Only six studies conducted formal user evaluations involving medical professionals, and seven provided no explainability evaluation.  

Reproducibility also remained limited. Forty-seven studies used publicly available datasets, including MIMIC, the Alzheimer's Disease Neuroimaging Initiative, BraTS, and The Cancer Genome Atlas, but only 26 made their code publicly available. Thirty-three studies relied entirely on private or restricted datasets, limiting independent validation of reported methods. Two studies did not clearly report data availability. 

The authors identified several limitations affecting both the reviewed literature and the review itself. Most included studies were retrospective and focused on model development or validation using historical data rather than prospective use in clinical workflows. Standardized explainability benchmarks were lacking, and there was little evidence that improved algorithmic explanation metrics translated into better clinical decisions, workflow efficiency, or clinician trust.

The review itself was limited to English-language, peer-reviewed publications through October 2025 and focused specifically on disease detection rather than broader clinical applications. As a scoping review, it mapped the existing literature without formally grading the evidence or calculating risk-of-bias scores. 

Looking ahead, the authors recommended moving beyond post hoc tools toward intrinsically explainable models, developing public benchmarks with ground-truth explanations, and conducting prospective evaluations in real-world clinical workflows.

The researchers acknowledged using Google Gemini 3 and ChatGPT 5.1 to assist with data extraction, with all extracted information independently verified by the investigators. They reported no conflicts of interest.  (Note: This article-in-press manuscript will undergo further editing before final publication and there may be errors present which affect the content. All legal disclaimers apply.)

Expert Insight

AACE Endocrine AI invited first author Kaiyuan Hu of the School of Computer Science at the University of Sydney in Australia to elaborate on the findings.  

Why does this study matter? 

While AI adoption in healthcare is increasing, most models act as “black boxes” that clinicians cannot easily verify and trust. Explainable AI (XAI) is critical to build clinical trust and prevent issues like “shortcut learning.” While prior literature has surveyed unimodal explainability or multimodal AI separately, our study is one of the first scoping reviews to map the specific intersection of XAI technologies designed for multimodal clinical data, which integrate multiple data sources such as medical images, electronic health records, laboratory results, and clinical notes. As a next-generation AI paradigm, multimodal learning enables models to capture complementary clinical information across modalities, supporting more comprehensive and context-aware decision-making. By analyzing 82 studies across 14 medical domains, we established a clear taxonomy and baseline for how explainability is currently implemented in multimodal disease detection. 

What data surprised you?  

What surprised us most was the dramatic disparity in endocrinology. Among all 82 included multimodal XAI studies, only one study focused directly on metabolic/endocrine disorders while two other studies related to endocrinology only considered it as a part of integrated disease detection. In contrast, fields like neurology dominated the collection with about 40 studies. Since endocrinology heavily relies on continuous, multi-stream data such as time-series continuous glucose monitoring, lab panels, and longitudinal EHRs, we were expecting higher representation. Furthermore, 62% of all surveyed studies still rely on “off-the-shelf” post-hoc tools (like SHAP or Grad-CAM) applied to modalities in isolation, which fail to explain true cross-modal interactions. 

How might the findings influence endocrinology clinical practice?  

Although studies directly related to endocrinology are sparse, in general, our scoping review demonstrates that parallel medical domains using similar data structures offer a proven roadmap. Techniques developed for longitudinal signal data or tabular clinical/EHR data can be directly transferred to endocrine care. Specifically, transitioning from isolated feature attribution toward interaction-aware mechanisms and concept-based reasoning will allow endocrinologists to see how clinical variables, lab metrics, and time-series data inform AI predictions, dramatically reducing the interpretive effort required from clinicians. More importantly, the findings highlight a necessary shift toward Type 4: Holistic Multimodal Reasoning. Instead of forcing endocrinologists to decode fragmented, abstract heatmaps or feature lists, it makes it possible to apply large multimodal architectures (using semantic fusion and visual grounding) to generate integrated, natural-language diagnostic explanations for complex endocrine care. 

Is there anything else you'd like to say about this work? 

We would like to encourage endocrinology researchers not to view this low related study count in our scoping review as a limitation, but as a frontier opportunity, especially given that emerging holistic multimodal reasoning architectures represent the future of human-centric clinical AI. We hope our framework helps bridge the gap between medical AI system developers and practicing endocrinologists to build inherently interpretable tools better tailored to metabolic health.