A large language model trained on longitudinal electronic medical records and physician-guided instruction outperformed competing language models and physician groups on expert evaluations of complex clinical case vignettes, according to a study published in npj Digital Medicine. 

 The researchers proposed that “Seasoned doctors and the strategic use of EMR can augment the abilities of LLM, enabling it to more closely mimic the clinical acumen of healthcare practitioners."

They developed AI4Doctor, a 32-billion-parameter large language model (LLM) designed to support clinical reasoning by integrating de-identified electronic medical records (EMRs), medical literature, textbooks, and physician expertise during training. Unlike many medical LLMs trained primarily on textbooks and published literature, it also incorporated complete hospitalization records, enabling it to learn temporal clinical reasoning from admission notes and progress notes, laboratory results, medical orders, diagnostic records, vital signs, and discharge summaries. 

“We propose that a combination of direct knowledge transfer from seasoned doctors and the strategic use of EMR can augment the abilities of LLM, enabling it to more closely mimic the clinical acumen of healthcare practitioners," wrote lead author Yan Zhuang, MD, of the Medical Innovation Research Department at Chinese PLA General Hospital in Beijing, China, and colleagues. 

The model underwent continued pretraining on more than 3 million inpatient EMRs from nine medical centers within the Chinese PLA General Hospital system, followed by curriculum-based instruction tuning, physician preference optimization, and retrieval augmentation using similar EMRs and clinical guidelines. In physician evaluations of 100 complex clinical vignettes, AI4Doctor achieved the highest overall score among all participants, including competing LLMs and physician groups. 

Responses were evaluated using a multidimensional framework assessing overall clinical performance, adherence to evidence-based guidelines, clinical relevance, potential iatrogenic risk, and therapeutic value. AI4Doctor scored 80.52, outperforming the general-purpose LLM Qwen-34B (74.03), the medical LLM HuatuoGPT-34B (73.07), senior physicians (72.99), junior physicians (67.65), and intermediate physicians (64.17).  

The researchers also evaluated medical documentation across four tasks: diagnostic planning, first admission records, first postoperative records, and discharge summaries. AI4Doctor achieved the highest average score of 67.73, narrowly exceeding Qwen-Chat at 67.13 and outperforming HuatuoGPT-II at 59.03 and a medical-record generation baseline at 54.33. The model demonstrated its strongest performance on discharge summaries, with a score of 68.3. 

On CMB-Clin, a benchmark of case-based clinical reasoning, AI4Doctor achieved a mean zero-shot accuracy of 55.70 across physician, nursing, pharmacy, technician, and graduate examination categories, outperforming all medical LLMs and trailing only GPT-4. 

In GPT-4-based qualitative evaluations of CMB-Clin responses, AI4Doctor also achieved the highest average score (4.36), with improvements in fluency, relevance, completeness, and proficiency compared with other medical language models.  When the model retrieved similar EMRs and clinical guidelines during medical consultation tasks, AI4Doctor demonstrated larger gains than comparator models, with improvements of 8.71 points in ROUGE-L and 10.92 points in F1 score. 

An ablation study found that the model's Odds Ratio Preference Optimization (ORPO)—a physician feedback alignment method that trains the model to favor clinically appropriate responses—improved performance across eight clinician-rated domains: comprehensiveness, professionalism, accuracy, ethicality, relevance, timeliness, comprehensibility, and discussion of risks and adverse effects.

The final performance of the model was assessed using C-Eval, an extensive evaluation framework, and CMMLU, which rigorously tests the knowledge and reasoning skills of large-scale Chinese language models. The AI4Doctor model achieved scores of 70.39 for C-Eval and 81.20 for CMMLU, outperforming HuatuoGPT-II (67.21 and 76.20, respectively) and Qwen1.5-Chat (69.30 and 77.60). The study had several limitations, including reduced performance for rare diseases and limited handling of encounters exceeding the model's context window, which could incorporate only a single retrieved document during inference. The researchers noted that future work will evaluate knowledge-graph re-ranking, larger context handling, governance frameworks, and deployment strategies before broader clinical implementation. 

The authors declared no personal or financial conflicts of interest.  

(Editor’s note: The researchers noted that the study manuscript will undergo further editing before final publication, and there may be errors present that affect the content. All legal disclaimers apply.)