A viewpoint published in JAMA Internal Medicine outlines how health systems may assess the return on investment (ROI) of clinician-facing artificial intelligence (AI) technologies as their use expands in clinical practice.

The opinion piece, authored by Lisa S. Rotenstein, MD, MBA, MSc, of the University of California in San Francisco, Robert M. Wachter, MD, also of the University of California, and David W. Bates, MD, MSc, of Brigham and Women's Hospital in Boston, Massachusetts, looked at AI technologies through the lens of efficiency and ROI. 

"Early evaluations of these technologies have largely assessed the accuracy of their outputs and their impact on clinician time, experience, and decision-making, with mixed results. These results have left health care system leaders unmoored as they struggle to determine which tools are worth the cost—in money, time, and political capital. These trade-offs are often framed in terms of return on investment (ROI)," the authors wrote.

Framework for Evaluating ROI

Clinician-facing AI tools are increasingly being introduced to support daily clinical work. Examples include AI-powered documentation assistants (“scribes”), tools that draft responses to patient messages in electronic health records (EHRs), systems that summarize medical records, radiology image analysis software, and AI-enabled clinical decision support. These tools are often intended to reduce administrative workload, streamline workflows, and support clinical decision-making.

Despite rapid adoption, determining whether these tools provide value for health systems remains challenging. Many early evaluations have focused on the accuracy of AI outputs or on how the tools affect clinician time and workflow, with mixed findings. As a result, health system leaders may find it difficult to determine whether the financial and operational costs of implementing AI technologies are justified.

The authors propose a framework for evaluating ROI that considers both measurable and less easily quantified benefits. Measurable benefits may include increased clinical efficiency, higher revenue associated with improved productivity, and reduced clinician turnover if AI tools improve the work experience.

Other potential benefits may be harder to measure but are still important. These include improved patient satisfaction, better clinician–patient communication, reduced malpractice risk, and reputational advantages for institutions adopting advanced technologies.

Costs should also be considered across several categories. One-time costs may include technology implementation, integration with existing EHR systems, and staff training. Recurring costs may involve licensing fees, maintenance, and ongoing technical support. Longer-term costs may include monitoring requirements, technology upgrades, and potential unintended consequences such as overreliance on AI systems. 

Multiple Stakeholders

The authors emphasize that ROI should be evaluated from the perspective of multiple stakeholders, including clinicians, patients, and health systems. In some cases, the organization paying for the technology may not directly receive all the benefits, which can complicate traditional ROI calculations.

Payment models may also influence how organizations evaluate these investments. Health systems operating under value-based payment structures may prioritize technologies that reduce unnecessary care, whereas those operating under fee-for-service models may focus on tools that improve patient throughput or access.

Because AI technologies continue to evolve rapidly, the authors recommend that health systems regularly reassess both benefits and costs as tools mature and as new data on their effectiveness become available and look to vendors for help with their analysis. Doing so, the authors stated, may require difficult choices about investment options from health systems but is ultimately worth the effort due to the potential upside. "We believe that AI offers many opportunities to improve care delivery and enhance clinicians' daily work," the authors wrote.

Dr. Rotenstein reported grants from the Agency for Healthcare Research and Quality; grants and nonfinancial support from FeelBetter Inc; personal fees from Phreesia and the American Medical Association; and stock options in Augmedix Inc and Eko Health. Dr Wachter reported personal fees from Commure and The Doctors Company and ownership of stock in several health technology companies. Dr Bates reported grants from the Agency for Healthcare Research and Quality, personal fees from AESOP and FeelBetter Inc, equity in several health technology companies, and a patent licensed to Brigham and Women’s Hospital.