Yale New Haven Health Study Finds Wide Variation in Performance of Hospital Early Warning Systems

NEW HAVEN, CT (October 15, 2024) – A new study led by researchers at Yale New Haven Health System (YNHHS) and Yale School of Medicine has found significant differences in the accuracy of various early warning systems used to predict clinical deterioration in hospitalized patients. The study, published today in JAMA Network Open, compared six different early warning scores across over 360,000 patient encounters at five YNHHS hospitals.

Key findings include:

• eCART, the only tool of the six to use advanced machine learning, outperformed all other scores in identifying patients at risk of clinical deterioration.
• The National Early Warning Score (NEWS), a simple publicly available tool, performed better than two proprietary artificial intelligence systems - the Epic Deterioration Index and Rothman Index.
• There was wide variation in accuracy between different systems, with some providing much higher false alarm rates or less advance warning time than others.

The study is the largest published analysis to date of the Epic Deterioration Index, a commonly used early warning system. It found that at comparable thresholds, eCART would have identified over 300 more deteriorating patients than the Epic tool while alerting on nearly 48,000 fewer patients overall.

“Early identification of at-risk patients can be life-saving, but false alarms create alarm fatigue and waste valuable clinical resources,” noted Deborah Rhodes, MD, principle investigator of the study, chief quality officer, YNHHS and associate dean of quality at Yale School of Medicine.  “These results highlight the importance of rigorous comparison of these tools prior to deciding which to implement. Based on these findings, we chose to implement eCART across our seven hospitals in January of this year, integrating the tool with clinical pathways to support best practice response to an elevated eCART score for both nursing and providers. In our continued analysis of eCART’s performance following go-live, eCART has allowed not only earlier recognition of patients at risk for clinical deterioration but also more effective and coordinated team response.” 

Chenxi Huang PhD, co-author and associate research scientist at the Center for Outcomes Research and Evaluation (CORE) at Yale, noted that the study “utilized a large, multi-center dataset from various hospitals and applied a high level of methodological rigor, allowing for a direct head-to-head comparison of six scoring systems. Time-matching and bootstrapping techniques were employed to address limitations of prior studies that often rely solely on the highest score per encounter, which may not reflect real-world clinical use. Importantly, the scores were applied without any modifications for external validation, providing a more accurate assessment of their performance in practice.”

Because the eCART tool generates data on response and outcomes for every elevated score, YNHHS is using these findings to optimize Rapid Response structure and processes. “We embarked on this analysis to find the best tool for our patients,” said Dr. Rhodes. “Given that the hospitals in our health system range from large academic quaternary care hospitals to small non-teaching hospitals, our comparison of these tools is likely generalizable to a broad range of hospitals. But equally important to the choice of tool is the process each hospital puts in place to respond to the tool and continually learn from it.”  

Researchers Dana P. Edelson, MD, MS, Matthew M. Churpek, MD, PhD; Kyle A. Carey, MPH; Zhenqui Lin, PhD; Chenxi Huang, PhD; Jonathan M. Siner, MD; Jennifer Johnson, MSN, APRN; Harlan M. Krumholz, MD, SM; Deborah J. Rhodes, MD contributed to the study.

Yale New Haven Health (YNHHS), the largest and most comprehensive healthcare system in Connecticut, is recognized for advanced clinical care, quality, service, cost effectiveness and commitment to improving the health status of the communities it serves. YNHHS includes five hospitals – Bridgeport, Greenwich, Lawrence + Memorial, Westerly and Yale New Haven hospitals, several specialty networks and Northeast Medical Group, a non-profit medical foundation with several hundred community-based and hospital-employed physicians. YNHHS is affiliated with Yale University and Yale Medicine, the clinical practice of the Yale School of Medicine and the largest academic multi-specialty practice in New England. Yale New Haven Hospital is the primary teaching hospital of Yale School of Medicine. www.ynhhs.org