Electronic Health Record Activity Changes Around New Decision Support Implementation: Monitoring Using Audit Logs and Topic Modeling
Comprehensive Overview
Published in JAMIA Open in 2025, this study asked whether a machine-learning method could automatically detect how a newly implemented clinical decision support alert changed healthcare providers' day-to-day use of the electronic health record, without relying on labor-intensive surveys or direct observation. Researchers developed a monitoring approach called topic modeling, applied to raw EHR audit logs, the automatic system records generated every time a provider clicks, views, or documents within the record. They applied this method to a new tobacco-cessation support alert rolled out across five cancer clinics, two medical oncology, two radiation oncology, and one survivorship clinic, at a major academic cancer center, analyzing over 3,400 EHR sessions tied to more than 1,600 patient encounters spanning the 12 months before and after the alert's implementation. Four clinical experts reviewed and labeled 14 distinct EHR activities the model automatically identified from these logs, grouping them into four broader categories: CDS-focused work, CDS-related work, EHR modification, and patient data review. The topline result was that this topic-modeling approach successfully detected measurable, statistically significant shifts in providers' EHR activity patterns following the alert's rollout, demonstrating that automated audit-log monitoring can reliably flag real behavioral change at scale.
Outcomes and Implications
The single most important finding was that, within the hour after patient check-in, providers significantly increased both CDS-focused activity (a 40% relative increase) and CDS-related activity like reviewing alerts and patient records (a 72% relative increase) after the tobacco-cessation alert was implemented, while significantly decreasing time spent modifying the EHR (a 40% relative decrease) and reviewing patient data (a 15% relative decrease). Activity-level analysis showed these reductions were concentrated in reviewing patients' visit summaries and performing administrative billing tasks. The model's validity was confirmed two ways: CDS-related activity was significantly more prevalent in sessions where the alert actually fired compared with sessions where it did not, and a separate machine-learning classifier using only the identified activity patterns could distinguish before-check-in from after-check-in sessions with 82% accuracy and a 0.88 area under the curve. In the four-hour window before check-in, when the alert had not yet fired, EHR-modification activity rose sharply instead (a 163% relative increase) while data review fell, showing the shift in behavior tracked closely with actual alert timing rather than reflecting a general trend. The authors interpret the reduced time on other documentation as a plausible workflow trade-off, with providers reallocating effort toward alert-prompted tasks like discussing treatment and placing referrals. They conclude that this audit-log and topic-modeling framework offers a scalable way to detect real-world shifts in provider EHR behavior following new decision-support tools, potentially applicable to immunization, chronic disease, and cancer-screening alerts, and could guide more targeted qualitative follow-up during implementation.