Big Data and AI Forum 2023-05-09

May 23, 2023 | General

The Department of Technology Partners (DTP) at Mount Sinai hosts a Big Data and AI Forum periodically. The latest event took place on May 9th, 2023 and covered a host of new machine learning projects, including:

  • Hospital Acquired Pressure Injury Risk Predictions (HAPI)
  • 48Hrs Discharge Planning Tool 2.0
  • HeartBEit: Integrating ECGs as language
  • BMEII AI Core

Below are some of the highlights from the Clinical Data Science team:

 

 

The Clinical Data Science team has deployed 14 applications throughout six hospital in the Mount Sinai Hospital System. These applications make over 10 million predictions a year in areas such as:

  • Malnutrition
  • Fall Prevention
  • Delirium Detection
  • Discharge Planning
  • Respiratory Insights
  • Patient Experience
  • Patient Health Deterioration
  • Vent Weaning
  • Oncology Infusion
  • Avoidable Admissions
  • Behavioral Health
  • Bed Census
  • Pressure Injury

The team presented new projects in two of these areas: pressure injury and discharge planning:

Project 1 – Hospital Acquired Pressure Injury Risk Prediction (HAPI)

Presented by Nhi Nguyen, M. Sc.

HAPI: definition

  • Pressure injury/ulcer: localized injury to the skin and/or underlying tissue, result from compression between a bony prominence and an external surface for a prolonged time
  • HAPI (hospital-acquired pressure injury): during an inpatient hospital stay
  • CAPI (community-acquired pressure injury): acquired outside the hospital

Challenges

  • Increased length of stay by 3-7 days [1]
  • Not reimbursed by CMS
  • Costly: $12K-40K per pressure injury [2]

Opportunities

  • Identify patients at highest risk of developing HAPI with a predictive machine learning model
  • Take preventative measures with WOCNs (Wound Care Nurses) to prevent HAPIs

 

Benchmark: the Braden Scale

  • Tool to assess and document patient’s risk for developing HAPIs (done at every shift for every patient)
  • Risk factors are rated on a scale from 1 to 4 and added
  • The total score indicates a patient’s risk for developing a HAPI

Historical Cohort

Inclusion Criteria:  

Age >= 18

Facility: MSH

Referred for a WOCN consult

Exclusion Criteria:

Patients with CAPIs (community-acquired pressure injury)

LOS < 2 days and > 180 days

Data:

WOCN consult notes

ADT

Vitals, Labs, Nursing Flowsheets

Source:   Epic, CERNER, Clarity

Data Timeframe: March 2018 to January 2023

 

Cohort Characteristic and Labeling Logic

Both data logics were validated via extensive chart reviews with WOCN from MSH, MSB and MSW

Gave ~98% accuracy

HAPI label Definition Label timestamp
1 Hospital-acquired pressure injury only First HAPI
0 No pressure injury during the visit Discharge date

Modeling and Results

Sampling Logic

Last observation selection:

For less frequent variables (e.g.: weight, blood culture)

Regular sampling:

For frequently available variables (e.g.: O2 saturation, respiration)

 

Best Model: XGBoost classifier

Data set Time Frame # visits # HAPIs (%) Threshold Sensitivity Specificity Precision Accuracy F1-Score AUC
Test 01/01/2018 – 01/15/2023 1776 314 (18%) 0.48 0.76 0.74 0.39 0.74 0.51 0.83
Hold-Out (MSH) 01/01/2016 – 12/31/2017 & 01/16/2023 – 02/12/2023 1820 197 (11%) 0.48 0.74 0.76 0.27 0.76 0.40 0.83
Hold-Out (non-MSH) 07/13/2021 – 03/17/2023 3831 204 (5%) 0.3 0.70 0.75 0.14 0.74 0.23 0.81

 

Next Steps

  • Deploy the application for Mount Sinai Hospital
  • Start the silent pilot and identify the best threshold for the active pilot units
  • Identify the acceptance criteria for scaling up across MSH units
  • Scale across MSH
  • Develop a scaling plan for and scale across MSB, MSW, MSM, MSQ, MSBI

 

Project 2 – 48Hrs Discharge Planning Tool V2.0: Optimizing Discharge Process Through Hybrid NLP-EMR Approach

 

 Optimization Opportunity and Main Goal

Challenges

  • Multidisciplinary Decision Making
  • Fragmented Teams
  • Prolonged LOS
    • Patient Safety: HAI, and Risk of Adverse Events
    • Patient Satisfaction
    • Resource Management : Service Quality
    • Resource Utilization and Cost of care

Opportunity: Optimize discharge process by identifying clinically ready patients 48hrs in advance

Goal: Develop a classifier to predict likelihood of clinically readiness within 48 hrs.

 

Historical Cohort Characteristics

 

Facility: MSH

 Data: ADT, Vitals, Labs, Nursing flowsheets, Progress Notes, Care Notes

 Inclusion Criteria: Age>=18, Patient Class: Inpatient, LOS>=48 hrs

Exclusion Criteria: PoC = ICU, Death Flag = Y

Source: CERNER, SCC, EPIC

Date of Case Manager Flowsheet: July 2019 to January 2022

 

Labeling and Sampling Logic

 

Label definition

Label Timestamp

Reference Timestamp

Positive

•”Clinically ready = Y”

•”Clinically ready = N” AND  “Discharge – Prediction <  3 days”

 

CM flowsheet timestamp

CM flowsheet Date

Negative

“Clinically ready = N” AND  “Discharge – Prediction >= 3days”

CM flowsheet timestamp

CM flowsheet Date

 

 

Historical Model Performance

 

Random Forest default threshold = 0.50

Dataset

Sample  Size

Positive Rate (%)

Sensitivity

(%)

Specificity

(%)

Precision

(%)

Accuracy (%)

AUC

Train​

9381

50

99.2

95.2

95.4

97.2

99.4

Test

5557

62

66.2

68.2

75.4

63.6

70.2

 

 

 Top 20 Predictors

 

Top 20 most important variables

1

Temperature (Oral)

11

Potassium

2

Diastolic BP

12

Glucose

3

Systolic BP

13

BUN

4

Albumin

14

Hgb

5

O2 Saturation

15

Platelet

6

Chloride

16

INR

7

Hospital SVC

17

Creatinine

8

Sodium

18

Age

9

WBC

19

Respiratory Pattern

10

Calcium

20

Admit Source

 

Hold-out Set Performance

 Random Forest based on balanced threshold

Dataset

Sample  Size

Positive Rate (%)

Threshold

Sensitivity

(%)

Specificity

(%)

Precision

(%)

Accuracy (%)

AUC

V1.0​

7074

79

0.64

57

55

83

56

0.59

V2.0

7074

79

0.51

61

63

86

61

0.67

 

 Next Steps

 

  •  Deploy in QA and measure the silent pilot performance
  • Develop upgrade plan for MSH
  • Upgrade the current engine to V2.0 in MSH