Explanation of Machine Learning Models Revealed Influential Factors of Early Outcomes in Acute Ischemic Stroke: A registry database study

JMIR Med Inform. 2022 Jan 24. doi: 10.2196/32508. Online ahead of print.

ABSTRACT

BACKGROUND: Timely and accurate outcome prediction plays a vital role in guiding clinical decisions in acute ischemic stroke. Early condition deterioration and severity after the acute stage are determinants for long-term outcomes. Therefore, predicting early outcomes is crucial in acute stroke management. However, how to interpret predictions and transform them into clinical explainable is as important as prediction itself.

OBJECTIVE: This work focused on machine learning model analysis in predicting the early outcomes of ischemic stroke and utilized model explanation skills in interpreting the results.

METHODS: Acute ischemic stroke patients registered to the Stroke Registry of the Chang Gung Healthcare System (SRICHS) in 2009 were enrolled for machine learning prediction of the two primary outcomes: modified Rankin Scale (mRS) at hospital discharge and in-hospital deterioration. Four machine learning models, Support Vector Machine (SVM), Random Forest (RF), Light Gradient Boosting Machine (LGBM), and Deep Neural Network (DNN), were compared with the Area Under Curve (AUC) of Receiver Operating Characteristic curves. Three resampling methods, Random Under Sampling, Random Over Sampling, and Synthetic Minority Over-sampling Technique, dealt with the imbalanced data. The models were explained by the ranking of feature importance and the SHapley Additive exPlanations (SHAP).

RESULTS: RF performed well in both outcomes (discharge mRS AUC 0.829 ± 0.018; in-hospital deterioration AUC 0.710 ± 0.023 on original data and 0.728 ± 0.036 on resampled data with Random Under Sampling for imbalanced data). In addition, DNN outperformed other models in predicting in-hospital deterioration on data without resampling (AUC 0.732 ± 0.064). In general, resampling contributed to the limited improvement of model performance in predicting the imbalanced data of in-hospital deterioration. The features obtained from the National Institutes of Health Stroke Scale (NIHSS), white blood cell differential counts, and age were key features for predicting discharge mRS. In contrast, NIHSS sum score, initial blood pressure, having diabetes mellitus, and features from hemogram were the top important features in predicting in-hospital deterioration. SHAP summary visualized the impacts of feature values on predicting each outcome.

CONCLUSIONS: Machine learning models are feasible in predicting early stroke outcomes. An enriched feature bank could improve model performance. Initial neurological and daily activity levels determined the activity independence at hospital discharge. In addition, physiological and laboratory surveillance aided in predicting in-hospital deterioration. The utilization of SHAP explanatory method successfully transformed machine learning predictions into clinically meaningful results.

PMID:35072631 | DOI:10.2196/32508