| Home | E-Submission | Sitemap | Editorial Office |  
top_img
Journal of Korean Neurosurgical Society > Volume 67(1); 2024 > Article
Li, Zhang, Hou, Wang, Li, Xu, Chen, Zhou, Wang, and Liu: Prediction Model for Unfavorable Outcome in Spontaneous Intracerebral Hemorrhage Based on Machine Learning

Abstract

Objective

The spontaneous intracerebral hemorrhage (ICH) remains a significant cause of mortality and morbidity throughout the world. The purpose of this retrospective study is to develop multiple models for predicting ICH outcomes using machine learning (ML).

Methods

Between January 2014 and October 2021, we included ICH patients identified by computed tomography or magnetic resonance imaging and treated with surgery. At the 6-month check-up, outcomes were assessed using the modified Rankin Scale. In this study, four ML models, including Support Vector Machine (SVM), Decision Tree C5.0, Artificial Neural Network, Logistic Regression were used to build ICH prediction models. In order to evaluate the reliability and the ML models, we calculated the area under the receiver operating characteristic curve (AUC), specificity, sensitivity, accuracy, positive likelihood ratio (PLR), negative likelihood ratio (NLR), diagnostic odds ratio (DOR).

Results

We identified 71 patients who had favorable outcomes and 156 who had unfavorable outcomes. The results showed that the SVM model achieved the best comprehensive prediction efficiency. For the SVM model, the AUC, accuracy, specificity, sensitivity, PLR, NLR, and DOR were 0.91, 0.92, 0.92, 0.93, 11.63, 0.076, and 153.03, respectively. For the SVM model, we found the importance value of time to operating room (TOR) was higher significantly than other variables.

Conclusion

The analysis of clinical reliability showed that the SVM model achieved the best comprehensive prediction efficiency and the importance value of TOR was higher significantly than other variables.

INTRODUCTION

Throughout the world, the spontaneous intracerebral hemorrhage (ICH) remains a significant cause of mortality and morbidity. Previous studies showed that excellent medical treatment, including timely removal of hematoma, management of blood pressure and glucose and control of intracranial pressure, can improve the outcome of ICH [6,10]. In recent years, it was demonstrated that evacuation of hematoma with minimally invasive methods could potentially increase microcirculation in the afflicted brain areas [7,8]. Several studies have revealed that individuals who had their surgery performed sooner have better results. Other research has found that surgery during the acute phase is related with a greater likelihood of postoperative rebleeding [15,19]. Consequently, the optimal time of ICH surgical evacuation to produce favorable results has yet to be determined. All in all, the factors influenced the outcome of ICH were still controversial. The majority of ICH predictor studies have used univariable and multivariable logistic regression analysis and the accuracy of these models is generally poor [5,22]. Consequently, it is essential to develop new models for evaluating the results of ICH utilizing new technologies, such as machine learning (ML).
The algorithms used in ML could extract patterns from huge datasets with numerous variables. Because of its capacity to execute complicated pattern recognition and uncover nonlinear contributions in big data sets, ML algorithms have the potential to enhance event and outcome prediction [4]. Once the result label is established, ML algorithms may autonomously optimize its parameters with minimal supervision. Unlike regression models, ML can deal with vast volumes of data and patient features while accounting for all of their relationships. As a result, ML algorithms have the potential to outperform regression models in terms of predicted accuracy.
The purpose of this retrospective study is to develop multiple models for predicting ICH outcomes using ML. We also examined the prediction accuracy of the ML models and selected the best model for clinical use.

MATERIALS AND METHODS

Subjects

Patient information was obtained from the Hospital’s computerized medical records. The Institutional Review Board (IRB) of Qingdao Hospital, University of Health and Rehabilitation Sciences (Qingdao Municipal Hospital) (ID : 082). Because the study was retrospective, the IRB did not require informed consent.
Between January 2014 and October 2021, we included ICH patients identified by computed tomography or magnetic resonance imaging and treated with surgery, including minimally invasive aspiration and standard craniotomies. We ruled out the following scenarios : 1) intracerebral bleeding caused by an aneurysm, arteriovenous malformation, or tumor; 2) patients suffering from severe end-organ failure, intracranial and systemic infections, and blood disorders; and 3) patients lost to follow-up (Fig. 1).
Age, sex, body mass index (BMI), systolic and diastolic blood pressure, midline shift, operation type and time, residual hematoma volume, and time to operating room, as well as premorbid chronic conditions (diabetes, hypertension, hyperlipidemia), were gathered.

Surgery and follow-up

The recommendations from the American Heart Association and American Stroke Association published in Stroke, 2015 were used to treat every patient [20]. Our hospital’s on-call neurosurgeons choose between doing a normal craniotomy or a minimally invasive aspiration employing neuron navigation (XPS Nexus; Medtronic, Minneapolis, MN, USA) depending on the patient’s circumstances. Every patient was monitored for at least 6 months. At the 6-month check-up, outcomes were assessed using the modified Rankin scale (mRS). In this investigation, we used the mRS score to categorize patient outcomes, with mRS 0-2 representing positive results and mRS 3-5 representing poor outcomes.

ML

In this study, four ML models, including Support Vector Machine (SVM), Decision Tree C5.0, Artificial Neural Network (ANN), Logistic Regression (LR) were used to build ICH prediction models. We evaluated each model’s performance using the area under the receiver-operator curve (AUC). The ML was performed with IBM SPSS Modeler 14.1 (IBM Corp., Armonk, NY, USA). A classifier known as SVM uses kernels to turn input data into a multidimensional hyperplane in order to distinguish between two groups. By deriving decision rules from the training data, decision trees predict class membership. The term "artificial neural network" (ANN) refers to a large parallel and linked network made up of straightforward adaptable components. LR is a binary dependent variable regression model (Fig. 2).

Statistical analysis

The categorical data were expressed as percentage and the continuous data were expressed as mean with standard deviation. We used the t test or Mann-Whitney U test for categorical data to compare the groups. Fisher’s exact tests or the chisquare test were applied to continuous variables. In order to evaluate the reliability and the ML models, we calculated the AUC, specificity, sensitivity, accuracy, positive likelihood ratio (PLR), negative likelihood ratio (NLR), diagnostic odds ratio (DOR). If the p-value was less than 0.05, we regarded the result as statistically significant. The statistical analyses were performed with IBM SPSS Statistics 26 (IBM Corp.).

RESULTS

Characteristics of the population

We identified 71 patients who had favorable outcomes and 156 who had unfavorable outcomes (Fig. 1). Between the two groups, there were no significant differences in terms of age, hypertension, sex, BMI, hyperlipidemia, hematoma side and location, residual hematoma volume, operation time, and type (Table 1).
However, two groups showed significant differences in the percentage of diabetes, systolic blood pressure (SBP), diastolic blood pressure (DBP), Glasgow coma scale (GCS), midline shift and hematoma volume. The details were showed in the Table 1. As the continuous data, we found there was no significant difference in the time to operating room (TOR) between two groups. Following that, we divided the cases into three categories : 1) <12 hours, 2) ≥12 and ≤36 hours, and 3) >36 hours. We discovered the patients who had surgery between 12 and 36 hours were more likely to have positive outcomes than the other groups (p<0.001) (Table 1).

ML models

We established four ML models. For the SVM model, the AUC, accuracy, specificity, sensitivity, PLR, NLR, and DOR were 0.91, 0.92, 0.92, 0.93, 11.63, 0.076, and 153.03, respectively. For the Decision tree C5.0, the AUC, accuracy, specificity, sensitivity, PLR, NLR, and DOR were 0.96, 0.87, 0.88, 0.87, 7.25, 0.148, and 48.99, respectively. For the ANN model, the AUC, accuracy, specificity, sensitivity, PLR, NLR, and DOR were 0.91, 0.91, 0.82, 0.97, 5.39, 0.037, and 145.68, respectively. For the LR model, the AUC, accuracy, specificity, sensitivity, PLR, NLR, and DOR were 0.77, 0.81, 0.73, 0.85, 3.15, 0.205, and 15.37, respectively. The results above showed that the SVM model achieved the best comprehensive prediction efficiency (Fig. 3 and Table 2).
We also calculated the variable importance values in the four models. For the SVM and LR model, we found the importance value of TOR was higher significantly than other variables. For the Decision tree C5.0 model, the sequence of importance value was midline shift, GCS, SBP and TOR. For the ANN model, the sequence of importance value was midline shift, TOR, DBP, and SBP. Moreover, the importance values of the above variables in the Decision tree C5.0 and ANN model were similar (Fig. 4).

DISCUSSION

In the study, we established four ML models including the SVM, Decision tree C5.0, ANN, and LR models. The analysis of clinical reliability showed that the SVM model achieved the best comprehensive prediction efficiency. For the SVM and LR model, we found the importance value of TOR was higher significantly than other variables. While, the importance values of the variables in the Decision tree C5.0 and ANN model were similar.
Spontaneous ICH is a form of stroke that is associated with a high death and disability rate. Regardless of therapy, physicians, families, and patients are most concerned with prognosis. Previous research has linked a variety of characteristics to a poorer prognosis following ICH, including advancing years, a lower GCS score, a greater hematoma volume and infratentorial hemorrhage [18,24]. Cheung and colleagues demonstrated that in a sample of 142 patients, the ICH score comprising the GCS, age, and ICH volume, helped to predict favorable prognosis with a sensitivity of 93.5% [3]. In 310 patients, Ruiz-Sandoval et al. [21] published an ICH score system with sensitivity of 70.0% for 30-day excellent outcome. Previously described prediction approaches, on the whole, concentrated on shortterm prognosis and had very poor accuracies. In our study, we found that the SVM model achieved the best comprehensive prediction efficiency and the importance value of TOR was higher significantly than other variables. While, the timing of surgery for ICH is still debatable. The previous study showed that the TOR<12 hours was early time windows [12]. Wang et al. [25] investigated ultra-early (or <7 hours), early (7-24 hours), and delayed (>24 hours) surgery in spontaneous ICH and discovered that surgery conducted within the 7-24 hours window resulted in the best outcome. Early surgery was linked to a greater risk of rebleeding, whereas delayed surgery was linked to severe consequences [11]. Moreover, in my previous paper, we found that the 21 hours of time to operation room was the turning point of outcome and the best outcome is obtainable at 12-36 hours surgery [14]. Consequently, we TOR was divided into three categories : 1) <12 hours, 2) ≥12 and ≤36 hours, and 3) >36 hours. Our results were consistent with previous studies.
ML techniques have been employed in medical research and frequently outperform traditional statistical models. ML has recently been used to the severity or outcome prediction model for neurological illnesses such as ischemic stroke and traumatic brain injury in research [2,13,23]. In aneurysmal subarachnoid hemorrhage, Maldaner et al. [16] discovered that ML models exhibit strong discrimination and calibration for a favorable functional result. Using decision tree analysis, Heschl et al. [9] discovered that patients with a high GCS score, less than 44.5 mL hematoma volume, and a relatively low premorbid mRS score have a changeable prognosis. However, the use of ML to predict outcomes following ICH is still uncommon. The majority of research looking for ICH predictors have used univariable and multivariable LR analysis. In general, the accuracy of these regression models is poor. In our study, we found that the SVM model achieved the best comprehensive prediction efficiency which was better than previous studies. SVM is a promising supervised ML that can learn from observing data sets, build complex models to capture the inherent relationships between input and output variables, and make data-driven predictions or decisions. It has been used successfully in a variety of biomedicine fields and has performed admirably [1,17,26]. The application of the SVM ML approach to predict the outcome of ICH was innovative in our studies. Our model achieved a sensitivity of 0.93, specificity of 0.92, and AUC of 0.91. This indicated that our model performed well in predicting the outcome of ICH and could be widely used in clinics.
There were several limitations to the study. To begin, this was retrospective research conducted at a single center. Large clinical data sets and multicenter validation are necessary to improve the model’s performance and to make precise comparisons among ML-based models. Furthermore, our study did not seek to pinpoint the precise reason of a particular good or bad result. Although our approach cannot replace clinical judgment, result prediction based on long-term human experience may exceed the best algorithm. In the future, more complex ML approaches will be applied to increase AUC.

CONCLUSION

The analysis of clinical reliability showed that the SVM model achieved the best comprehensive prediction efficiency than other models. For the SVM model, we found the importance value of TOR was higher significantly than other variables. Using ML techniques, we may be able to identify the factors that have been previously neglected and to develop new treatment methods to improve outcomes according to these factors.

Notes

Conflicts of interest

No potential conflict of interest relevant to this article was reported.

Informed consent

This type of study does not require informed consent.

Author contributions

Conceptualization : WW; Data curation : JZ; Formal analysis : XH, YW; Funding acquisition : ML, TL; Methodology : SL; Project administration : ZX; Visualization : FC; Writing - original draft : YZ; Writing - review & editing : ML

Data sharing

None

Preprint

None

Acknowledgements

This work was supported by the China’s government under grant of National Natural Science Foundation (#82001184&82001253).

Fig. 1.
Flow chart outlining inclusion and exclusion criteria. ICH : intracerebral hemorrhage, mRS : modified Rankin scale, SVM : Support Vector Machine, ANN : Artificial Neural Network, LR : Logistic Regression.
jkns-2023-0118f1.jpg
Fig. 2.
Modeling process using machine learning. Four machine learning models, including Support Vector Machine (SVM), Decision Tree C5.0, Artificial Neural Network (ANN), Logistic Regression (LR) were used to build intracerebral hemorrhage oucome prediction models.
jkns-2023-0118f2.jpg
Fig. 3.
The receiver operator characteristic curve analysis of machine learning (ML) models. We established four ML models. For the SVM, the Decision tree C5.0, ANN, LR models, and the area under the receiver operating characteristic curve were 0.91, 0.96, 0.91, and 0.77, respectively. LR : Logistic Regression, SVM : Support Vector Machine, ANN : Artificial Neural Network.
jkns-2023-0118f3.jpg
Fig. 4.
The variable importance values in the four models. For the SVM and LR model, we found the importance value of TOR was higher significantly than other variables. For the Decision tree C5.0 model, the sequence of importance value was midline shift, GCS, SBP and TOR. For the ANN model, the sequence of importance value was midline shift, TOR, DBP, and SBP. Moreover, the importance values of the above variables in the Decision tree C5.0 and ANN model were similar. SVM : Support Vector Machine, TOR : time to operating room, GCS : Glasgow coma scale, SBP : systolic blood pressure, DBP : diastolic blood pressure, ANN : Artificial Neural Network, LR : Logistic Regression.
jkns-2023-0118f4.jpg
Table 1.
Clinical summary of patients’ data
Characteristic Favorable outcome group (n=71) Unfavorable outcome group (n=156) p-value
Age (years) 62.24±7.79 61.76±12.47 0.070
Sex 0.298
 Male 49 (69.0) 95 (60.9)
 Female 22 (31.0) 61 (39.1)
Body mass index (kg/m2) 25.51±2.59 26.09±4.34 0.209
Diabetes 51 (71.8) 132 (84.6) 0.030
Hypertension 65 (91.5) 148 (94.9) 0.376
Hyperlipidemia 8 (11.3) 15 (9.6) 0.813
SBP (mmHg) 154.4±29.4 169.2±32.9 0.001
DBP (mmHg) 88.1±15.4 93.1±18.8 0.034
GCS 10.6±5.0 6.7±3.5 <0.001
Location 0.075
 Thalamus, Basal ganglia 32 (45.1) 95 (60.9)
 Frontal, temporal, parietal, occipital lobe 34 (47.9) 53 (34.0)
 Cerebellum 5 (7.0) 6 (3.8)
 Brainstem 0 (0.0) 2 (1.3)
Affected side 0.474
 Left 41 (57.7) 81 (51.9)
 Right 30 (42.3) 75 (48.1)
Midline shift (mm) 4.01±3.57 7.49±5.02 <0.001
Hematoma volume (mL) 42.56±26.62 62.17±36.47 <0.001
Residual hematoma volume (mL) 10.10±11.81 10.91±13.92 0.670
Operation type 0.622
 Craniotomy surgery 55 (77.5) 115 (73.7)
 Minimally invasive surgery 16 (22.5) 41 (26.3)
Time to operation room (hours) 26.83±22.26 29.66±47.09 0.539
Time to operation room group <0.001
 ≤12 hours 14 (19.7) 110 (70.5)
 >12 and <36 hours 46 (64.8) 15 (9.6)
 ≥36 hours 11 (15.5) 31 (19.9)
Operation time (hours) 153.03±57.13 165.71±70.86 0.187

Values are presented as mean±standard deviation or number (%). SBP : systolic blood pressure, DBP : diastolic blood pressure, GCS : Glasgow coma scale

Table 2.
The comparison of prediction effciency for four models
Item AUC Accuracy Specificity Sensitivity PLR NLR DOR
SVM 0.91 0.92 0.92 0.93 11.63 0.076 153.03
Decision tree C5.0 0.96 0.87 0.88 0.87 7.25 0.148 48.99
ANN 0.91 0.91 0.82 0.97 5.39 0.037 145.68
LR 0.77 0.81 0.73 0.85 3.15 0.205 15.37

AUC : area under the receiver operating characteristic curve, PLR : positive likelihood ratio, NLR : negative likelihood ratio, DOR : diagnostic odds ratio, SVM : Support Vector Machine, ANN : Artificial Neural Network, LR : Logistic Regression

References

1. Alabi RO, Youssef O, Pirinen M, Elmusrati M, Mäkitie AA, Leivo I, et al : Machine learning in oral squamous cell carcinoma: current status, clinical concerns and prospects for future-A systematic review. Artif Intell Med 115 : 102060, 2021
crossref pmid
2. Alghamdi NS, Zakariah M, Hoang VT, Elahi MM : Neurogenerative disease diagnosis in cepstral domain using MFCC with deep learning. Comput Math Methods Med 2022 : 4364186, 2022
crossref pmid pmc pdf
3. Cheung RT, Zou LY : Use of the original, modified, or new intracerebral hemorrhage score to predict mortality and morbidity after intracerebral hemorrhage. Stroke 34 : 1717-1722, 2003
crossref pmid
4. Choi RY, Coyner AS, Kalpathy-Cramer J, Chiang MF, Campbell JP : Introduction to machine learning, neural networks, and deep learning. Transl Vis Sci Technol 9 : 14, 2020

5. Ding W, Gu Z, Song D, Liu J, Zheng G, Tu C : Development and validation of the hypertensive intracerebral hemorrhage prognosis models. Medicine (Baltimore) 97 : e12446, 2018
crossref pmid pmc
6. Garg R, Biller J : Recent advances in spontaneous intracerebral hemorrhage. F1000Res 8 : F1000, 2019
crossref pmid pmc pdf
7. Hanley DF, Thompson RE, Rosenblum M, Yenokyan G, Lane K, McBee N, et al : Efficacy and safety of minimally invasive surgery with thrombolysis in intracerebral haemorrhage evacuation (MISTIE III): a randomised, controlled, open-label, blinded endpoint phase 3 trial. Lancet 393 : 1021-1032, 2019
pmid pmc
8. Hersh EH, Gologorsky Y, Chartrain AG, Mocco J, Kellner CP : Minimally invasive surgery for intracerebral hemorrhage. Curr Neurol Neurosci Rep 18 : 34, 2018
crossref pmid pdf
9. Heschl S, Bernard S, Andrew E, Smith K : Characteristics of paediatric patients with altered conscious state attended by road ambulances in a metropolitan area - an 8 year observational study. Australas Emerg Care 23 : 142-146, 2020
crossref pmid
10. Hostettler IC, Seiffge DJ, Werring DJ : Intracerebral hemorrhage: an update on diagnosis and treatment. Expert Rev Neurother 19 : 679-694, 2019
crossref pmid
11. Jussen D, Marticorena S, Sandow N, Vajkoczy P, Horn P : Ultra-early decompressive hemicraniectomy in aneurysmal intracerebral hemorrhage: a retrospective observational study. Minerva Anestesiol 81 : 398-404, 2015
pmid
12. Kellner CP, Schupper AJ, Mocco J : Surgical evacuation of intracerebral hemorrhage: the potential importance of timing. Stroke 52 : 3391-3398, 2021
crossref pmid
13. Liu G, Wu J, Dang C, Tan S, Peng K, Guo Y, et al : Machine learning for predicting motor improvement after acute subcortical infarction using baseline whole brain volumes. Neurorehabil Neural Repair 36 : 38-48, 2022
crossref pmid pdf
14. Liu M, Wang Z, Meng X, Zhou Y, Hou X, Li L, et al : Predictive nomogram for unfavorable outcome of spontaneous intracerebral hemorrhage. World Neurosurg 164 : e1111-e1122, 2022
crossref pmid
15. Luzzi S, Elia A, Del Maestro M, Morotti A, Elbabaa SK, Cavallini A, et al : Indication, timing, and surgical treatment of spontaneous intracerebral hemorrhage: systematic review and proposal of a management algorithm. World Neurosurg 124 : e769-e778, 2019
crossref pmid
16. Maldaner N, Zeitlberger AM, Sosnova M, Goldberg J, Fung C, Bervini D, et al : Development of a complication- and treatment-aware prediction model for favorable functional outcome in aneurysmal subarachnoid hemorrhage based on machine learning. Neurosurgery 88 : E150-E157, 2021
crossref pmid pdf
17. Matheny ME, Resnic FS, Arora N, Ohno-Machado L : Effects of SVM parameter optimization on discrimination and calibration for postprocedural PCI mortality. J Biomed Inform 40 : 688-697, 2007
crossref pmid pmc
18. Nemoto M, Masuda H, Sakaeyama Y, Okonogi S, Node Y, Ueda K, et al : Clinical characteristics of subarachnoid hemorrhage with an intracerebral hematoma and prognostic factors. J Stroke Cerebrovasc Dis 27 : 1160-1166, 2018
crossref pmid
19. Polster SP, Carrión-Penagos J, Lyne SB, Gregson BA, Cao Y, Thompson RE, et al : Intracerebral hemorrhage volume reduction and timing of intervention versus functional benefit and survival in the MISTIE III and STICH trials. Neurosurgery 88 : 961-970, 2021
pmid pmc
20. Powers WJ, Derdeyn CP, Biller J, Coffey CS, Hoh BL, Jauch EC, et al : 2015 American Heart Association/American Stroke Association focused update of the 2013 guidelines for the early management of patients with acute ischemic stroke regarding endovascular treatment: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke 46 : 3020-3035, 2015
pmid
21. Ruiz-Sandoval JL, Aceves-Montoya J, Chiquete E, López-Valencia G, Lara-López A, Sánchez-González M, et al : Hospital arrival and functional outcome after intracerebral hemorrhage. Rev Invest Clin 74 : 51-60, 2022
crossref pmid
22. Schlunk F, Chang Y, Ayres A, Battey T, Vashkevich A, Raffeld M, et al : Blood pressure burden and outcome in warfarin-related intracerebral hemorrhage. Int J Stroke 11 : 898-909, 2016
crossref pmid pdf
23. Tozlu C, Edwards D, Boes A, Labar D, Tsagaris KZ, Silverstein J, et al : Machine learning methods predict individual upper-limb motor impairment following therapy in chronic stroke. Neurorehabil Neural Repair 34 : 428-439, 2020
crossref pmid pmc pdf
24. Wang D, Wang W, Wang A, Zhao X : Association of severity and prognosis with elevated homocysteine levels in patients with intracerebral hemorrhage. Front Neurol 11 : 571585, 2020
crossref pmid pmc
25. Wang YF, Wu JS, Mao Y, Chen XC, Zhou LF, Zhang Y : The optimal timewindow for surgical treatment of spontaneous intracerebral hemorrhage: result of prospective randomized controlled trial of 500 cases. Acta Neurochir Suppl 105 : 141-145, 2008
crossref pmid
26. Zhang Y, Xia R, Lv M, Li Z, Jin L, Chen X, et al : Machine-learning algorithm-based prediction of diagnostic gene biomarkers related to immune infiltration in patients with chronic obstructive pulmonary disease. Front Immunol 13 : 740513, 2022
crossref pmid pmc
Editorial Office
1F, 18, Heolleung-ro 569-gil, Gangnam-gu, Seoul, Republic of Korea
TEL: +82-2-525-7552   FAX: +82-2-525-7554   E-mail: office@jkns.or.kr
About |  Browse Articles |  Current Issue |  For Authors and Reviewers
Copyright © Korean Neurosurgical Society.                 Developed in M2PI
Close layer