Development and Internal Validation of Machine Learning to Predict Postoperative Worse Functional Status after Surgical Treatment for Thoracic Spinal Stenosis

Introduction

Surgical treatment for thoracic spinal stenosis (TSS) is a high-risk procedure and confers a high risk for intraoperative neurological injury and postoperative functional decline [1–3]. The corresponding symptoms can range from muscle weakness to paraplegia, or even paraparesis [2]. The prevention of worse functional status after surgery was our goal [2].

Intraoperative neuromonitoring (IONM) facilitates early detection of neurological injuries by providing real-time spinal signals [4]. Multimodal IONM, by combining somatosensory-evoked potentials (SSEP) and motor-evoked potentials (MEP), presented excellent diagnostic [5,6] and moderate prognostic [6,7] value in TSS decompression surgery in our previous studies [6,8]. The quality of preoperative SSEP correlates well with the postoperative neurological recovery ratio in cervical spinal surgery [9,10]. MEP baseline quality was associated with postoperative prognosis [11]. However, there are limited data on the reliability of using the intraoperative SSSEP/MEP baseline for predicting postoperative neurological prognosis in TSS decompression surgery.

Furthermore, machine learning (ML) models have been shown to be effective in predicting postoperative functional improvement [12] or deterioration [13] in patients who underwent cervical and lumbar spinal surgery. However, due to the rarity of TSS and relative lack of clinical experience, there is limited literature on determining the predictive value and the corresponding predictors of worse functional status in TSS patients by ML algorithms containing multimodal IONM variables. Accurately identifying patients with TSS at higher risk for postoperative functional decline can be beneficial, facilitating more transparent communication regarding the risks associated with the surgery, as well as expectations for postoperative rehabilitation and planning.

Therefore, our primary endpoint was the development and validation of ML algorithms to predict worse functional status for patients who underwent surgical treatment for TSS. We also aimed to identify important predictors associated with postoperative functional decline. Our secondary endpoint was the prediction of postoperative neurological status assessed by the JOA score.

Material and Methods

ETHICS AND PATIENTS:

Our study was deemed exempt by the Ethics Committee and the Institutional Review Board of our hospital, as it solely relied on our de-identified retrospective research and the corresponding functional databases (IRB number 202201007l; registration at ChineseClinicalTrialRegistry.cn [ChiCTR2000032155]), and we did not change any standard clinical practice. All procedures remained strictly under the ethical standards of the Helsinki Declaration.

A total of 432 patients in our hospital who underwent surgical treatment for TSS between March 2010 and March 2022 were retrospectively identified in our electronic health record. Our inclusion criteria were as follows: (1) age >18 years; (2) an American Society of Anesthesiologists (ASA) status between I and III; and (3) radiologic findings showed evidence of thoracic spinal compression or stenosis. The exclusion criteria were as follows: (1) patients lacked IONM data (either preoperative SSEP or intraoperative SSEP/MEP baseline); (2) patients lost to 30-day and 6-month follow-up visits; (3) patients had stroke with residual neurological disorders or co-existing cervical and/or lumbar spinal disorders; (4) history of thoracic spinal surgeries; and (5) patients had more than 30% missing data.

ACQUISITION OF SSEP AND MEP: We chose the anterior tibialis or abductor hallucis muscles [11] to obtain MEP waveforms. Parameters for acquiring MEP signals were as follows: constant voltage of 220–360 V, multiple trains of 5–8 pulses, and duration of 300 μs. We set the bandpass filter between 10 and 1500 Hz. The time base was 100 ms.

The posterior tibial nerve was chosen to record SSEP signals. Parameters for obtaining SSEP were set as follows. The posterior tibial nerves were stimulated at 25 mA. Single-pulse stimulation was performed with a frequency of 1 Hz, and the duration was 200 μs. Furthermore, we placed surface-stimulating electrodes at the ankles. Needle electrodes were placed for signal recording according to the international 10–20 system of electrode placement.

On the day before surgery, only SSEP was collected in our electrophysiological center, which we defined as preoperative SSEP. However, we did not acquire MEP at the same time, since patients cannot tolerate transcranial MEP recordings without general anesthesia or proper sedation.

During TSS surgery, intraoperative SSEP and MEP baseline signals were acquired after the paraspinal muscle was exposed, and we defined them as multi-modal IONM baseline waveforms [14].

CRITERIA FOR THE POOR QUALITY OF IONM BASELINE WAVEFORMS: A poor SSEP waveform was defined as abnormal latency or amplitude or absent waveform, type III and type IV, according to Hu et al [9]. Furthermore, an amplitude less than 3.6 μV or absent waveform was described as a poor MEP [11,15].

A simultaneous abnormality in SSEP and MEP was defined as poor quality of SSEP and MEP, and an abnormality in either or both modalities was defined as poor quality of SSEP or MEP [16].

PREDICTIVE VARIABLES: The following data were collected as variables (Table 1): demographic data (age >50 years vs ≤50 years [reference]), sex (male vs female [reference]), body mass index ([BMI] >24 vs ≤24 [reference]), smoking history (yes vs no [reference]), previous spine surgery history (yes vs no [reference]), clinical symptoms and signs before the surgery with the preoperative JOA score (mild ≥16 [reference], moderate <16 and >12, and severe ≤12) [17], ambulatory status (assistance vs independent [reference]) [13], preoperative neurological deficits, including bladder deficit (yes vs no [reference]) [18], sensory deficit (yes vs no [reference]), and motor deficit in the lower extremities (yes vs no [reference]). The motor deficit was defined as manual muscle testing grade [19] of the lower extremities less than or equal to grade III [11], duration of symptoms (>6 months vs ≤6 months [reference]), operated level (T1–T4, T5–T8, and T9–T11 [reference]) [1], number of decompression levels involved (3 or more levels vs 1 or 2 levels [reference]), bleeding loss (≥400 mL vs <400 mL) [5], poor quality of preoperative SSEP (yes vs no [reference]) [16], and poor quality of intraoperative SSEP and/or MEP baseline (yes vs no [reference]), including poor quality of SSEP and MEP (n=13), poor quality of SSEP or MEP (n=63), and comorbidities (diabetes, hypertension, cerebral vascular disease, acute coronary disease, use of anticoagulants and/or antiplatelet drugs).

We defined intraoperative variables as follows: number of decompression levels involved, bleeding loss, and poor quality of intraoperative SSEP and/or MEP baseline. Other variables were defined as preoperative variables. Furthermore, the cutoff values of the demographic variables and intraoperative variables were judged to be clinically appropriate by all the authors [5] (eg, bleeding loss and duration of symptoms).

OUTCOMES MEASURES:

Our outcome was the dichotomized change in perioperative JOA score according to whether it deteriorated (worse functional status) [13] or did not deteriorate. It was evaluated by comparing pre- and postoperative physical examination findings and JOA scores at the following 3 time points: before surgery, 30 days after surgery, and 6 months after surgery.

ENDPOINTS:

The primary endpoint was the development and validation of an ML model to predict whether worse functional status would be present after TSS surgery at the 2 follow-ups, and the determination of the associated predictors of postoperative functional decline. The secondary endpoint was the prediction of postoperative neurological function measured by the JOA score.

Binary classification models were used to predict whether worse functional status was developed, and regression models were used to predict postoperative JOA scores. To assess the performance for worse functional status predicting models, the c-statistic and accuracy were calculated. Furthermore, to assess the performance for postoperative JOA score predicting models, the coefficient of determination (R²), root mean square error (RMSE), and mean absolute error (MAE) were used.

GUIDELINES: We followed the TRIPOD 22-item checklist [20] and guidelines for ML predictive models [21].

DATA PROCESSING: We split the eligible patients into training (70%) and testing (30%) sets. The training set was used for algorithm training. The independent testing set was used to assess the performance of the models. To prevent overfitting, we removed predictors with less than 10 positive samples [21]. We used multiple imputation algorithms to replace the missing data [22]. All of the variables were entered into the random forest algorithm, and a subset of features was identified by recursive feature selection in our final ML model. After using the recursive feature elimination method, ML model performance did not improve. All independent variables were incorporated into the ML model to maximize performance [12].

DEVELOPMENT AND VALIDATION OF THE ML MODELS:

Two main models were developed. In model A, only preoperative variables were used. In model B, preoperative variables and intraoperative variables were implemented in the model.

We used the binary classification models Naïve Bays, LightGBM, XGBoost, logistic regression, and random forest to predict whether worse functional developed. The calibration and discrimination of the model were internally validated by 10-fold cross-validation [23]. Furthermore, we used the regression models Naïve Bays, LightGBM, XGBoost, logistic regression, random forest, and Elastic Net to predict postoperative JOA scores at the 2 follow-up visits.

We assessed ML model performance by using the Scikit-learn library by discrimination (accuracy, c-statistic), calibration, and overall performance (Brier score). We performed all ML algorithms for data analysis in Python 3.6 (Python Software Foundation, Wilmington, DE, USA). The hyperparameters of the model were optimized automatically by Optuna. R², RMSE, and MAE were used to compare the performance of our regression models.

After hyperparameters tuning, we selected the best-performing model based on the c-statistic. Then, the model was assessed on the test set for internal validation.

Results and Evaluations

STUDY POPULATION:

A total of 433 patients who underwent TSS surgery were identified. However, 106 patients were excluded: 21 patients lacked IONM data, 15 patients were lost to the follow-up visits, 10 patients had coexisting cervical/lumbar spinal disorders, 10 patients had a stroke history with residual neurological disorders, 7 patients had a history of spinal surgery, 3 patients had an ASA status of IV or V, and 40 patients had more than 30% missing data. Finally, a total of 327 patients with TSS in our hospital who underwent surgical treatment for TSS between March 2010 and March 2022 were retrospectively identified in our electronic health record. Forty-seven (14.4%) patients were diagnosed with thoracic disc herniation, 20 (6.1%) with ossification of the posterior longitudinal ligament, 224 (68.5%) with ossification of the ligamentum flavum, and 36 (11.0%) with ossification of the posterior longitudinal ligament and ossification of the ligamentum flavum. Of the eligible 327 patients, 228 were randomly allocated to the training set and 99 to the testing set. Figure 1 shows the flowchart of our study. Demographic and preoperative data are described in Table 1.

Thirty-seven patients had worse functional status 30 days after surgery, and 16 patients achieved complete recovery during the 6-month follow-up. Twenty-one patients had postoperative functional decline, compared with before surgery, at the last follow-up.

ML MODEL IN PREDICTING WORSE FUNCTIONAL STATUS IN THE 2 FOLLOW-UP VISITS:

ML model performance predicting worse functional status at the 30-day and 6-month follow-up visits is summarized in Table 2.

At the 30-day and 6-month follow-up visits, random forest had the highest c-statistic (0.72, and 0.70, respectively) and accuracy (85.40%, and 81.34%, respectively) for predicting worse functional status in model A. However, when preoperative variables and intraoperative variables were implemented in the ML model (ie, model B), the best-performing model for the short-term prediction was XGBoost; its c-statistic value and accuracy increased to 0.83 and 88.17%, respectively. The optimal ML model for the long-term prediction was Naïve Bays; its c-statistic value and accuracy at the long-term prediction increased to 0.80 and 86.03%, respectively (Table 2).

Model B yielded better predictive value at the 30-day and 6-month follow-up visits. Therefore, the XGBoost algorithms in model B for predicting short-term worse functional status and the Naïve Bays algorithms in model B for predicting long-term worse functional status were selected for calibration and testing.

In internal validation (Table 3), to predict 30-day worse functional status in the XGBoost algorithm, the accuracy and c-statistic were 88.17% and 0.83, respectively. To predict 6-month worse functional status in the Naïve Bayes algorithm, the accuracy and c-statistic were 86.03% and 0.80, respectively, indicating good discriminative ability. Furthermore, both Brier scores (0.06 for the XGBoost algorithm, and 0.08 for the Naïve Bays algorithms) indicated good calibration of the predicted probabilities.

ML MODEL IN PREDICTING POSTOPERATIVE JOA SCORES IN THE 2 FOLLOW-UP VISITS:

We assessed regression model performance in predicting postoperative JOA scores by the R2, RSME, and MAE values, and summarized the results in Table 4. Random forest exhibited the highest R2 (0.74, and 0.63, respectively) for predicting postoperative JOA scores at short- and long-term follow-ups.

FEATURE IMPORTANCE:

Model B yielded the best prognostic value in predicting impending worse functional status (Table 2). Among all included features, poor quality of intraoperative SSEP/MEP baseline, poor quality of preoperative SSEP, motor dysfunction of the lower extremity, preoperative JOA score, and diabetes were identified as having the most importance in the short-term prediction ML models. Furthermore, poor quality of intraoperative SSEP/MEP baseline, poor quality of preoperative SSEP, duration of symptoms, operated level, and motor dysfunction of the lower extremity were identified as having the most importance in the long-term prediction ML models (Figure 2A, 2B).

Discussion

REASONS FOR BETTER PERFORMANCE OF THE ML MODELS:

A predictive model with a c-statistic between 0.80 and 0.89 is good, and between 0.70 and 0.79 is fair [24]. Khan et al [25] reported that the highest c-statistic was 0.74 when predicting worse functional status in surgery for degenerative cervical myelopathy. Maki et al [12] stated that the highest c-statistic was 0.75 when predicting neurological improvement after cervical spine surgery.

However, in our study, the highest c-statistic values for predicting short- and long-term postoperative functional decline were good. The reasons we achieved better performance of the ML models are probably multi-factorial. First, compared with cervical spine surgery, TSS is a high-risk procedure. The diameter of the thoracic cord is much smaller, which results in reduced blood supply at this level, making it more susceptible to intraoperative injuries and potentially worse functional outcomes [1]. Therefore, it is a necessity to perform TSS surgery under IONM to prevent and mitigate neurological injuries. In contrast, cervical spinal surgery is generally regarded as a lower-risk procedure. Previous studies have demonstrated that cervical spinal surgery without IONM can reduce the medical cost without adversely increasing neurological deterioration [26]. As a result, the authors [12,25] did not include IONM parameters as variables in their model development. Second, combined MEP with SSEP has a complementary role in providing real-time signal feedback from the spinal cord [5–7]. Furthermore, multimodal IONM has shown superior diagnostic accuracy [16,27,28] in detecting impending worse functional status and prognostic value [7] in predicting neurological recovery. Especially in high-risk spinal surgery, the sensitivity and specificity of IONM by combining SSEP and MEP are excellent [29]. Third, previous studies demonstrated that abnormalities in the preoperative SSEP baseline waveform can predict the progression of spinal myelopathy [9,10]. Poor quality of MEP baseline was correlated with worse postoperative prognosis [11]. Fourth, patients exhibiting poor quality in intraoperative SSEP/MEP baseline are more vulnerable to developing postoperative functional decline, due to a corticospinal tract with more damage before the operation that is less resilient against intraoperative hypotension, hypoperfusion, or neurologic injuries [30].

REASONS FOR BETTER PERFORMANCE OF MODEL B:

Although intraoperative SSEP baseline latency was longer than preoperative SSEP latency (P<0.05), no significant difference was found in amplitude between the 2 groups, as shown inTable 5. Furthermore, no significant difference was observed in the number of normal and abnormal SSEP waveforms between the 2 groups (Table 6).

SSEP assesses only the integrity of the spinal cord dorsal and posterior column, and MEP assesses the integrity of the anterior column and the lateral cord [31]. Moreover, changes in SSEP lag behind changes in MEP by 5 min [32] to 33 min [33] in spinal surgeries. MEP has become the criterion standard to evaluate cortico-spinal tract function, owing to the high sensitivity and specificity and the importance of the motor function [11,34]. Furthermore, it was demonstrated that SSEP alone leads to poor predictive and prognostic value in spinal surgeries [4,35].

Therefore, the better performance of model B can be mainly attributed to the addition of the intraoperative MEP baseline.

DISTINCTION IN VARIABLE IMPORTANCE BETWEEN SHORT-TERM AND LONG-TERM PREDICTIVE MODELS:

Orthopedic surgeons should pay special attention to variables significant in the long-term predictive model, because those variables harm the long-term spinal neurological function postoperatively.

Variables in the long-term predictive model included duration of symptom and operating level, which were not among the top 5 important variables in the short-term predictive model. Indicating those 2 variables can predict the long-term prognosis of TSS surgery. Specifically, a longer duration of symptoms suggests prolonged compression of the thoracic spinal cord, which can lead to irreversible neuronal damage, due to demyelination and necrosis of gray matter [36]. Furthermore, previous studies demonstrated that a longer symptom duration is an important predictor of long-term worse functional status after cervical spinal surgery [12,37]. Additionally, the upper- and middle-thoracic spinal cord (T1–8) has a smaller diameter of the spinal cord that receives less blood supply, making these patients less resilient to intraoperative hypo-perfusion, blood pressure fluctuation, or neurological injuries.

LIMITATIONS:

This study presents several limitations. First, it was a retrospective study, which can introduce potential bias into our data, including the patient selection, surgical indication, and follow-up time window. Second, our patients came from a single institution, which can limit the generalizability of our findings; thus, further testing with external datasets is necessary. The primary aim of this study was to provide a predictive tool for spine surgeons. Consequently, the model we internally validated will be made available for external validations. Third, the number of eligible patients was relatively small for an ML study, because TSS is a relatively rare disease clinically [28]. The small sample size can limit the optimal calibration of our models. Future investigations will focus on external validation studies with multi-institutions cohorts or even international cohorts to improve the performance of the models.

Conclusions

Our ML models yielded good prognostic value in predicting impending worse functional status in patients who underwent TSS surgery. The following variables were identified as significant predictors of worse functional status during follow-up visits: poor quality of intraoperative SSEP/MEP baseline, motor dysfunction of the lower extremity, poor quality of preoperative SSEP, duration of symptoms, and operated level. The surgical team can use the tool to assist with clinical decision-making and postoperative planning.