Comparison of the Performance of 24 Early Warning Scores with the Updated National Early Warning Score (NEWS2) for Predicting Unplanned Intensive Care Unit (ICU) Admission in Postoperative Brain Tumor Patients: A Retrospective Study at a Single Center

Background

Postoperative patients with brain tumors in general hospital wards may suffer clinical deterioration due to underlying conditions [1,2]. Unnoticed patient deterioration and failure to intervene early may then cause severe adverse events (SAEs) [3], such as cardiac arrest or unplanned admissions to the Intensive Care Unit (ICU), which are associated with prolonged hospital stays, unfavorable prognosis, and even death [4,5]. Of all patients, however, those undergoing neurologic surgery have more than twice the average risk of perioperative mortality, according to Noordzij et al [6]. Even upon successful surgery, brain tumor patients may experience clinical deterioration within 72 h after surgery [7], so early recognition and timely intervention are essential for preventing in-hospital SAEs in this patient population.

According to one study’s statistics, 368 117 primary brain and other central nervous system tumors were diagnosed in the United States from 2009 to 2013 [8]. However, the characteristics of severe illness and rapid disease progression in postoperative brain tumor patients, along with some patients’ unconsciousness or unclear communication, make it difficult to detect patient deterioration [7,9]; consequently, SAEs can occur, which may contribute to a high mortality rate among these patients [6]. Nuño et al found that the mortality rate of brain tumor patients who suffer postoperative SAEs increases by 7.6 times [2], and Gold et al similarly reported that the in-hospital mortality rate associated with unplanned neurological ICU transfers was 17% [10]. These data thus urgently call for enhanced screening of these patients.

Despite the difficulties identifying them, research has shown that SAEs do not happen suddenly or are impossible to detect [2]. Some surveys have even suggested that about one-third of potentially preventable deaths in the U.K. involve poor clinical monitoring [11]. Substantial evidence from many studies has indicated that there are changes in physiological parameters in the hours preceding clinical deterioration [12,13]; therefore, SAEs could be prevented if changes in physiological parameters are monitored [14]. In response, early warning scores (EWSs), or track and trigger systems (TTS), using several physiological variables and established thresholds have been designed to identify patients at risk of deterioration at an early stage and have been adopted worldwide [15]. Gerry et al pointed out that due to the limitations in methods and reports of previous validation studies on various EWSs, the effect of various EWSs in clinical practice is inconsistent with that reported in the literature [16]. Therefore, based on the existing evidence, it is impossible to determine which EWS is the most effective and which is the most suitable for postoperative patients with intracranial tumors.

The National Early Warning Score is one of the most commonly used early warning system in the world [17]. In December 2017, the Royal College of Physicians updated the National Early Warning Score (NEWS) to the National Early Warning Score 2 (NEWS2) to standardize acute illness severity assessment in hospitals [17]. The NEWS2 includes 7 physiological parameters: respiratory rate, oxygen saturation, supplemental oxygen, systolic blood pressure, pulse, temperature, and consciousness level (Table 1). Based on NEWS, NEWS2 added a new assessment point: new confusion or delirium, to the alert, verbal, pain, unresponsive (AVPU) scale for the alert, new confusion, or delirium, voice, pain, unresponsive (ACVPU) scale; and NEWS2 set a new SpO2 scale for patients with known hypercapnic respiratory failure by adjusting oxygen saturation to 88–92% [18].

NEWS/NEWS2 is mainly used in hospitalized patients, but also has been evaluated in emergency and ambulance environments [19,20]. The application of NEWS/NEWS2 in patients with sepsis, acute pancreatitis, and liver disease has been reported [2042]. Tan et al used NEWS to evaluate the predictive value of unplanned ICU transfer and death in patients with acute pancreatitis [21]. Hydes et al used NEWS to evaluate the predictive value of unplanned ICU transfer, death, and cardiac arrest in liver disease patients within 24 h of observation [22]. In addition to independently evaluating NEWS/NEWS2, Smith et al compared NEWS with 33 other EWSs to analyze their predictive value in unplanned ICU transfer, death, and cardiac arrest of emergency patients [23]. Le Lagadec et al compared 12 EWSs, including NEWS, for predicting severe adverse events in in-hospital patients [24]. Silcock et al compared the predictive value of NEWS and quick Sepsis-related Organ Failure Assessment for ICU transfer and death of emergency patients [19]. Martín-Rodríguez et al reported the better performance of NEWS2 among 4 EWSs in predicting ICU admission of patients with brain trauma [25].

To the best of our knowledge, there have been few studies evaluating EWS systems to identify early clinical deterioration in patients following brain tumor surgery who are admitted to the ICU. The updated NEWS2 is an established method used in the National Health Service in the U.K. to improve care for in-hospital patients. Therefore, this retrospective study from a single center aimed to compare NEWS2 with 24 other types of EWSs to evaluate unplanned ICU admissions within 72 h after brain tumor surgery.

Material and Methods

ETHICAL CONSIDERATIONS:

Ethics approval was obtained from the Xiangya Hospital, Central South University Research Ethics Committee (201911778). The requirement for patient informed consent was waived because our study did not involve performing interventions. This research conformed to the provisions of the 1995 Declaration of Helsinki. Anonymity and confidentiality (access to records, data encoding, and information archiving) were guaranteed throughout the entire research process.

STUDY DESIGN AND SETTING:

This retrospective study followed a case-control design. The study was implemented in a university-affiliated tertiary hospital with 3500 beds in Hunan Province, China.

EARLY WARNING SCORE SELECTION:

Based on 33 EWSs reviewed by Smith et al [26], 24 EWSs were available when we searched the literature databases. In addition to the NEWS2, 24 other EWSs were assessed in our study (Table 2), including some commonly used EWSs like the Modified Early Warning Score [27], the NEWS [17], and the Patient-At-Risk (PAR) EWS [28,29].

PARTICIPANT SELECTION:

All patients who were diagnosed with a brain tumor and had an unplanned ICU admission within the first 72 postoperative hours in the neurosurgical general ward from June 1, 2014 to June 1, 2019 were reviewed for eligibility. Patients who had an unplanned ICU admission in the operation theater, anesthesia recovery room, or emergency department were not considered, nor were children (<16 years of age) and pregnant patients during the study period. The included patients with unplanned ICU admissions (cases) were matched to those without unplanned ICU admissions (controls) with the same hospital, admission year, ward, sex, and admission diagnosis based on the International Classification of Diseases, Tenth Revision (ICD-10) [30].

Per the formula n=(Zα/2/δ)2 P (1–P), we determined that the NEWS2’s sensitivity and specificity in the literature are 0.907 and 0.578, respectively [31]. The required case and control group sizes were estimated to be 60 and 83 samples, respectively, where α and δ were 0.05 and 0.10, respectively. We initially selected 92 cases and 271 controls. According to the exclusion criteria, 25 patients were excluded for being under 16 years of age, 9 were excluded for being transferred to the ICU from the emergency department, and 3 were excluded for being transferred to the ICU from the anesthesia recovery room. This yielded 69 cases and 257 controls for the study, which was adequate (see Figure 1). Of these 326 patients, 156 were male and 170 were female.

DATA COLLECTION:

Demographic and diagnostic information and physiological variables were collected from patient electronic medical records. The worst physiological variables prior to ICU admission within 72 h after brain tumor surgery were collected. These variables included body temperature, respiration rate, oxygen saturation, supplementary oxygen mode, heart rate, blood pressure, urine output, and level of consciousness. Unanticipated ICU admissions were identified from the ICU admissions database. For patients who experienced multiple unplanned ICU transfers during the same hospitalization, only the first admission was included in the analysis. Only fully recorded physiological variables were analyzed, though if there were missing data during the data collection, the latest recorded data were taken, as per Parshuram et al [32]. All data were collected and confirmed independently by 2 researchers.

STATISTICAL ANALYSES:

The physiological parameters collected were applied to the NEWS2 and other EWSs. All statistical analyses were performed using IBM SPSS Statistics version 22.0 (IBM, Armonk, NY, USA). The demographic data were analyzed using simple descriptive statistics and an independent-samples t test. The group significantly differed when P<0.05. Continuous data were summarized using means with standard deviations, and for categorical data, frequency tables were used.

The area under the receiver operating characteristic (AUROC) curve was analyzed by receiver operating characteristic (ROC) analysis to assess the ability of EWSs to distinguish between patients at risk and not at risk of having unexpected ICU admissions within 72 h after surgery [24]. An AUROC value of 0.7–0.8 is considered to have a fair discriminatory ability, and a value above 0.8 is considered good [33]. The Youden index values, optimal cutoff values, sensitivity, specificity, positive predictive values (PPV), and negative predictive values (NPV) were then calculated for comparison. The Youden index indicates an EWS’s total ability to identify patients at risk of experiencing unplanned ICU admissions and patients who are likely to have uneventful hospital stays; the higher the value, the better the EWS’s effectiveness and the greater the authenticity. The Youden index’s maximum value was chosen as the optimal cutoff value; that is, the point closest to the AUROC curve in the upper left corner with the lowest total number of false positives and false negatives. Sensitivity indicates the EWSs’ ability to accurately identify deteriorating patients – that is, those with high EWS scores – who may experience unplanned ICU transfers. Specificity represents the EWSs’ ability to accurately identify patients whose condition is not deteriorating (ie, patients with low EWS scores who may not require an ICU transfer) [24]. The PPV refers to the proportion of true positives in the total number of unanticipated ICU admissions the EWSs detected. The NPV represents the ratio of true negatives in non-deteriorating patients the EWSs identified. A value of P<0.05 was considered statistically significant.

Results

PATIENTS:

A total of 326 patients were included, of which 69 were case patients; 156 (47.90%) were male and 170 (52.10%) were female. The case patients’ average age was significantly higher than that of the control patients [52±16.32 vs 48±12.72, year, P=0.007], and their length of stay (LOS) was significantly longer than the control patients’ [23±10.92 vs 15±5.86, day, P=0.000]. Data on 4 co-morbidities were collected, of which the most common was hypertension (case=27.5%, control=19.1%). Heart disease significantly differed between the case and control patients (10.1% vs 2.7%, P=0.018). In addition, 9 types of brain tumor were identified, meningioma being the most common (case=34.78%, control=16.34%; Table 3). There were also significant differences in oxygen saturation (92.7±12.81 vs 97.9±1.56, P=0.000), systolic blood pressure (131.2±27.25 vs 116.5±19.53, P=0.000), heart rate (111.9±26.78 vs 94.4±18.78, P=0.008), respiratory rate (21.3±7.70 vs 19.6±5.86, P=0.000), temperature (37.7±0.90 vs 37.1±0.73, P=0.001), and consciousness (84.1% vs 23.7%, P=0.000) between the 2 study groups (Table 4).

PROGNOSTIC SCORE ACCURACY:

The AUROC curve, calculated from the total 25 EWSs using case-control (ie, unplanned ICU admission or no unplanned ICU admission) as a criterion, ranged from 0.737 to 0.870 (Table 5). The AUROC value for the NEWS2 (0.860, P=0.000) was higher than that of the NEWS (0.852, P=0.000). The PAR EWS had the highest AUROC value (0.870, P=0.000), while centile-based EWS had the lowest value (P=0.000) [34].

The Youden index values of the NEWS2, NEWS, and PAR score were 0.542, 0.513, and 0.589, respectively, and their corresponding cutoff values were 7, 7, and 4. Using these values as thresholds, the sensitivity of these 3 EWSs scores were 0.768, 0.739, and 0.710, respectively, while their specificity scores were 0.774, 0.774, and 0.879. The NPV values of the NEWS2, NEWS, and PAR score were 0.926, 0.917, and 0.919, respectively (see Table 5).

Discussion

To the best of our knowledge, this is the first study to investigate whether the NEWS2 specifically accurately identifies the risk of postoperative brain tumor patients’ unplanned ICU admissions. Our finding showed that NEWS2 performed well, with an AUROC of 0.860. The PAR score had a higher AUROC (0.870) than NEWS2.

Regarding unexpected ICU transfer predictions, we found that the NEWS2 and NEWS had greater discriminatory power in brain tumor patients than other EWSs. These results are similar to the findings of Smith et al for emergency patients within 24 h of ICU admission (AUROC=0.857) [23] and to the results of Martín-Rodríguez et al for brain trauma patients transferred to the ICU (AUROC=0.888) [25]. The NEWS2 showed high sensitivity (0.768) and specificity (0.774), meaning that it can identify 76.8% of the patients at risk of being admitted to the ICU after brain tumor surgery and 77.4% of the patients not at high risk of clinical deterioration.

In contrast, Teasdale found that the NEWS was not suitable for assessing acute brain damage patients, arguing that its 2-tier AVPU system is inadequate for assessing consciousness level and that this system should include eye, verbal, and motor responses, limb movements, and pupil reactivity [35]. This difference could be explained by the fact that 50.56% of the unplanned ICU admissions in our study were due to low oxygen saturation, and the SpO2 scale in the NEWS2 can be a good indicator of deterioration in postoperative brain tumor patients.

The major complications in postoperative brain tumor patients include hematomas and cerebral edema [36], which can lead to increased intracranial pressure and Cushing reflex. As a result, these patients’ consciousness worsens, their heart rates and respiration slow, and their blood pressure rises. Although the NEWS2’s 2-tier ACVPU system does not fully reflect consciousness like the Glasgow Coma Scale, the NEWS2 uses a physiological composite score to comprehensively diagnose changes in patients’ vital signs. Once a patient’s consciousness worsens, other physiological parameters change and the overall NEWS2 score increases, indicating the patient’s deterioration [17]. In addition, as new confusion or delirium in the NEWS2’s ACVPU scale is allocated 3 points and postoperative brain tumor patients commonly have confusion or delirium, this can explain the NEWS2’s specificity in this patient population.

Our study showed that the NEWS2’s cutoff point was 7, which means postoperative brain tumor patients with a NEWS2 score ≥7 may have a high risk of unplanned ICU admission within 72 h. This is consistent with the clinical response to the NEWS2’s risk classification in the guideline published by Royal College of Physicians [17], which indicate that patients with a NEWS2 score ≥7 need to be considered for transfer to a high-level care facility, such as the ICU. This is in accordance with the results of Martín-Rodríguez et al for brain trauma patients within 48 h of an ICU transfer (cutoff=7) [25]. This finding indicates that 7 may be a reasonable NEWS2 threshold value for medical teams to identify postoperative brain tumor patients’ deterioration and increase their care level.

Compared to the 24 other EWSs, while the NEWS2 has the higher AUROC value (0.860) and Youden index (0.542), these values are still lower than the PAR EWS’s (AUROC=0.870, Youden index=0.589). This might indicate that the PAR score has the greatest discriminatory power. The PAR score was developed by Goldhill et al and then used in the Royal London Hospital [28,29]. It has been reported to prevent surgical deaths [28] based on 7 variables: temperature, heart rate, systolic blood pressure, ventilatory frequency, oxygen saturation, consciousness, and urine output. Unlike the NEWS2, it incorporates urine output as a physiological parameter, which is an important observation index, as patients with sellar tumors may postoperatively generate a large amount of urine due to intraoperative injury to the hypothalamus or pituitary gland. Moreover, the PAR score’s 4-tier ACVPU system can assess consciousness level more accurately than the NEWS2’s 2-tier ACVPU system. These factors may account for why the PAR score worked better than the NEWS2 for postoperative brain tumor patients. However, even without the urine volume parameter, the NEWS2’s 2-tier ACVPU system can make it easier and more accurate to calculate patient deterioration in a busy clinical environment. Therefore, clinical workers can carefully choose EWSs based on their clinical practice. Overall, this indicates that vital signs combined with other specific neurosurgical variables, suitable parameter bounds, and weighted scores can identify deteriorating brain tumor patients more accurately.

In addition, a systematic review reported that the actual effect of EWSs in clinical application was not as good as the present evidence [16]. This suggests that the clinical use of EWSs may not be solely dependent on its predictive results without clinical nurses’ professional judgment. Meanwhile, Nielsen et al developed the Individual Early Warning Score (I-EWS), which is the combination of NEWS and clinical assessment of clinical nurses, and its prediction ability is better than NEWS alone [37]. The I-EWS allows clinical nurses to adjust the NEWS score according to the clinical evaluation according to nurses’ experience and patients’ condition, and the I-EWS score can be modified range from −4 to +6 points [37]. This suggests that I-EWS could be used and further validated in predicting deterioration of postoperative brain tumor patients by combining EWSs with individual clinical evaluation of nurses to improve the accuracy.

This was a retrospective study at a single center and these factors may have introduced study bias. Importantly, as shown in the recent systematic review and appraisal published by Gerry et al [16], although EWSs are now widely used in clinical practice to identify early clinical deterioration in patients when in hospital, most EWS systems have methodological weaknesses that could adversely affect patient care. For this reason, these systems should continue to be tested, developed, and modified.

Conclusions

The findings from this retrospective study showed that although the NEWS2 performed well when used to evaluate unplanned ICU admissions of patients with brain tumors within 72 h after surgery, the PAR score was also an accurate EWS.