Evaluation of a Previously Developed Predictive Model for Infective Endocarditis in 320 Patients Presenting with Fever at 4 Centers in Japan Between January 2018 and December 2020

Background

Diagnosing infective endocarditis (IE) is difficult due to its non-specific manifestations and physical and laboratory findings [1]. When using the modified Duke criteria (m-DC), various advanced medical modalities, including multiple blood cultures, transthoracic or transesophageal echocardiography, computed tomography (CT), and magnetic reasoning image (MRI), are required [2]. Thus, reaching a definitive IE diagnosis is almost impossible at medical facilities that do not have access to these modalities. It is also impractical to perform these examinations on all patients with fever of unknown cause, even in hospitals that provide high-level medical services, because of the burden on patients and medical staff, as well as the high medical cost. In such situations, antibiotics may be administered before obtaining blood culture samples [3,4], possibly making diagnosis more difficult and with a poorer prognosis [4,5].

Therefore, developing prediction models that are easier to use than the m-DC and that are usable before performing examinations with advanced medical modalities is desirable. Previous prediction models can only be used after detection of specific bacteria, such as Staphylococcus aureus, Viridans streptococci, or Enterococcus spp., by blood culture [6–10]. Thus, in a previous single-center case-control study, we developed an IE prediction model for use in patients with undiagnosed fever (UF) at a rural university hospital in Japan, which was reported in 2021 [11]. The prediction model formula is as follow: −8.350 + {transfer by ambulance (Present=4.087, Absent=0)} + {cardiac murmur on admission (Present=2.844, Absent=0)} + {pleural fluid (Present=1.440, Absent=0)} + {neutrophil count (%)×0.090} + {platelet count (103/μL)×(−0.057)}. The predictive occurrence of IE is then calculated according to the following formula: 100×exp(score) ÷ {1+exp(score)}. This model makes it possible to evaluate the probability of IE immediately after admission without knowing the blood culture results using only simple and readily available indicators, including ambulance transfer; presence of cardiac murmur; presence of pleural effusion; blood neutrophil percentage; and platelet count. However, it is unclear whether our model is also valid at other hospitals because the characteristics of patients and the reasons for ambulance transfer may differ among regions [12]. Therefore, this study aimed to evaluate a previously developed predictive model for IE in 320 patients (completely different patients from our previous study) presenting with fever at 4 centers in Japan between January 2018 and December 2020.

Material and Methods

ETHICAL CONSIDERATIONS:

The Ethics Committee of Saga University Hospital approved the study (file number: 2020-11-R-10), which was conducted in accordance with Ethical Guidelines for Medical and Health Research Involving Human Subjects in Japan and the Declaration of Helsinki. The requirement for written informed consent was waived by the Ethics Committee of Saga University Hospital due to the retrospective study design. All patients were informed about the purpose of the study via the website of the clinical research center of each university hospital and provided consent as per the comprehensive agreement method. Patient anonymity was protected. This study is registered in the University hospital Medical Information Network (UMIN) at www.umin.ac.jp (UMIN ID: UMIN000042965).

STUDY DESIGN AND PARTICIPANTS:

This study was a multi-center retrospective study for external validation study of our previous study published in 2021 [11]. The participants in the previous study were not included in the present study. Among patients aged ≥20 years admitted to 4 university hospitals in Japan owing to fever from January 2018 to December 2020, patients categorized as I-330 (IE) and R-50-9 (UF) according to the International Statistical Classification of Diseases and Related Health Problems-10 were enrolled. All patients without a body temperature of >37°C before admission were excluded. Among patients coded as I-330, those with a nosocomial onset or referred for valve surgery were excluded. Additionally, among patients coded as R-50-9, those with a diagnosis before admission or who had not been examined by chest X-ray, blood tests, or urinalysis on admission were excluded. More than 2 physicians at each hospital reviewed the diagnoses using the m-DC, dividing patients into the IE group and the group with fever caused by other diseases (UF group).

Among patients coded as I-330, those diagnosed with “definite IE” were allocated to the IE group. Patients showing accumulation to the prosthetic valve on 18F-fluorodeoxyglucose positron emission tomography CT (18F-FDG PET-CT), which was judged by the radiologist, were allocated to the IE group, even when the result of the m-DC was not definite, because 18F-FDG PET-CT is diagnostically more useful in such patients [13–15]. Patients coded as I-330 without a definite diagnosis of IE were excluded from the analysis because they could have been given the code due to lack of the required examinations with the m-DC, such as multiple blood cultures, rheumatoid factor, urinalysis, echocardiography, CT, MRI, and fundoscopy, which meant that the reviewed diagnosis was “not definite.”

Among patients coded as R-50-9, those not diagnosed with definite IE were allocated to the UF group. Patients diagnosed with definite IE were allocated to the IE group, except those with a definite diagnosis of non-infectious disease, who were excluded. The final diagnoses of patients in the UF group were determined by reviewing all medical records, consequently identifying the causes of hospitalization from their codes. If the causative disease was unknown, “fever of unknown origin” was the final diagnosis.

SETTING:

This study was performed at 4 university hospitals in Japan: Saga University Hospital, Dokkyo University Hospital, Juntendo University Hospital, and Toho University Hospital. Saga University Hospital is the only university hospital in Saga Prefecture, a rural city with 811 000 population in Japan, which has 602 inpatient beds. Dokkyo University Hospital is one of 3 university hospitals in Tochigi Prefecture, a suburban city with 1 933 000 population in Japan, which has 1195 inpatient beds. Juntendo University Hospital and Toho University Hospital are 2 of 15 university hospitals in Tokyo prefecture, urban area with 14 048 000 population in Japan, which have 1036 and 916 inpatient beds, respectively. These 4 university hospitals have 29, 31, 31, and 34 departments, respectively, including the departments of general medicine, cardiology, or cardiovascular surgery, and provide high-level medical services, mainly for patients in acute disease phases.

DATA SOURCES:

Two or 3 physicians from the department of general medicine at each hospital reviewed the medical records and referral letters from previous doctors to collect the data of potentially eligible patients. The survey items are shown in Supplementary Table 1. Prolonged antimicrobial use for chronic diseases was excluded from the definition of antibiotic administration before obtaining blood cultures. The presence of pleural effusion or pulmonary edema on chest X-ray or chest CT on admission was determined by a radiology specialist. If interpretation by a radiologist was not possible, it was determined by 2 general physicians (GPs). When 2 GPs agreed on the presence of pleural effusion or pulmonary edema, it was determined as “present.” When 2 GPs disagreed, a decision was reached by consensus.

STATISTICAL ANALYSIS:

Every examination required for the m-DC evaluation was not necessarily performed on all patients; rather, each examination was performed according to the clinical decision. In cases where some of the examinations were not performed, the results of these examinations required by the major criteria or minor criteria were judged as “no abnormalities.” Values for other missed survey items were excluded from the univariate and multivariate analyses. Categorical variables are expressed as percentages and were compared using the χ2 test. Continuous variables are expressed as median and interquartile range (IQR) and were compared using the Mann-Whitney U test. Statistical significance was set at p<0.05. The multivariate logistic regression analysis with the forced-entry method was performed on 5 predictive factors of the model developed in our previous study: ambulance transfer, presence of cardiac murmur, presence of pleural effusion, blood neutrophil percentage, and platelet count [11]. Subsequently, the odds ratio (OR) and 95% confidence interval (CI) were calculated for each factor. Additionally, the predictive probability of definite IE determined by the m-DC was calculated. The model formula developed in our previous study was used without modification in the present study. The predictive performance and calibration of the model were evaluated using the area under the curve (AUC) and shrinkage coefficient, respectively. The stratum-specific likelihood ratio was also calculated. Finally, we calculated the probability of definite IE, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for each cut-off point corresponding to a sensitivity of 90%, Youden’s index, or a specificity of 90% of the prediction model in the present study and in our previous study [11]. A subgroup analysis was performed by comparing the AUC values of the prediction model between the IE group and the UF group divided by the presence of the survey items that were not used in the prediction model. SPSS (version 25; IBM Corp., Armonk, NY, USA) was used for statistical analysis.

We calculated a sample size of 20 patients based on an effect size of 0.30 (predicted AUC: 0.8; null hypothesis AUC: 0.50), an alpha error of 0.05, and a beta error of 0.20, which were estimated by the AUC of 0.893 reported previously [11]. However, we adopted 100 for each group commonly required in external validation [16].

Results

PATIENT ENROLLMENT AND ALLOCATION:

Patient enrollment and allocation to either the IE or UF group is shown in Figure 1. Ninety-seven of 264 patients coded as I-330, who were without a fever of ≥37°C (n=55), with a nosocomial onset (n=23), and with a referral for valve surgery (n=19) were excluded. Of the remaining 167 patients, 112 were diagnosed with definite IE and were allocated to the IE group. Of the 112 patients, 1 was diagnosed with definite IE because 18F-FDG PET-CT showed accumulation to the prosthetic valve, despite the result of “non-definite” according to the m-DC. Fifty-five patients coded as I-330 without a “definite” diagnosis were excluded. Similarly, 140 of 348 patients coded as R-50-9, who were without a fever of ≥37°C (n=40), in whom the cause of fever had been identified before admission (n=82), and without blood tests, urinalysis, or chest X-ray before admission (n=18), were excluded. Of the remaining 208 patients, 7 were diagnosed with definite IE and were allocated to the IE group. Thus, 201 patients were allocated to the UF group. A total of 320 patients (119 patients in the IE group and 201 patients in the UF group) were included. The number of patients included from each hospital is shown in Supplementary Table 2.

PATIENTS’ CHARACTERISTICS:

Patients’ characteristics are shown in Table 1. The ratio of males (67% vs 48%, p=0.001) and the rates of 30-day mortality (11% vs 3%, p=0.002), in-hospital mortality (18% vs 3%, p<0.001), IE history (9% vs 1%, p<0.001), prosthetic valve replacement (30% vs 7%, p<0.001), chronic dermatological disorder (13% vs 5%, p=0.007), dental problems within 6 months (50% vs 33%, p=0.021), cardiac murmur on admission (64% vs 12%, p<0.001), pulmonary edema on admission (13% vs 4%, p=0.001), and pleural effusion on admission (39% vs 22%, p=0.001) were significantly higher in the IE group than in UF group. Additionally, the duration of hospital stay was longer in the IE group (46 [IQR: 25-60] vs 15 [IQR: 9–25] days, p<0.001). Conversely, the percentage of patients administered antibiotics before obtaining blood culture samples (19% vs 32%, p=0.018) and the percentage of patients that used steroids before admission (3% vs 10%, p=0.031) were significantly higher in the UF group. Vital signs and laboratory findings on admission in both groups are shown in Table 2. The IE group had a significantly higher respiratory rate (20 [IQR: 16–24] vs 19 [16–21] beats/min, p=0.036), white blood cell (WBC) count (9.9×103 [IQR: 7.7–14.2] vs 8.4×103 [5.1–12.5]/μL, p=0.001), blood neutrophil percentage (86.8 [IQR: 81.7–91.7] vs 79.2 [IQR: 69.8–86.1]%, p<0.001), serum total bilirubin concentration (0.7 [IQR: 0.5–1.1] vs 0.7 [0.4–0.9] mg/dL, p=0.013), serum lactate dehydrogenase concentration (275.0 [IQR: 223.0–367.0] vs 238.5 [IQR: 180.0–299.5] IU/L, p<0.001), blood urea nitrogen concentration (21.0 [IQR: 13.0–35.1] vs 15.0 [IQR: 11.0–23.2] mg/dL, p<0.001), and serum creatinine concentration (0.9 [IQR: 0.7–1.5] vs 0.8 [IQR: 0.6–1.1], p<0.002). Systolic blood pressure (116 [IQR: 105–127] vs 121 [108–137] mmHg, p=0.016), platelet count (15.1×104 [IQR: 11.2–21.7] vs 20.5×104 [14.7–31.5]/μL, p<0.001), and serum albumin concentration (2.9 [IQR: 2.5–3.4] vs 3.1 [2.6–3.6] g/dL, p=0.021) were significantly lower in the IE group. The differences between the 2 groups in the results of the m-DC, causative organisms, and the execution rate of each examination required by the m-DC in each group are shown in Supplementary Tables 3–5. A breakdown of the final diagnoses in the UF group is shown in Supplementary Table 6.

PERFORMANCE OF THE PREDICTIVE MODEL FOR IE:

The results of the multivariate logistic regression analysis of predictive factors are shown in Table 3. The presence of cardiac murmur (OR 13.13, 95% CI 6.69–27.36, p<0.001), blood neutrophil percentage (OR 1.09, 95% CI 1.06–1.14, p<0.001), and platelet count (OR 0.96, 95% CI 0.93–0.99, p=0.003) were significant factors. No significant differences were observed in ambulance transfer (OR 1.81, 95% CI 0.91–3.55, p=0.089) or presence of pleural effusion (OR 2.34, 95% CI 0.62–2.42, p=0.565). This study demonstrated the excellent discriminative ability of the model with an AUC of 0.783 (95% CI 0.732–0.834; Figure 2). The model was well calibrated, with a shrinkage coefficient of 0.963 (Figure 3). The stratum-specific likelihood ratio was 0.1–3.4, which increased as the score increased (Table 4). The cut-off values with a sensitivity of 90%, Youden’s index, and a specificity of 90% were −1.75, −0.72, and 3.15, respectively. The PPV and NPV derived from each cut-off value were 52% and 90%, 59% and 88%, and 66% and 69%, respectively (Table 5). The PPV and NPV calculated from the cut-off values of −1.68, −1.1, and −0.38 derived in our previous study were 53% and 90%, 55% and 88%, and 60% and 87%, respectively (Table 5).

SUBGROUP ANALYSIS:

To identify the factors that could improve the performance of the IE prediction model, subgroup analyses were performed on the survey items with more than 10 cases, including the types of cardiac valve (native or prosthetic), use of steroids or immunosuppressant agents, and presence of glomerulonephritis as a complication, in both the IE group and the UF group. The AUC of the prediction model was >0.8 in patients with native cardiac valves (AUC: 0.820, 95%CI 0.768–0.872) and without glomerulonephritis (AUC: 0.822, 95%CI 0.760–0.884) (Table 6).

Discussion

In hospitals without advanced medical modalities, antibiotics may be administered to patients with fever without typical symptoms or findings of IE before obtaining blood culture samples, which makes it difficult to make a precise diagnosis [4,5]. Our previous study in Japan from 2007 to 2017 revealed that antibiotics were administered to 43.2% of patients before obtaining blood culture samples [17]. In addition, a previous study reported that early diagnosis and provision of treatment opportunities is essential for patients with IE in the current era [18]. Therefore, development of the IE prediction model, which is easier to use than the m-DC and can be used at an early stage prior to examinations with advanced medical modalities, is required. Existing models can be used only after a specific bacterial strain is detected on blood culture [6–10], which led us to develop the prediction model for definite IE diagnosed with the m-DC among patients with UF [11]. The present study was performed to validate the applicability of our model in the multi-center setting, revealing a high AUC and good calibration. The results suggest that our model has broad utility in many hospitals. Additionally, the presence of cardiac murmur, blood neutrophil percentage, and platelet count were particularly important, because they were significant factors for the IE prediction in the present study as well as in our previous study [11]. Therefore, the use of our model in patients with UF could make it possible to predict IE immediately after admission, which would prevent antimicrobial administration before obtaining blood cultures and help physicians to select patients to undergo investigation with advanced medical modalities.

Our prediction model demonstrated a high AUC, although it was slightly lower than in our previous study [11]. Ambulance transfer was not an independent factor in the present study because the percentage of patients transferred by ambulance in the IE group was lower (39% vs 51%) and that in the UF group was higher (26% vs 1%) than in the previous study [11]. In addition to realizing the seriousness of their symptoms, various socioemotional factors, including strong anxiety, difficulty in accessing hospitals solely by the will of the patients themselves without an ambulance, or their desire to avoid causing inconvenience to family members or caregivers, could also be reasons for patients choosing to call for an ambulance [12]. Hospital accessibility could also depend on the area of residence (eg, urban vs rural) [12]. This study was a multi-center retrospective study involving hospitals at various cities in Japan: a large metropolitan city, Tokyo; a medium-scale city, Tochigi; and a typical rural city, Saga. However, our previous study was conducted only in a rural city [11]. Therefore, hospital accessibility may have differed between the present study and our previous study. Additionally, despite an equivalent in-hospital mortality rate between the IE group in the present study and our previous study (18% vs 17%), pulmonary edema on admission (13% vs 29%), pleural effusion on admission (39% vs 64%), pulse rate (90 bpm vs 102 bpm), respiratory rate (20 breaths/min vs 23 breaths/min), serum WBC count (9.9×103/μL vs 13.5×103/μL), blood neutrophil percentage (86.8% vs 89.6%), and serum C-reactive protein (CRP) concentration (8.3 mg/dL vs 10.4 mg/mL) in the IE group were lower than in the previous study [11]. Accordingly, patients in the IE group in the present study might have had a less severe condition on admission, possibly resulting in fewer ambulance transfers. However, the serum WBC count (8.4×103/μL vs 10.5×103/μL) and CRP concentration on admission (8.6 mg/dL vs 11.7 mg/mL) in the UF group in the present study were lower than in our previous study [11]. Additionally, pulmonary edema on admission (4% vs 1%), pleural effusion (22% vs 29%), pulse rate (91 bpm vs 90 bpm), respiratory rate (19 breaths/min vs 18 breaths/min), and in-hospital mortality (3% vs 6%) were the same in the UF group in both studies. Accordingly, the severity in the UF group in the present study might not have been higher; rather, it may have been equal or lower. Nevertheless, the percentage of ambulance transfers in the UF group was higher than in our previous study, which might have been influenced by various socioemotional factors other than disease severity.

The fact that pleural effusion was not an independent factor in the present study. unlike in our previous study. could be another reason for the lower AUC in this study. Pleural effusion increases with the elevation in hydrostatic pressure due to heart failure, liver cirrhosis, or renal failure; the decrease in oncotic pressure due to hypoalbuminemia; and increased vascular permeability [19,20]. The percentage of patients with pleural effusion in the present study was lower in the IE group (39% vs 64%) and the same in the UF group (22% vs 29%) compared with our previous study [11]. Fewer patients might have been affected by increased vascular permeability or decreased oncotic pressure in the IE group, resulting in a lower percentage of patients with pleural effusion, which was suggested by the fact that the IE group had a higher serum albumin concentration (2.9 vs 2.5 g/dL) and a lower serum WBC count and CRP concentration than the IE group in the previous study [11]. Consequently, pleural effusion was not an independent factor for our prediction model in the present study.

The difference in the severity of the condition on admission between the present study and our previous study could have caused the difference in the AUC. Our prediction model used data obtained on admission. However, the condition of some patients with few symptoms on admission can worsen with time [21]. A lower percentage of patients transferred by ambulance and the presence of pleural effusion might indicate a less severe condition in patients in the IE group in the present study, which could have influenced the performance of our model.

Finally, the subgroup analysis showed a higher AUC with the IE prediction model in patients with native valve than in the overall patients in this study. The IE prediction model, which was developed in our previous study and externally validated in this study, can estimate the possibility of definite IE according to the m-DC. However, the sensitivity and NPV of the m-DC for diagnosis patients with IE with prosthetic valves were low at 50–60% [14]. This may be the cause of the low AUC of the IE prediction model when used in patients with prosthetic valves. Therefore, selectively using this prediction model in patients with native cardiac valves would help to improve the discriminatory ability of the model.

This study has some limitations that should be noted. Although this was a multi-center retrospective study conducted at 4 university hospitals in Japan, the relatively small number of participating hospitals may not be representative of hospitals all over Japan, especially in terms of the severity of the condition on admission and the percentage of patients transferred by ambulance. Furthermore, Japan is the most aged society in the world. Accordingly, the age distribution, types of causative disease, disease severity, percentage of patients transferred by ambulance, and healthcare insurance system in Japan differ from other countries [22,23], which could have introduced selection bias. Moreover, a previous study reported that there are geographic differences in the characteristics of patients with IE between countries [18]. Therefore, multi-center studies involving many different types of hospitals worldwide are ideal for further model generalization.

Conclusions

In this multi-center, retrospective, external validation study performed at university hospitals in Japan, our IE prediction model among patients aged ≥20 years with UF was well-validated. The model makes it possible to predict IE among patients with UF at an earlier stage after admission, preventing antibiotic administration before obtaining blood culture samples or helping physicians to determine the indications for advanced medical modalities.