04 June 2023: Clinical Research
Usability Study on Patient Monitoring Systems: An Evaluation of a User Interface Based on User Experience and PreferenceYourim Kim1ABCDEFG, Jimin Son1BF, Wonseuk Jang12ABCDEFG*
Med Sci Monit 2023; 29:e938570
BACKGROUND: The poor ergonomic design of patient monitoring systems can cause user errors and patient harm. This paper presents the results of a comparative usability study based on user experience and the results of a user preference survey.
MATERIAL AND METHODS: We conducted a usability study of 3 patient monitoring systems: Mediana M50, Philips IntelliVue MP70, and Philips IntelliVue MX700. Thirty-nine Coronary Care Unit nurses and 19 Pulmonology and Allergy Care Unit nurses participated in this usability study. User experience was assessed with the Post-Study System Usability Questionnaire and the National Aeronautics and Space Administration Task Load Index. A user preference survey was conducted to evaluate the subjective medical device design preferences for the M50 system’s user interface.
RESULTS: Nurses from the Coronary Care Unit recognized a higher system usability for the MP70 than the M50 (P=0.001) system, and a lower workload for the MP70 compared with the M50 (P=0.005) system. There was no significant (P>0.05) difference in perceived system usability and workload between the M50 and MX700 systems for the nurses from the Pulmonology and Allergy Care Unit. Nurses preferred to activate the arrhythmia alarms except for the ST alarms and missed the beat alarm. They also preferred having a wave freeze function, standby mode, and early warning scoring function, which provides a signal for a patient’s deterioration in health.
CONCLUSIONS: The study provides valuable data on a user interface evaluation based on user experience and preference. The outcome of this study will be helpful for designing next-generation patient monitors with improved patient safety.
Keywords: Intensive Care Units, Monitoring, Physiologic, Personal Satisfaction, User-Centered Design, Workload
A patient monitoring system is a device that monitors a patient’s condition by measuring the patient’s biological data in a variety of patient care settings in a hospital and is one of the important components in an Intensive Care Unit (ICU) . This device measures the patient’s biological data, including invasive blood pressure, electroencephalogram (EEG), electrocardiogram (ECG), carbon dioxide (CO2) gas measurements, and transcutaneous oxygen saturation (SPO2) and provides visible and audible alarms when hazardous conditions are detected .
In the ICU, doctors and nurses do not treat only one patient at a time [3,4], typically complex and error-prone equipment is used, and there is too much patient data for one person to effectively process . The high workload required by the medical staff can be the root cause of medical errors . The evaluation of workload is a key point in user interface studies designed for higher levels of comfort, satisfaction, and efficiency in the workplace and has become an increasingly important tool in system evaluations .
In addition, experts in patient safety confirm that adverse events are often caused by poorly designed user interfaces that do not consider the end user’s capabilities and limitations and that defects in user interfaces are the root cause of adverse events [8–10].
Usability needs to be considered when designing user interfaces of medical devices and systems to facilitate their use and prevent usage errors, and the evaluation of a medical device’s user interface can be achieved by usability testing [11,12]. The U.S. Food and Drug Administration (FDA) requires human factor engineering to be applied to product design and development processes to meet minimum use safety requirements, and usability testing of medical devices has become increasingly important in recent years .
The primary purpose of this study was to compare and evaluate the user experience of ICU nurses with the Mediana M50, Philips IntelliVue MP70, and Philips IntelliVue MX700 patient monitoring systems used with patients admitted to the ICU. A user preference survey was also conducted to evaluate the subjective medical device design preferences for the graphical user interface of the M50 system. The study provides valuable data on a user interface evaluation based on workload, system usability, and user preference. It is expected that the results of this study will contribute to improving safety by identifying user requirements for usability improvement and reducing device use errors through usability evaluation.
Material and Methods
The 3 tested patient monitoring systems were the M50 (Mediana Co., Ltd., Gangwon-do, KOREA), MP70 (M8007A, Philips Medical Systems, Orlando, FL, USA), and MX700 (Philips Medical Systems, Orlando, FL, USA). The Philips MP70 and MX700 are products that have been used for a long time in the ICU. The M50 is a domestic product that was for comparison with market leader equipment in this study. The Philips IntelliVue MP70 is a bedside monitor of patient vital signs; the Mediana M50 monitors the ECG, heart rate, noninvasive blood pressure, arterial pressures, SpO2, pulse rate, and respiration rate; and the Philips IntelliVue MX700 is a bedside patient monitor that is a live video monitor of vital signs. Since the patient monitoring systems undergo a training process along with explanations of each manufacturer’s product usage when the product is installed in the ICU, staff training was conducted according to the level of each system. Also, in this study, there is no difference in the competency levels required for each system, as we focused on monitoring the ECG of patients.
This study was conducted independently at 2 ICUs located at Severance Hospital (Seodaemun-gu, Seoul, Republic of Korea) and Yongin Severance Hospital (Yongin-si, Gyeonggi-do, Republic of Korea). Severance Hospital conducted a study comparing the M50 and MP70, and Yongin Severance Hospital conducted a study comparing the M50 and MX700. For each patient monitor installed in the ICU, participants were required to complete the U.S. National Aeronautics and Space Administration Task Load Index (NASA-TLX) and Post-Study System Usability Questionnaire (PSSUQ) after completing their ICU duty. User preference was evaluated only for the design of the M50 equipment. The patient monitoring device installed in the ICU targeted all patients who were hospitalized at the investigation site and needed ECG monitoring, rather than a specific disease or patient’s condition. However, patients with burns, long-term cortisone treatment, severe heart failure after surgery, mechanical auxiliary heart or ECMO, or implantable defibrillator, and severely ill patients during the end of life were excluded.
Participants in this study were recruited from the ICUs in 2 hospitals who voluntarily participated in this study through informed consent. Intensive care nurses were selected as the target group of this study since they are the major daily users of patient monitors at Severance Hospital. Thirty-eight nurses from the Coronary Care Unit at Severance Hospital and 19 nurses from the Pulmonology and Allergy Care Unit at Yongin Severance Hospital of Yonsei University participated in the ventilator usability test. The nurses in the ICU who participated in this study had a degree in nursing from a 4-year university and had worked in the ICU for an average of 7 years. They had clinical expertise and knew how to use and operate patient monitoring systems. The participants’ information was coded and held in confidence with the test results. Before starting the test, all participants were asked to sign the participant consent form and were made aware of the anonymity of their responses.
This study was approved by both the Institutional Review Board of Yonsei University Health System, Severance Hospital (No: 1-2021-0030) and the Institutional Review Board of Yonsei University College of Medicine, Yongin Severance Hospital (No: 9-2021-0080).
User experience was evaluated through perceived system usability and workload .
Perceived system usability was measured by the PSSUQ , which consisted of 16 questions that assessed the users’ perceived satisfaction with a product or system by an overall score, with 3 subscales: system usefulness, information quality, and interface quality [15–17]. In healthcare, the PSSUQ has been used to evaluate radiotherapy , clinical monitoring , anesthesia , and telerehabilitation systems . The outcome of the PSSUQ is a score that ranges from 1 to 7, in which lower scores indicate greater satisfaction with the perceived system usability [15,17,21]. Each factor also provided a “not applicable” option and a freeform text section for detailed feedback . The score for each factor was determined by averaging the responses to the appropriate questions [17,18].
Workload was evaluated by the NASA-TLX [22,23], which relies on 6 subscales that are associated with workload, including mental demand, temporal demand, physical demand, performance, effort, and frustration [14,22]. This is a validated and commonly used instrument in human factors engineering [24,25]. Although it was originally designed for aeronautics, the NASA-TLX has been widely used in healthcare  to evaluate medical devices such as a radiotherapy system , ventilators [21,26–28], infusion pumps , and physiological monitoring displays . The result of the NASA-TLX is a score from 0 to 100, in which a higher score refers to a higher mental workload and a more difficult user interface .
User preference was derived from a survey of 13 questions related to specific functions and graphical user interface, alarm pattern and pitch, alarm occurrence criteria, and arrhythmia alarm activation. Each question provided several options to choose from and a freeform text section. When each option had a higher selected percentage, its user preference was higher.
The analyses were performed using the statistics software IBM SPSS Statistics 26 (IBM Corp, Armonk, NY, USA). For the results of the user experience, descriptive statistics (mean, standard deviation, median, Q1, and Q3) were presented for the overall score and subscales. The results of user preference were expressed as a percentage for each item. Differences between the user experiences of the patient monitoring systems were analyzed through independent 2-sample
Table 1 shows the results of the system usability evaluation and statistical comparison of the Mediana M50 and Philips MP70 systems. The results of the Mann-Whitney U test comparing the PSSUQ scores of the M50 and MP70 show a statistically significant difference in system usefulness (P=0.001) and interface quality (P<0.001). Information quality was compared with an independent 2-sample t test, and there was a significant difference in the scores of the Mediana M50 and the Philips MP70 (P=0.044) systems. There was a statistically significant difference in overall PSSUQ scores between the MP70 and M50 (2.19 [1.88–2.56] vs 2.84 [2.20–3.29]; P=0.001).
Table 2 shows the results of the system usability evaluation and statistical comparison of the Mediana M50 and Philips MX700 systems. The results of the independent 2-sample t test on the PSSUQ scores for the M50 and MX700 systems, including usefulness (P=0.391) information quality (P=0.555), interface quality (P=0.091), and overall PSSUQ score (P=0.299), show no statistically significant difference between the system usability scores of the 2 devices.
Table 3 shows the results of the comparison of workload between the Mediana M50 and Philips MP70 systems. The results of the Mann-Whitney U test comparing the TLX scores of the Mediana M50 and Philips MP70 systems, including physical demand (P=0.119), temporal demand (P=0.870), and frustration (P=0.355), did not show a significant difference in scores; however, there was a statistically significant difference in the scores for mental demand (P=0.019), performance (P=0.009), and effort (P=0.045). Thus, the TLX score of the M70 was statistically significantly lower than that of the M50 (35.00 [6.67–39.17] vs 39.17 [34.79–43.40]; P=0.005).
Table 4 shows the results for workload and its statistical comparison between the Mediana M50 and Philips MX700 systems. The results of the independent 2-sample t test comparing the TLX scores of the M50 and MX700 systems, including mental demand (P=0.236), physical demand (P=0.246), temporal demand (P=0.638), and TLX (P=0.627) did not show a statistically significant difference between the perceived workload scores of the 2 devices.
The results of the Mann-Whitney U test comparing the PSSUQ TLX scores of the M50 and MX700 show that there was no statistically significant difference in performance (
The results of the patient monitoring device preference survey are shown in Figures 1–4. Based on the results of the patient monitoring devices display setting preference survey, English was preferred for the language setting of the patient monitoring system, and the frequency of use was also higher in English. The respondents preferred to automatically change the screen composition according to the detachment of the patient monitoring module. The most preferred location for the alarm message was the parameter area, followed by the waveform area, top of screen, and bottom of screen area.
As the results of the patient monitoring devices alarm preference survey showed, Philips’ sound patterns were preferred over Mediana’s sound patterns for audible alarm patterns. The user preference was for the pulse tone pitch to change according to the SPO2 change; moreover, for the tachycardia and bradycardia alarms, it was preferred for those to sound immediately outside the alarm limit range. Users preferred that the tachycardia and bradycardia alarm levels were high-level alarms rather than low-level alarms.
Figure 3 shows the preference survey results for the need for 6 special functions. User preference scores showed that a wave freeze function standby mode and early warning scoring functions are necessary for specific functions of the patient monitoring system, except for the drug calculation function, and that a display is necessary for the non-measured parameter menu.
For the preferred settings for activating the arrhythmia alarm, users preferred its activation for all situations except for missed beat, ST high, ST low, and ST multi alarm, and irregular HR alarm showed the same user preference for its activation and deactivation (Figure 4).
This usability study aimed to evaluate the difference in user experience for 2 Philips patient monitoring system models widely used in Korea and Mediana’s patient monitoring system and to evaluate user interface design preferences. Previous studies have proven that the PSSUQ [15–17] and NASA-TLX [22,23] have the capacity for evaluating the usability of medical devices . When the M50 was compared with the MP70 and MX700, the MP70 showed a better perceived usability and perceived workload, and there was no difference in the perceived usability and workload of the MX700.
Through the user experience evaluation, the present study showed that the design of the user interface can affect system usability and workload [14,21]. For instance, regarding workload for the M50, because of its unfamiliar alarm pattern and the inability to stop the alarm for 5 to 10 min, the continuous sounding of its alarm was the cause for the higher score in mental demand (Table 3). The lack of sensitivity of the M50’s touchscreen was also an important reason for a higher score in effort (Table 3). System usability was also affected by information quality due to the decreased readability because of the location, size, and color of alarm messages, and the interface quality was affected by the unfamiliar design of the M50 compared to that of the Philips equipment.
These results can also be verified through the user preference results for the user interface, and the preference survey results can be used as formative evaluation data to improve the user interface design of patient monitoring devices . In the ICU, imaging tests such as computed tomography, magnetic resonance imaging, and radiography are frequently needed to detect pathological changes in patients and to plan treatment ; therefore, the standby mode function is preferred. In the case of a drug calculation function, it would be convenient for drugs that require the calculation of the injection dose ; however, because it can be calculated with a computer calculator, the user preference is to exclude it if it makes the monitor too complicated. Even when an alarm is acknowledged, it is preferable to have an acknowledging alarm function that can temporarily stop the alarm because frequent alarms can interfere with treatment . Regarding arrhythmia detection, which is important in the treatment of cardiovascular diseases , for tachycardia and bradycardia alarms for which the alarm limit range can be adjusted directly, it is preferable to set the alarm to sound immediately when it is out of the range.
This study on 3 patient monitoring systems has several limitations that should be discussed. First, the participants in our study were nurses in the Coronary Care Unit and Pulmonology and Allergy Care Unit, representing only some categories of patient monitoring device users. Therefore, the results of this study cannot be applied to other users. Second, a comparative study between 1 Mediana device and 1 Philips device was performed by each institution, so it was not possible to conduct a comparative analysis between the 2 Philips patient monitoring devices. Third, compared with other usability studies [14,21,28] comparing the 3 models, testing of patient monitoring devices from 2 brands (Medinan and Philips), and excluding General Electric’s patient monitoring device, may not be sufficient. However, our intention was to compare it with market-leading equipment that we were already familiar with, and the Philips equipment was most used in the ICU, where we conducted the study. Finally, compared with other studies , the participants’ experiences with the 3 devices were not balanced, which meant that nurses were more accustomed to using the Philips patient monitors, and there was a difference in proficiency. However, our intention was to compare the usability of the new equipment with that of those already used in the ICU.
This study provides valuable data on a comprehensive user interface evaluation based on workload, system usability, and user preference. The user experience results showed that the MP70 had better system usability and less workload from the end users. In addition, we were able to identify sources of lower usability and higher workload for the M50. The outcome of the user preference survey also confirmed that the design of the M50’s user interface reduced usage errors and reinforce patient safety. It is expected that the results of this study will contribute to improving safety by identifying user requirements for usability improvement and reducing device use errors through use usability evaluation.
FiguresFigure 1. Results of the Preference Survey of Patient Monitoring Devices display setting: (A) Frequency of setting the language between English and Korean. (B) Language setting preference. (C) Changes in the screen composition according to the detachment of the module. (D) The display position of the alarm message. (Excel, 16.0, Microsoft). Figure 2. Results of the Preference Survey of Patient Monitoring Devices alarm: (A) Preference for auditory alarm pattern. (B) Pulse sound pitch changes according to SPO2 change. (C) Criteria for the occurrence of tachycardia and bradycardia alarms. (D) Alarm level of tachycardia and bradycardia alarms. (Excel, 16.0, Microsoft). Figure 3. Results of the preference survey on the need for specific functions of patient monitoring devices. (Excel, 16.0, Microsoft). Figure 4. Results of the preference survey on the setting of the arrhythmia alarm activation of the patient monitoring device. (Excel, 16.0, Microsoft).
TablesTable 1. Post-Study System Usability Questionnaire (PSSUQ) score and comparison results for the Mediana M50 and Philips IntelliVue MP70 in the Coronary Care Unit at Severance Hospital. Table2. Post-Study System Usability Questionnaire (PSSUQ) score and comparison results for Mediana M50 and Philips IntelliVue MX700 in the Pulmonology & Allergy Care Unit at Yongin Severance Hospital. Table 3. National Aeronautics and Space Administration Task Load Index (NASA-TLX) workload scores and comparison results for Mediana M50 and Philips IntelliVue MP70 in the Coronary Care Unit at Severance Hospital. Table 4. National Aeronautics and Space Administration Task Load Index (NASA-TLX) workload scores and comparison results for Mediana M50 and Philips IntelliVue MX700 in the Pulmonology & Allergy Care Unit at Yongin Severance Hospital.
1. Andrade E, Quinlan L, Harte R, Novel interface designs for patient monitoring applications in critical care medicine: human factors review: JMIR Hum Factors, 2020; 7(3); e15052
2. Notification MoFaDSP. Regulation on Medical Device Groups and Class by Group: Medical device products and their assigned classes January 24, 2017, Ministry of Food and Drug Safety Public Notification
3. Effken JA, Loeb RG, Kang Y, Lin ZC, Clinical information displays to improve ICU outcomes: Int J Med Inform, 2008; 77(11); 765-77
4. Surgenor SD, Blike GT, Corwin HL, Teamwork and collaboration in critical care: Lessons from the cockpit: Crit Care Med, 2003; 31(3); 992-93
5. Donchin Y, Seagull FJ, The hostile environment of the Intensive Care Unit: Curr Opin Crit Care, 2002; 8(4); 316-20
6. Rothschild JM, Landrigan CP, Cronin JWThe Critical Care Safety Study, The incidence and nature of adverse events and serious medical errors in intensive care: Crit Care Med, 2005; 33(8); 1694-700
7. Rubio S, Díaz E, Martín J, Puente JM, Evaluation of subjective mental workload: A comparison of SWAT, NASA-TLX, and workload profile methods: Appl Psychol, 2004; 53(1); 61-86
8. Israelski EW, Muto WH, Human factors risk management as a way to improve medical device safety: A case study of the therac 25 radiation therapy system: Jt Comm J Qual Saf, 2004; 30(12); 689-95
9. Crowley J, Kaye R, Identifying and understanding medical device use errors: J Clin Eng, 2002; 27; 188-93
10. Perry SJ, An overlooked alliance: Using human factors engineering to reduce patient harm: Jt Comm J Qual Saf, 2004; 30(8); 455-59
11. Scalliet P, Risk, society and system failure: Radiother Oncol, 2006; 80(3); 275-81
12. : Wiklund Usability testing of medical devices, 2011, Florida, Taylor & Francis
13. US Department of Health and Human Services, Food and Drug Administration, Center for Devices and Radiological Health Office of Device Evaluation (US): Applying human factors and usability engineering to medical devices – guidance for industry and food and drug administration staff (issued 2016 Feb 3)
14. Jiang M, Liu S, Gao J, A usability study of 3 radiotherapy systems: A comparative evaluation based on expert evaluation and user experience: Med Sci Monit, 2019; 25; 578-89
15. Lewis JR, Psychometric evaluation of the PSSUQ using data from five years of usability studies: Int J Hum Comput Interact, 2002; 14(3–4); 463-88
16. Savoy A, Patel H, Flanagan ME, Comparative usability evaluation of consultation order templates in a simulated primary care environment: Appl Ergon, 2018; 73; 22-32
17. Lewis JR, IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use: Int J Hum Comput Interact, 1995; 7(1); 57-78
18. Dosani M, Hunc K, Dumont GA, A vibro-tactile display for clinical monitoring: Real-time evaluation: Anesth Analg, 2012; 115(3); 588-94
19. Görges M, Winton P, Koval V, An evaluation of an expert system for detecting critical events during anesthesia in a human patient simulator: A prospective randomized controlled study: Anesth Analg, 2013; 117(2); 380-91
20. Schutte J, Gales S, Filippone A, Evaluation of a telerehabilitation system for community-based rehabilitation: Int J Telerehabilitation, 2012; 4(1); 15
21. Morita PP, Weinstein PB, Flewwelling CJ, The usability of ventilators: A comparative evaluation of use safety and user experience: Crit Care, 2016; 20; 263
22. Hart SG, Staveland LE: Advances in Psychology: Elsevier, 1988; 139-83
23. Hart SG, NASA-task load index (NASA-TLX); 20 years later, Sage CA: Los Angeles, CA, Sage Publications
24. Karsh B, Holden R, Alper S, Or C, A human factors engineering paradigm for patient safety: Designing to support the performance of the healthcare professional: BMJ Qual Saf, 2006; 15(Suppl 1); i59-i65
25. Carayon P: Handbook of human factors and ergonomics in patient safety, 2012, Mahwah, NJ, Lawrence Erlbaum
26. Marjanovic N, L’Her E, A Comprehensive approach for the ergonomic evaluation of 13 emergency and transport ventilators: Respir Care, 2016; 61(5); 632-39
27. Wachter SB, Johnson K, Albert R, The evaluation of a pulmonary display to detect adverse respiratory events using high resolution human simulator: J Am Med Inform Assoc, 2006; 13(6); 635-42
28. Jiang M, Liu S, Gao J, Comprehensive evaluation of user interface for ventilators based on respiratory therapists’ performance, workload, and user experience: Med Sci Monit, 2018; 24; 9090-101
29. Carayon P, Wetterneck TB, Hundt AS, Advances in patient safety observing nurse interaction with infusion pump technologies: Advances in patient safety: from research to implementation (Volume 2: Concepts and methodology), 2005, Rockville (MD), Agency for Healthcare Research and Quality (US)
30. Görges M, Staggers N, Evaluations of physiological monitoring displays: A systematic review: J Clin Monit Comput, 2008; 22(1); 45-66
31. International Electrotechnical Commission: IEC 62366-2: 2016 Medical devices – Part 2: Guidance on the application of usability engineering to medical devices, 2016
32. Rollandi GA, Biscaldi E, De Rito MRStudy with thoracic and abdominal spiral CT in intensive care unit patients: Radiol Med, 1998; 96(5); 485-91 [in Italian]
33. Herout PM, Erstad BL, Medication errors involving continuously infused medications in a surgical Intensive Care Unit: Crit Care Med, 2004; 32(2); 428-32
34. Poncette AS, Wunderlich MM, Spies C, Patient monitoring alarms in an Intensive Care Unit: Observational study with do-it-yourself instructions: J Med Internet Res, 2021; 23(5); e26494
35. Serhani MA, HTEK , Ismail H, Nujum Navaz A, ECG monitoring systems: Review, architecture, processes, and key challenges: Sensors (Basel), 2020; 20(6); 1796
21 Sep 2023 : Clinical ResearchThe Impact of Implementing the Vortex Approach on Airway Management Performance in Stressed Medical Student...
Med Sci Monit In Press; DOI: 10.12659/MSM.940372
20 Sep 2023 : Clinical ResearchA Retrospective Study to Compare Pharyngeal Anatomical Changes Measured Before and After Treatment for Malo...
Med Sci Monit In Press; DOI: 10.12659/MSM.941749
20 Sep 2023 : Database AnalysisComparative Finite Element Analysis of Endocrowns and Traditional Restorations for Endodontically Treated M...
Med Sci Monit In Press; DOI: 10.12659/MSM.941314
Most Viewed Current Articles
13 Nov 2021 : Clinical ResearchAcceptance of COVID-19 Vaccination and Its Associated Factors Among Cancer Patients Attending the Oncology ...
Med Sci Monit 2021; 27:e932788
30 Dec 2021 : Clinical ResearchRetrospective Study of Outcomes and Hospitalization Rates of Patients in Italy with a Confirmed Diagnosis o...
Med Sci Monit 2021; 27:e935379
08 Mar 2022 : Review articleA Review of the Potential Roles of Antioxidant and Anti-Inflammatory Pharmacological Approaches for the Man...
Med Sci Monit 2022; 28:e936292