Evaluation of IMPALA 2.0: Addressing Patient Monitoring in Low-Resource Hospitals in Malawi

Introduction

Globally, mortality rates among children under the age of 5 years continue to persist, with sub-Saharan Africa, a low-resource setting (LRS), exhibiting a greater incidence than high-resource settings [1]. Studies have revealed that monitoring vital sign trends can be a crucial predictor of clinical deterioration, enabling timely interventions that lead to improved health outcomes [2,3]. Patient monitoring systems (PMSs), or vital sign monitors, are essential tools in modern healthcare, providing real-time monitoring of patient vital signs to support clinical decision-making [4]. In LRSs, such as Malawi, where critical care remains underdeveloped due to a lack of necessary resources, the role of PMS becomes even more critical [5,6]. The scarcity of critical care resources in Malawi hampers the effective management and monitoring of critically ill patients [6], highlighting the urgent need for systems that can provide early warnings of potential deterioration and support clinical decision-making. It is against this background that the IMPALA project (Innovative Monitoring in Paediatrics in Low-resource settings: an Aid to save lives) developed IMPALA continuous PMS for hospitalized children in LRSs. The monitoring system focuses on providing an early warning of potential deterioration in clinical conditions and helps healthcare professionals reach a specific diagnosis.

The IMPALA continuous PMS was originally developed as version 1.0 (Figure 1A). In adapting the device for LRSs, based on the user feedback and observations from a mixed-method study (unpublished) that was conducted, the device underwent significant enhancements, leading to the release of version 2.0 (Figure 1B). Building on the insights gathered from the initial version, the updated IMPALA monitor was designed to provide a more user-friendly, reliable, and effective experience in real-world healthcare settings. Notable improvements included a larger font size, streamlined menu features, labeled buttons and alarms, an increased screen size from 5 inches to 10 inches, and the incorporation of larger alarm lights. Additionally, enhancements were made to the selection of reusable sensors, and the battery life was extended to a minimum of 4 h to better meet user needs.

Although PMSs are indispensable tools for healthcare professionals in managing cases of critically ill patients [7], they have significant usability issues that pose threats to patient safety [8]. One major issue is the prevalence of false-positive alarms, which can lead to alarm fatigue and increase the likelihood that critical alarms are overlooked [9–11]. Usability is also affected by data artifacts resulting from sensor malfunctions, patient movement, or electrical interference, leading to inaccurate physiological measurements [12,13]. These challenges are further compounded by practical issues, such as limited battery life, worn connectors, and the cumbersome nature of multiple sensor cables, all of which can disrupt patient care [14]. Studies have also revealed that healthcare professionals struggle with the interpretation of vital sign data [15] and experience cognitive overload due to complex and non-intuitive system interfaces [16]. The lack of standardization across different PMS manufacturers exacerbates these challenges, as variations in user interfaces and functionalities hinder seamless transitions between systems [14,16]. Tscholl et al [14] identified 6 key thematic areas of PMS usability challenges – alarms, artifacts, software, hardware, human factors, and system factors – underscoring the critical need for improved design, standardization, and enhanced usability, to ensure that PMSs effectively support healthcare professionals in delivering safe patient care.

The effectiveness of PMSs in an LRS, such as Malawi, is a crucial concern because these systems may be developed without considering the unique challenges and requirements of LRSs, which are often characterized by limited access to advanced technology, low-level digital literacy, and power outages [17,18]. Ignoring these factors can result in poor usability, which can impact patient safety, cause frustration for users, disrupt workflows, and increase medical errors [19–21]. Therefore, it is essential for medical device developers to conduct usability evaluations that consider the diverse needs of users in these settings [21,22]. Existing usability research on PMSs predominantly originates from high-resource settings, often disregarding the unique needs of LRSs, with significant healthcare disparities [9,14–16,23]. As a result, design recommendations may not be suitable for the local context, highlighting the importance of studies that provide insights from underprivileged populations with limited access to technology resources [22]. This is especially crucial for emerging technologies, such as the IMPALA continuous PMS, where poor usability could hamper effective healthcare delivery and hinder the objective of reducing pediatric morbidity and mortality through timely interventions. The main aim of this study was to evaluate the usability of the IMPALA PMS (version 2.0), optimized for use in an LRS, by assessing the opinions and experiences of 24 healthcare professionals.

Material and Methods

ETHICAL CONSIDERATION:

Ethical clearance and permission for conducting this study were granted by the College of Medicine Research Ethics Committee (Ref. number: P.01/22/3552). All participants provided informed consent by signing a consent form, as specified by the approved study protocol.

STUDY DESIGN:

This usability study was conducted in parallel with the clinical research on the IMPALA continuous PMS. The study utilized an embedded mixed-method study design using quantitative and qualitative approaches. The quantitative part had a cross-sectional design. The qualitative part incorporated 2 usability research methods: contextual inquiry, to analyze the underlying cause of usability issues, and co-design sessions, to propose ways of improving usability dimensions. The study was structured into 3 distinctive phases, each serving a specific purpose. In the first phase, a quantitative study was conducted, with the participation of clinicians, clinical nurses, and research nurses. The second phase was a qualitative study that involved research nurses only. The last phase involved co-design sessions, with all 3 categories of study participants, which facilitated the identification of ways of improving factors that impede the usability of the IMPALA monitoring system.

STUDY LOCATION:

The study took place in the pediatric High-Dependency Unit (HDU) ward at Zomba Central Hospital, located in the southern region of Malawi.

STUDY PARTICIPANTS AND SAMPLING METHODS:

During the study period, nurses and clinicians working in the pediatric ward were sampled using purposive sampling. Specifically, the participants recruited were those working in the pediatric HDU, where the IMPALA continuous PMSs were mounted. The study participants were categorized into 3 groups: research nurses, clinical nurses, and clinicians. Research nurses were part of the IMPALA project and were responsible for recruiting critically ill children under 5 years old to participate in the IMPALA study. They were also responsible for operating the patient monitoring system in the HDU, having received formal user training on its use. Clinical nurses and clinicians, on the other hand, were government healthcare professionals who worked in the pediatric ward and had received only basic orientation user training on how to use the PMS. The term “clinicians” referred to doctors and clinical officers. For the quantitative part, the ward had 32 healthcare professionals, but only 24 participants were recruited, as the other 8 were not available during the study period.

For the qualitative, contextual inquiry, 5 participants were required. The probabilistic model of problem discovery described by Sauro and Lewis [24] was applied to determine sample size, with a target of discovering 85% of usability issues at an average probability of 0.31, as highlighted by Nielsen [25]. The number of 5 participants was calculated based on the formula P = 1 - (1 - p)n, where P is the problem discovery rate, p is the average probability of detection, and n is the required number of participants [24].

INCLUSION AND EXCLUSION CRITERIA:

Nurses and clinicians who had worked in the pediatric ward at Zomba Central Hospital for at least 2 months were included in the study. Healthcare professionals not yet registered with the nurses or medical council and students on attachment were excluded from the study.

DATA COLLECTION METHODS AND TOOLS:

The data collection methods and tools used in this study included the Usefulness, Satisfaction and Ease of Use (USE) questionnaire, contextual inquiry, and a co-design session.

USE QUESTIONNAIRE: In the first phase, a REDCap-based digital USE questionnaire was administered to all participants, using a tablet. The USE questionnaire is one of the methods and questionnaires used for evaluating or assessing usability of technological products based on user perception [26]. The USE questionnaire evaluated 4 key dimensions of usability: usefulness, ease of use, ease of learning, and satisfaction, represented by 30 questions with 10-point Likert scale options. It also included 2 open-ended questions to gather participants’ views on aspects not covered by closed-ended questions. The aim of using the USE questionnaire was to capture participants’ views of their user experience with the IMPALA continuous PMS through their ratings, which reflected their perceived usability across all 4 dimensions. Faria et al [27], suggested that the dimensions utilized in the USE questionnaire are critical components in evaluating usability. The questionnaire items were crafted to be easily comprehensible to survey participants while being straightforward and broadly applicable, as emphasized by Lund [28].

CONTEXTUAL INQUIRY: In the second phase of the research, the contextual inquiry usability research method was utilized to investigate the usability challenges reported by users. Contextual inquiry is a usability research method commonly used for field testing. It involves observing end users as they interact with the device or system, and interviewing them when clarification of their actions is needed [29,30]. The present study involved 5 HDU research nurses who participated directly. The inquiry focused on device and user contexts, to identify any underlying causes of challenges reported by participants, as well as any usability issues that may not have been reported. A contextual inquiry approach was chosen to gain a comprehensive understanding of work practices and behavior. This method is especially effective in revealing intricate and often overlooked aspects that have a significant impact on the user experience. It provides more insightful and relevant information than self-reported or lab-based research techniques [31]. We used a modified cognitive walkthrough to ensure the inquiry was as authentic as possible. In this approach, participants were not given pre-assigned tasks but performed tasks based on real clinical scenarios as they naturally occurred. This contrasts with the traditional cognitive walkthrough [32], in which tasks are typically assigned by the researcher. Each participant was observed for 2 days and asked to provide think-aloud responses to gain a deeper understanding of their actions and perspectives.

CO-DESIGN SESSIONS:

The third phase consisted of co-design sessions. Data were collected through audio recordings and an interview guide in this phase. The interview guide included open-ended prompts that encouraged participants to reflect on their experiences with the patient monitoring system and suggest potential improvements. Additionally, closed-ended questions were used for participants to rank various designed or proposed solutions. The audios were just partly utilized, since some had high background noise. In our collaborative design session, we engaged a group of individuals who included 5 clinical nurses, 6 study nurses, and 3 clinicians as users. Additionally, we welcomed 2 social scientists to serve as independent observers and 1 biomedical engineer from Goal 3 (the company that developed the device) as a technical expert for the monitoring system. The investigator, along with the monitor developers (technical team), convened with small groups of 3 to 6 participants. This small group size was chosen to encourage active participation from each participant during the co-design sessions.

DATA ANALYSIS:

Quantitative data from the USE questionnaire were analyzed using R statistical software to produce descriptive statistics, including mean, standard deviation and range, for the overall scores of all 4 variables of the USE questionnaire. Furthermore, a box plot of all 30 individual questions was plotted for a graphical presentation of device attributes. Qualitative data from the open-ended questions, as well as data from contextual inquiry and co-design sessions, were analyzed using thematic analysis. This comprehensive approach ensured that the quantitative and qualitative aspects of the data were thoroughly examined, to provide a robust evaluation of the IMPALA system’s usability.

Results

PERCEIVED USABILITY:

A total of 24 participants filled out the USE questionnaire. Most participants (n=15, 62.5%) were clinical nurses, followed by research nurses (n=8, 25%), and clinicians (n=3, 12.5%). The findings indicated diverse perceptions of usability across the 4 key variables, as detailed in Table 1. Perceived usefulness received high ratings across all participant groups, with mean scores from clinicians of 8.71, nurses of 9.18, and research nurses of 9.21. However, there was considerable variation in responses regarding ease of use, with clinicians giving it the lowest mean score of 5.42, clinical nurses scoring it 7.86, and research nurses scoring it 8.11.

The general overview of how the participants rated the patient monitor is summarized in Table 2. Usefulness received the highest mean score, of 9.13, and a minimum score of 7. User satisfaction, on the other hand, had a mean score of 8.49 and a minimum score of 6.33. Ease of learning had a mean score of 7.83 and a minimum score of 4, while ease of use had the lowest mean score, of 7.60, and a minimum score of 4.75.

Although the average scores for all usability attributes were above the 5-point threshold, indicating generally positive perceptions, there were notable variations. Figure 2 shows an analysis of participants’ responses to the USE questionnaire, which assessed various usability attributes of the PMS on a 10-point scale. Attributes related to device usefulness did not have minimum scores of below 5, and the satisfaction section had only a few outliers with scores below 5, but overall maintained a minimum score above this threshold. In contrast, the attributes related to ease of use and ease of learning showed lower minimum scores, with some responses falling below 5 points. For instance, the ease of use had a minimum score of 1, and ease of learning had a minimum score of 2, highlighting areas of concern. Specifically, respondents expressed difficulty in operating the monitor without instructions and noticed inconsistencies in its use, as evidenced by average scores of 6.0 and 6.4, respectively, and minimum scores of 1. Furthermore, respondents reported a slower learning curve and difficulty in remembering how to use the monitor, indicated by minimum scores of 2 for both questions.

COMMON USABILITY CHALLENGES:

The same group of users was asked to provide 3 usability challenges encountered in the use of the IMPALA PMS. One of the participants submitted only 1 challenge, whereas 2 participants reported 4 challenges, and the remainder reported the required 3 challenges. The responses were categorized into 5 key thematic areas: alarms, inaccurate results, hardware, software, and human factors, according to Tscholl [14], but did not include system factors, as they were not reported. Additionally, 5 research nurses working in the HDU were recruited for contextual inquiry to identify any underlying causes of challenges reported by participants, as well as any usability issues that may not have been reported. The extracted main themes, together with subthemes and examples of the comments, are shown in Table 3.

HUMAN FACTORS:

The survey results showed that 17 participants (70.8%) had difficulties using the IMPALA patient monitoring system due to a lack of knowledge or skills. During the contextual inquiry, it also became clear that certain challenges arose as a result of human factors, including a lack of knowledge or skills due to inadequate training. In this study, participants reported struggling with features on the monitors that they were unfamiliar with, leading to a perceived feature creep (the tendency for product designs to become bloated with unnecessary or redundant features) and a lack of usefulness for crucial features. While features such as customized alarm settings, acknowledge, pause, and history were useful to some, only 1 participant had a full understanding of their use, and most participants had challenges to even differentiate the functionality of alarm features, such as acknowledge and pause. In the follow-up interviews, it became clear that participants who received only a simple theoretical orientation training had more difficulties using these features.

HARDWARE:

The study revealed hardware challenges consisting of 4 subthemes: sensor detachment, short battery life, lack of temperature probe, and cable entanglement. Sensor detachment or loose connections were reported as the second most prevalent challenge (58.3%). SpO₂ probes would frequently disconnect, and active children would remove ECG connectors and SpO₂ probes, disrupting continuous monitoring of the patient’s vital signs. Additionally, the battery life of the IMPALA system was a concern raised by 8 (33%) participants. Patient monitors tended to shut down during blackouts of more than 4 h, since the battery did not last more than 4 h. Out of 24 participants, 6 (25%) also expressed the need for a temperature probe, emphasizing the importance of comprehensive monitoring capabilities. Lastly, 16.7% of participants highlighted cable entanglement as a challenge that could compromise patient safety and affect guardians’ handling of the patient.

DATA ARTIFACTS:

Out of 24 participants, 9 (37.5%) expressed that inaccurate readings due to data artifacts, especially in SpO₂ measurements, were a major challenge. This was reported to be caused primarily by poorly fitting sensors and patient movement that affected data acquisition, which in turn resulted in inaccurate vital signs readings, such as falsely low SpO₂ levels that did not reflect the actual patient condition.

ALARMS:

According to the survey results, 8 participants (33.3%) reported experiencing problems related to alarms, with both false alarms and frequent alarms being cited as significant challenges. These findings aligned with the discoveries from the contextual inquiry, in which it was observed that false alarms were triggered more often, which led nurses to become irritated and desensitized, and sometimes led to their ignoring or delaying responding to alarms altogether. Furthermore, it was noted that healthcare professionals usually took time to notice technical issues (detached sensors, NIBP unstable signals or overpressure, air leaks) since they are displayed as text messages and not accompanied by audio alarms that could quickly alert the user. Such technical alarms would last up to 2 h without being attended to, since the nurses sometimes were tasked with other duties or stayed at the nurses’ station rather than in the HDU.

SOFTWARE:

The survey results showed that a few of the nurses (8.3%) reported the challenge of loss of patient data. The issue was also noted during the contextual inquiry, in which patient data in the patient information window was lost, and the participant had to enter the data again. Additionally, the contextual inquiry also revealed that terminology used for certain features, such as “history”, did not align with clinical terminology, leading to confusion among users.

OTHER:

Four participants shared other feedback that did not align with the 5 primary themes; therefore, it was sorted into the “other” category. Specifically, 1 participant noted that prolonged attachment of SpO₂ probes on the same extremity could be harmful to the skin. Additionally, some participants mentioned that connecting the monitor to the patient could be time-consuming.

Through the contextual inquiry, in which each of the 5 research nurses were followed for 2 days, it was discovered that many of the usability challenges were interdependent regarding the root cause. For instance, one major challenge was the frequent occurrence of false-positive alarms, contributing to alarm fatigue. This was primarily caused by poorly fitting sensors and patient movement that affected data acquisition, which in turn resulted in inaccurate vital signs readings, such as falsely low SpO₂ levels that did not reflect the actual patient condition. As a result of these incorrect readings, false alarms were triggered more often, which led nurses to become irritated and desensitized, and they sometimes ignored or delayed responding to alarms altogether.

Similarly, the human factor usability challenge of lack of knowledge or skills due to inadequate training also led to other multiple usability issues. In this study, we observed that participants were unfamiliar with the functionality of features, including acknowledge, pause, and customized alarm settings for alarm management. This lack of understanding also contributed to alarm fatigue, as users became frustrated with frequent alarms from monitors connected to patients with special conditions requiring customized alarm settings. Responding to multiple alarms contributed to cognitive load and led to distraction.

USER-BASED PROPOSED SOLUTIONS AND RECOMMENDATIONS:

During the co-design session for the patient monitoring system, users provided valuable recommendations on the issues identified in the questionnaires and observations. These recommendations focused on several key aspects, including the history feature, alarm functionality, pause and acknowledgment feature, user training, and instructions for use. One notable suggestion was to rename the history feature to “patient trend” or “trend” to better reflect the trends of vital signs, addressing a clarity issue in the terminology. Users also expressed practical concerns, such as their lack of familiarity with certain features, highlighting the need for hands-on training to ensure effective utilization.

Participants emphasized the importance of incorporating a color-coded system in trend tabular displays for quick and accurate interpretation of results, especially when values are out of range in real-world clinical settings. Additionally, there was a notable variation in preferences regarding the presentation of vital sign trends: doctors and clinical officers tended to prefer graphical (wave-form) presentations, while nurses preferred tabular presentations.

Users also stressed the necessity of a delay mechanism for alarms, citing instances in which alarms were triggered by minimal sensor movement. They underscored the significance of customizable alarm settings based on patient age and conditions, reflecting the dynamic nature of healthcare scenarios. Participants emphasized that local customization should involve alarm threshold adjustments handled by healthcare professionals.

Regarding instructional materials, users preferred a digital manual accessible via phone for on-the-go reference, suggesting that the paper manual should serve as a quick reference job aid rather than an extensive booklet.

Discussion

LIMITATIONS:

The limitations of this study are primarily related to its small sample size, as it exclusively involved nurses and clinicians from the pediatric ward, due to the availability of IMPALA patient monitors solely in the pediatric HDU at Zomba Central Hospital. This limited scope necessitates the inclusion of a broader spectrum of healthcare professionals and diverse hospital settings in future studies to enhance the generalizability of findings. Furthermore, the reliance on questionnaires and contextual inquiry as indirect usability testing methods limited the study’s ability to measure performance and gather psychophysiological data, which could provide deeper insights into error rates, task completion rates, and more objective feedback. To address these limitations, future research should incorporate a wider variety of patient monitor models and employ direct usability testing methods that enable comprehensive evaluation of user interactions and system performance.

Conclusions

This study is unique as it is the first to assess a PMS specifically adapted for use in an LRS. The use of a mixed-method approach has provided valuable insights into the necessary design and implementation adjustments needed for successful adoption. Successful implementation of PMS in LRSs hinges on context-specific, user-centered design and comprehensive training. Our study demonstrates that improvements that were implemented in version 2.0 contributed to the positive reception and utility of the continuous patient vital sign monitor among healthcare professionals in an LRS. Users expressed satisfaction with the device, highlighting its potential to enhance routine clinical work. The extension of monitor battery life to 4 h was a notable improvement, partly addressing a crucial need for continuous patient monitoring. Moreover, the engagement of users through co-design sessions proved invaluable in embracing user-centered design principles. This empowered users to contribute to meaningful solutions, thereby fostering greater acceptance and utilization of the technology.

However, there are critical areas that require further attention. The design of robust and durable SpO2 sensors for children under 5 years of age needs improvement to ensure reliable data acquisition, especially in the presence of patient movement. Additionally, extending battery life beyond 4 h is essential to support continuous monitoring in environments where power outages are common. Enhancing user training with practical, tailored content is also vital to optimize the system’s effectiveness and to ensure confident use among healthcare professionals of varying experience levels. These findings underscore the importance of ongoing adaptation and user involvement in the design and implementation of PMSs.