i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1068–1074 journal homepage: www.ijmijournal.com Use of a human factors approach to uncover informatics needs of nurses in documentation of care Michelle L. Rogers a,∗ , Paulina S. Sockolow b , Kathryn H. Bowles c , Kristin E. Hand c , Jessie George d a Drexel University College of Information Science and Technology, 3141 Chestnut Street, Philadelphia, PA 19104, United States Drexel University College of Nursing and Health Professions, United States c University of Pennsylvania School of Nursing, United States d University of Pennsylvania Health System, United States b a r t i c l e i n f o a b s t r a c t Article history: Purpose: The success of health information technology implementations is often tied to the Received 20 April 2013 impact the technical system will have on the work of the clinicians using them. Considering Received in revised form the role of nurses in healthcare, it is shocking that there is a lack of evaluations of nursing 20 June 2013 information systems in the literature. Here we report on how a human factors approach can Accepted 7 August 2013 be used to address barriers and facilitators to use of the nursing information system (NIS). Human factors engineering (HFE) approaches provide the theoretical and methodological Keywords: underpinning to address these socio-technical issues. User–computer interface Methods: This study investigated the use of an NIS, a module of the electronic health record Health information technology (EHR) previously implemented throughout the hospital system. The study took place in Nursing informatics two hospitals (760 beds and 300 beds) within a three-hospital health system. Earlier in the Workflow year, the NIS was implemented throughout the health system. We applied a scenario-based Electronic health records evaluation technique in order to understand the barriers and facilitators to nurse use of the Information systems NIS as part of improving the healthcare delivery system. The scenarios were designed to have the nurses interact with the major components of the NIS. The research team developed the standardized scenarios to cover the major functions of the system. Results: Twelve nurses completed the study and results show that documentation within the NIS was hindered by several aspects of the interface. This paper discusses the themes associated with the usability of the NIS interface analyzing them using usability heuristics. The team also identified facilitators to use and proposed avenues to support or enhance these facilitators. Conclusions: This study examined the use of an NIS to standardize care and documentation in nursing. It used scenario-based usability testing, applying the “think-aloud” protocol technique to assess the use of the NIS in documenting patient care. This method of usability evaluation exposed an understanding of how nurses use the NIS and their perspective on the system. We hypothesize that this method will offer key insights into how the usability of the NIS not only impacts use but also informs redesign opportunities. In addition, this is one of the few rigorous studies of NIS and provides direction and recommendations for informaticians, developers and nurse decision makers. © 2013 Elsevier Ireland Ltd. All rights reserved. ∗ Corresponding author at: Drexel University, 3141 Chestnut Street, Philadelphia, PA 19119, United States. Tel.: +1 215 895 2922; Fax: +1 215 895 2494. E-mail address: mrogers@drexel.edu (M.L. Rogers). 1386-5056/$ – see front matter © 2013 Elsevier Ireland Ltd. All rights reserved. http://dx.doi.org/10.1016/j.ijmedinf.2013.08.007 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1068–1074 1. Introduction Healthcare information technology (HIT) holds great promise for improved patient outcomes, increased cost-effectiveness, and better patient and staff satisfaction [1,2]. The implementation of new technologies in hospitals often has unanticipated, and frequently undesirable, consequences on workflow [3,4]. Nursing information systems (NIS) are being implemented in an increasing number of facilities with the goal of supporting interdisciplinary communication improving quality of care and increasing patient safety [5]. It should be noted that an NIS contains data collection and integration functionality for nurses and could be used in addition to or alongside an electronic medical record. Often the success of clinical information system implementations are tied to the impact the systems will have on work. These outcomes can be the result of a design based on over-simplified models of human decision making and work practices. The formal descriptions of work practices, such as written procedures, are idealized and may not include important complexities and subtleties that are present in the actual work [6]. In order to understand the impact of such technologies on the work of nurses, it is necessary to study the environment and practice before the technology is implemented. This cannot be done with the traditional randomized control trial methodology. While early studies of clinical information needs focused on physicians, more recent studies have found that nurses’ information needs differ from those of physicians, and may include information about protocols and procedures [7], drug therapy, and diagnosis [8]. Nursing responsibilities in healthcare organizations run the gamut from educating patients to medication administration to executing physician orders and ensuring safety. Several researchers have pointed out the danger in trying to compare nurse and physician use [5,8,9]. Human factors engineering (HFE) approaches provide the theoretical and methodological underpinning to address these socio-technical issues [10,11]. HFE is a discipline that seeks to design devices, software, and systems to meet the needs, capabilities, and limitations of users, rather than expecting the users to adapt to the design of the system [10]. The field is multidisciplinary and benefits from the input of experts from diverse domains such as psychology, engineering, computer science, biomechanics, medicine and others. Since the NIS requires a considerable amount of cooperation from a variety of personnel [9], it is imperative that we understand how their processes are changed by the introduction of technology. Often that is most easily understood by investigating the design of the user interfaces. User testing offers a valid and reliable method to investigate the complex factors that impact work [12–14]. Within the literature regarding NIS use and implementation usability and usefulness have been identified as an under investigated yet influential factor affecting practitioners’ acceptance [15,16]. Usability is a complex construct defined by both ease of use and learnability. Heuristic evaluation considers different aspects of an interface. The most widely used and validated heuristics were conceptualized by Nielsen et al. [17] and include ten (10) heuristics (see Table 1). 1069 Here we report on how a HFE approach can be used to investigate the impact of usability and usefulness of system interfaces on the use of the NIS. The objectives were to understand if usability heuristics can be used as an analysis tool given that they can be used in the design of successful human–computer interfaces. 2. Methods We applied a scenario-based evaluation technique in order to understand the barriers and facilitators to nurse use of the NIS as part of improving the system. Usability heuristics were then used to analyze the resulting themes that emerged from the evaluation. We hypothesize that this method will offer key insights into how the usability of the NIS not only impacts use but informs redesign opportunities. 2.1. Organizational setting For this study, a case study methodology was used and approval from the researchers’ academic institution’s Institutional Review Boards was secured. Strong backing from nurse leadership at the health system level as well as in the individual hospitals was obtained. This study investigated the use of an NIS, a module of the electronic health record (EHR) previously implemented throughout the hospital system. Major components of the system include clinical practice guidelines, care plans and the ability to document nursing care. Included in the NIS are approximately 200 evidence-based, interdisciplinary clinical practice guidelines (CPGs) from which nurses select to guide the patient’s care during a visit. CPGs are recommendations that are identified to assist in clinical decision making. Earlier in the year of the study, the NIS was implemented throughout the health system. Nurses attended eight hours of training in advance of the implementation. A complete description and evaluation of the NIS implementation is discussed more fully in another publication [18]. The study took place in two hospitals (760 beds and 300 beds) within a three-hospital academic health system. 2.2. NIS use The NIS was used to document patient related issues including the admission profile, physical assessment, educational interventions, vital signs, intake and output measurements, medication administration, assessment findings, interventions completed, and significant event summaries. Upon patient admission, nurses access the NIS to document the clinical visit. Once a clinical practice guideline (CPG) is selected, the patient record is populated with care plans that can be used to collect structured data regarding assessment and education. Nurses would then be prompted to complete documentation based upon the condition and/or problem identified upon admission. Care plans, designed with information from CPGs, are further individualized by nurses for each patient. Content from the care plan populates throughout the assessment and education flow-sheets producing a comprehensive and detailed assessment specific to the chosen plan of care, prompting nurses to recognize important elements of 1070 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1068–1074 Table 1 – Nielsen’s usability heuristics and their definition. Heuristic Visibility of system status Match between system and the real world User control and freedom Consistency and standards Error prevention Recognition rather than recall Flexibility and efficiency of use Aesthetic and minimalist design Help users recognize and recover from errors Help and documentation Description Addresses the manner in which the system keeps users informed about what is going on, through appropriate feedback Addresses using clear language and concepts familiar to the users Addresses navigation and items to help a user to find his/her way. For example, the ability of the nurses to move from screen to screen Addresses keeping consistency with similar labels and items throughout the system. For example, the “save” function looks the same on each screen Addresses the way in which the system supports ease of use. For example, displaying which fields are mandatory with an asterisk Addresses minimizing the load on memory by making functionality and options visible. For example, users do not have to remember items from screen to screen Addresses ways to reduce the work for some tasks. For example, quick-links, “saved searches” Addresses how elements of the design should not obstruct the function. For example, reducing clutter, clear action buttons or text Addresses ways to help the user recover from this problem in the easiest way possible. For example, useful error messages Addresses ways to describe the system instructions that are clear and concise. For example, FAQs, “?” icons, advanced search the selected care plan. The NIS is accessible on newly installed computers in each patient room as well as existing computers in the hall and at nursing stations. • selecting a new CPG, • wound care, • discharging a patient to home. 2.3. A sample scenario for selecting a new CPG read: “A patient has been on the unit for 3 days but the Clinical Practice Guideline (CPG) has not been filled out. What do you do?” A different set of scenarios and questions were asked of each participant, ensuring that all scenarios and questions were asked at least once for each unit. Additional questions prompted by the responses and not on the field note form were asked of the participants. Nurses were recruited until saturation was reached, that is, observations offered no new information or a functionality was seen at least three times. One member of the research team would introduce the scenarios and served as the facilitator of the testing. As participants completed the scenarios, they were prompted by one member of the team to “think-aloud’ or verbalize their thoughts. Probes included phrases such as “what did you expect to happen” or “is this how you usually use the system”. The exact terms used in the probes depended on when the person stopped verbalizing while they used the system. At the end of the scenarios, the participants were asked a series of follow-up questions to elicit general attitudes towards documenting in the NIS, team communication and NIS impact on workflow. Participants A purposeful sample of nurses that worked on units in the two hospitals was approached. Inclusion criteria used to select the units within the hospitals were (1) representative of most units in the hospital, e.g., medical or surgical unit; (2) had a conference room with a computer where the study could be conducted; and (3) had a contact person on the unit known to a research team member (to facilitate introduction of the study to the staff). For participants to be eligible, it was necessary for them to be registered nurses who provided and documented patient care. Nursing leadership agreed to allow us to approach staff that were currently working. A member of the research team randomly approached nurses were who were working that day if they were available for 20 min. All subjects had completed the training for the NIS, but experience varied among subjects. A research team member obtained individual consent before the participant entered the conference room. 2.4. Data sources and sampling The research team included three academic researchers and two clinical nurses. The study was conducted from March to May of 2012 in private conference rooms on units that had a computer with access to the NIS. One member of the research team presented scenarios to the nurse participants as a modified think-aloud protocol [17], which is a standard methodology used to elicit data about cognitive reasoning that occurs during a problem solving task. The scenarios were designed to have the nurses interact with the major components of the NIS. The research team developed the standardized scenarios to cover the major functions of the system: • admission to the unit, • patient falls, 2.5. Data analysis Each evaluation session was audio recorded, transcribed, content analyzed and coded for themes by members of the research team (MR, PS, KB). The content analysis was independently completed by each of the researchers analyzing the transcripts of the answers to the scenarios and interview questions. The inter-rater agreement goal was 100% and discussion was held until it was reached. Similarly, the data were individually coded for themes. After the coding was completed, the themes were mapped to the concepts and components within the Health Information Technology Reference-based Evaluation Framework (HITREF). The HITREF is a comprehensive i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1068–1074 framework firmly grounded in research evidence that provides a comprehensive list of 20 evaluation criteria related to HIT characteristics [19]. The criteria are grouped into six dimensions: (1) structural quality, (2) quality of information logistics (usability is included here), (3) effects on quality of processes (workflow is included here), (4) effects on outcomes and quality of care, (5) unintended consequences or benefits, and (6) barriers or facilitators to clinicians’ adoption. This paper discusses the themes associated with the usability of the NIS interface in particular. Those themes coded as “usability problems” and “impact on workflow” were selected for further analysis in this paper using usability heuristics. All discrepancies in coding were resolved by discussion until consensus was reached. A list of usability heuristics is described in Table 1. 3. Results Participants were 12 registered nurses (RN’s) on two units of the flagship hospital and one unit of an acute care hospital – four nurses from each unit. To protect their anonymity no socio-demographics were collected. 3.1. Study findings The data were analyzed using the usability industry standard heuristics (discussed n Table 1) as the guiding framework. Our analysis evaluated 9 of the 10 usability heuristics. We chose not to evaluate “Help documentation” as it was outside of the scope of this study and the HITREF framework. What follows is a description of the interface characteristics, representative examples from the analysis as well as the impact on workflow. 3.1.1. Visibility of system status Positive demonstration: Visibility of system status is key to enabling users to understand how best to use an information system and take advantage of the data that is generated and stored. We did observe several interactions with the system that were positive in terms of visibility of system status. For instance, there were tabs in the interface for significant events, which allowed the nurses to know if anything significant occurred even if they were not told this information during the shift change. Violation: Violations of this heuristic were most often observed in interdisciplinary communication. Nurses repeatedly reported that they were unaware of which of their colleagues saw their documentation. As a result, several of the nurses would call a colleague in addition to documenting in the NIS in order to make sure the person saw the documentation. Example coded statements include “. . .Well, there is the provider notification note, I would do that too after I spoke with the provider, but I’d speak with them first, then I’d write the note about it. I don’t know if they even see that or check it.” While the functionality to alert the doctor was present to serve as a facilitator of knowledge sharing across disciplines, it was not used as designed. There was no feedback to users and thus nurse were not confidant that others saw their documentation. An additional way this heuristic was violated was in the lack of feedback regarding data input. Nurses reported 1071 entering data and not knowing where the data went in the system. This resulted in wasted time looking for that information. An example coded statement is “In this system you can’t do [things] side by side, always on only one aspect of it, you can’t look at it overall, get a general idea”. 3.1.2. Match between the system and the real world Positive demonstration: The NIS had limited success in managing the match between the system and the real world. Positive comments that emerged included “. . .can follow much easier what other disciplines do” and “We get to see what everybody is documenting on, and then we can call them out on it”. Violation: Where this heuristic was violated most was in the reliance of several nurses on the use of free text areas. The nurses reported they used the free text capability in the NIS because they could not find a concept in the options made available to them. As demonstrated in the quotes below, nurses found it easier to use free text than to select an inaccurate option. Example coded statements include “I free text a lot, ‘cause it’s, sometimes not everything is in there that you need. Or other times it’s just easier to free text to write in what, something that’s more, that fits better, or makes more sense. ‘Cause not everything is in the 20 check boxes. . .” Similarly, nurses stated, “There are some of them [patients] that come in and nothing really applies. So you’re grasping at straws to put something in. And you have to have something. And it’s populating things that make you say I’m following a plan of care and it’s really not even appropriate, so it’s kind of weird”. Using free text is a problem in that it doesn’t lend itself to searching when nurses and or researchers are looking over the data at a later date for analysis. 3.1.3. User control and freedom Violation: The nurses reported frustration with how the NIS hindered them from completing tasks they wanted to do because of navigation issues. Example coded statements include “. . .there is so many [guidelines], and the one that you want is so hard to find. And then when you do find it, there’s like 17 pages of stuff . . .” As the example quote states, nurses didn’t feel the interface supported their information needs or allowed them the control to be able to find needed items. 3.1.4. Consistency and standards Positive demonstration: The NIS maintained consistent terms and labels throughout the system. The participants did not report any difficulty with using the NIS because of inconsistency in terms or labels used. 3.1.5. Error prevention Description: Addresses the way in which the system supports ease of use. For example, displaying which fields are mandatory with an asterisk. Positive demonstration: There was some planning to prevent errors in the design of the NIS. Example coded statements include 1 − “There is always a flag that pops up that says new order or new result or something like that”. 2 − “[the CPGs] are helpful especially some of the times we get surgeries I’m not familiar with. So I know what to look for, sometimes 1072 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1068–1074 abdominals surgeries, GI surgeries, when they come over, I can look for different things that I might not look for”. Violation: Nurses reported to be wary of using some functions of the system because they were unclear of what their action would do and they didn’t want to make the documentation more complicated. A sample quote “you click something and a thousand things open up, click something else, and then you forget what you originally went in for.” They spoke of developing work-arounds to complete their necessary tasks. 3.1.6. Recognition rather than recall Positive demonstration: There were some positive comments identified, “There is an automatic skin bundle that reminds people of little things to do to prevent skin breakdown”, Similarly, “I find especially with my thoracic patients, I really like it because they have so much going on, they have a-lines, they have chest tubes, they have epidurals, and it’s easier to be like, up forgot to look at this and you can just peek at ‘em real quick and finish your charting” Violation: This heuristic was violated frequently. Often nurses felt as if they had to remember the actions they had completed and what was still left for them to do. A quote that is representative of the group – “. . .a lot of times it will skip right over that safety piece [falls risk], so unless you know you have to document on it, it’ll skip, it’ll miss it. . .I have to usually go back and find it, like scroll back all the way up.” Another statement, “. . .you click something and a thousand things open up, click something else and then you forget what you originally went in for.” As the quotes demonstrate, the system did not assist in easing the memory load of the nurses. 3.1.7. Flexibility and efficiency of use Positive demonstration: The NIS did offer limited assistance in terms of flexibility and efficiency of use. A representative statement “. . .when you have people that come in again and again, that their admission sheet’s already done.” Violation: Nurses seemed to take advantage of this heuristic only after interacting with other users. The ability to save frequent actions or use shortcuts was not often readily available or sufficiently communicated. 3.1.8. Aesthetic and minimalist design Violation: There was only one observation of a violation of this heuristic. A nurse reported that the system had some information which was irrelevant to their documentation or was rarely needed. 3.1.9. Help users recognize and recover from errors We did not observe the users experiencing errors or having to recover from them. 3.1.10. Help and documentation We did not test the help documentation nor did we observe nurses using the help documentation. 3.1.11. Impact on workflow Responses that emerged from the analysis of the scenario testing transcripts for those items that changed how the nurses completed their care or documented what was done were coded as “impact on workflow”. Positive impact on workflow was identified in ability to document at the bedside, notification of new orders or new results, the ability to see documentation from other clinicians and the ability to document CPGs. For instance, the staff reported that the NIS had visual cues on the interface to signify changes in the status of the patient, orders or documentation needs. This supported nurses’ information needs and could keep them up to date on the status of the patient. Each of these behaviours was reported by nurses to improve their efficiency and effectiveness of their work. Negative impacts on workflow identified include (1) limited visual feedback regarding which clinicians view their documentation; (2) complicated menu structure and navigation; (3) the NIS screen inputs not matching clinical practice. Having limited visual feedback in the NIS was identified as a problem because nurses felt a need to develop back up communication plans since they were not sure that other members of the clinical team saw their documentation. For instance, nurses repeatedly reported that they called the doctor taking care of their patient to communicate updates in status in addition to documenting in the NIS. They stated that they were not sure if their colleagues saw their documentation so they made additional calls to assure that the information was noticed and shared. Secondly, the complicated menu structure hindered workflow in that nurses expressed a burden in trying to locate where they were to document different aspects of the care. Similarly, often the screens required what they deemed a “significant amount of scrolling”. Finally, when the NIS screen inputs did not match the clinical practice, nurses reported to have to document care in ways that increased their time spent on documentation. For instance, while completing certain CPG’s, the interface presents the data fields to be completed in a certain order, e.g., specific values are entered in a specific order rather than usual order completed during the physical exam. Nurses reported being frustrated with this aspect of the system because this method of data entry forced a specific sequence of values to be entered which did not match the way the clinical exam or data collection usually went. This resulted in wasted time for the nurses because they would have to complete a section on the interface that didn’t match the physical activity they were doing. Often it would require them to do their exam in a disjointed fashion because of the way the data was entered into the NIS and they could not change that order. 4. Discussion This study used a novel human factors approach to evaluate the design of the interface of an NIS. We applied the “thinkaloud” protocol analysis method to evaluate the usability of the NIS and the resulting impacts on workflow. The findings provide insight into the flaws of existing interface designs and identifies ways in which the workflow is hindered. When designers and administrators of health information systems speak of usability, they often only consider how the interface looks or functions. What is often missed is the impact that the design of the display and information presented have on workflow. Nurses’ use of HIT and the specific issues raised by nursing practice have become even more important as EMR implementations have struggled [20,21] to be successful. Our i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1068–1074 1073 Table 2 – NIS design flaws and related recommendations. Heuristic Visibility of system status Match between system and the real world User control and freedom Interface design flaw Design recommendation Requirement to document in multiple locations Menu’s without applicable text Error prevention Functionality does not have feedback Recognition rather than recall Flexibility and efficiency of use Entered data not propagated throughout the system Shortcuts not visible Aesthetic and minimalist design Irrelevant data cluttering screen finding that nurses were not able to capitalize on the information available to them because of the lack of usability of the interface is similar to the work of Michel-Verkerke [8]. Nurses are often required to manage and balance relationships with multiple groups of clinicians usually distributed by time and space. As we saw in this study, lack of trust in the NIS (as reported by the participants) led nurses to not rely on the system to manage communication within teams. Given the dynamic nature of health care, a constant challenge is the maintenance of an accurate picture of each case and work pending. At any given time, the nurse may need to monitor multiple activities going on in the care of their patients and work done by other staff. This study showed that when visibility is difficult, efficiency work-arounds were an expected outcome. For instance, nurses reported to call other clinicians to report updates on their patients even though they also document care in the NIS. Designers, decision-makers and system implementers would do well to consider the impact of the use of the NIS on the tasks, organization, users, and physical environment. The implications of these findings can be directly mapped onto design recommendations (Table 2). The human factors literature has identified design heuristics that can be used to guide useful and usable interface design. Given the design concerns raised in this article, there are a number of existing human–computer interaction concepts applicable to a system such as this [11,14]. Qualitative methods have emerged as a rich source of data to complement the traditional quantitative evaluation [8,12] We were unable to find any studies that capitalized on the strengths of the “think-aloud” method to analyze use of an NIS and the impact that the design of the interface has on the workflow of nurses. We feel that the findings from the current study are critical to build an understanding of how to design an NIS to positively support workflow as well as evaluate the resulting design. 5. Limitation This study is limited in that it only looks at one NIS and the nurses who volunteered for the study. Thus, we may have nurses who are more likely to be critical of the system. Make status information visible, especially changes Reduce over-reliance on memory Increase observeability of work to encourage a shared mental model of the work environment Support ability to plan/predict work memory prompts Make status information for other services and patients visible Make shortcuts visible Increase system flexibility Allow for system modifications/customizations However, none of the participants would agree to go back to using paper for clinical documentation. There was an acknowledgement that the system was useful to some extent. 6. Conclusion This study examined the use of an NIS to standardize care and documentation in nursing. It used scenario-based usability testing, applying the “think-aloud” protocol technique to assess the use of the NIS in documenting patient care. This method of usability evaluation exposed an understanding of how nurses use the NIS and their perspective on the system. While there have been limited studies that have investigated the use of NIS, this is the first study to our knowledge that applied scenario-based user testing as a way to evaluate the impact of the usability of the NIS interface on clinical workflow. Information can be manipulated electronically, and thus the design of work is no longer constrained by physical objects. The constraints on work processes and activities are more likely to be imposed by the organization, environment, task and individual rather than by the technology itself. As was seen in this study, the NIS is a social system as well as a computer system since it is a means of communication among care team members. It should then be studied as such. Even though the patient data are recorded and manipulated electronically, the tasks to complete the work process are still manipulated to meet the needs of the staff. Work processes and activities are both altered when an NIS is introduced into a clinical environment albeit in different ways. Authors’ contributions Michelle Rogers wrote the majority of the manuscript and participated in the analysis. Paulina S. Sockolow was the originator of the project, participated in the analysis and did substantial editing of the manuscript. Kathryn H. Bowles participated in the analysis and edited the manuscript. Kristin E. Hand participated in the study and the analysis as well as an edit of the manuscript. Jessie George participated in the study and the analysis as well as an edit of the manuscript. 1074 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1068–1074 Summary points What was already known on the topic? [7] • Nursing information systems (NIS) are being implemented in an increasing number of facilities with the goal of supporting interdisciplinary communication improving quality of care and increasing patient safety. • Nurses are often concerned about impact on work efficiency, content design and the redesign of workflow. [8] [9] What this study added to our knowledge? • The use of scenario-based testing is feasible and informative to the redesign of NIS interfaces. • Information generated by the NIS scenario-based testing and usability evaluation provide feedback that can be used inform work practice and system use. Conflicts of interest [10] [11] [12] [13] The authors have no conflicts of interest to report. [14] Acknowledgments This study was made possible by support from the exceptional staff of the Hospitals. [15] references [16] [1] E. Ammenwerth, F. Rauchegger, F. Ehlers, B. Hirsch, C. Schaubmayr, Effect of a nursing information system on the quality of information processing in nursing: an evaluation study using the HIS-monitor instrument, Int. J. Med. Inf. 80 (1) (2011) 14. [2] J.S. Ash, D.F. Sittig, E.G. Poon, et al., Types of untintended consequences related to computerized provider order entry, J. Am. Med. Inform. Assoc. 13 (2006) 547–556. [3] P. Darbyshire, ‘Rage against the machine?’: nurses’ and midwives’ experiences of using computerized patient information systems for clinical information, J. Clin. Nurs. 13 (1) (2004) 17–25. [4] B.-T. Karsh, R.J. Holden, S.J. Alper, C.K.L. Or, A human factors engineering paradigm for patient safety: designing to support the performance of the healthcare professional, Qual. Saf. Health care 15 (Suppl 1) (2006) i59–i65. [5] M.F. Dahm, B. Wadensten, Nurses’ experiences of and opinions about using standardised care plans in electronic health records—a questionnaire study, J. Clin. Nurs. 17 (1) (2008) 9. [6] E.N. Munyisha, P. Yu, D. Hailey, The changes in caregivers’ perceptions about the quality of information and benefits of [17] [18] [19] [20] [21] nursing documentation associated with the introduction of an electronic documentation system in a nursing home, Int. J. Med. Inf. 80 (2) (2011) 116–124. T. Lee, Nursing information: users’ experiences of a system in Taiwan one year after its implementation, J. Clin. Nurs. 17 (6) (2008) 763–771. B. Michel-Verkerke Margreet, Information Quality of a Nursing Information System depends on the nurses: a combined quantitative and qualitative evaluation, Int. J. Med. Inform. 81 (10) (2012) 662–673. L. Poissant, J. Pereira, R. Tamblyn, Y. Kawasumi, The impact of electronic health records on time efficiency of physicians and nurses: a systematic review, J. Am. Med. Inform. Assoc. 12 (5) (2005) 505–516. B.M. Kleiner, Macroergonomics: analysis and design of work systems, Appl. Ergon. 37 (1) (2006) 81–89. A.L. Russ, A.J. Zillich, M.S. McManus, B.N. Doebbeling, J.J. Saleem, A human factors investigation of medication alerts: barriers to prescriber decision-making and clinical workflow, AMIA Annu. Symp. Proc. (2009) 548–552. A.F. Rose, J.L. Schnipper, E.R. Park, E.G. Poon, Q. Li, B. Middleton, Using qualitative studies to improve the usability of an EMR, J. Biomed. Inform. 38 (2005) 51–60. T. Zayas-Caban, J.L. Marquard, K. Radhakrishnan, N. Duffey, D.L. Evernden, Scenario-based user testing to guide consumer health infomatics design, AMIA Annu. Symp. Proc. (2009) 719–723. A.W. Kushniruk, E.M. Borycki, S. Kuwata, J. Kannry, Emerging approaches to usability evaluation of health information systems: towards in situ analysis of complex healthcare systems and environments, Stud. Health Technol. Inform. 169 (2011) 915–919. M.L. Rogers, The socio-technical impact of electronic medical records on the work activities and work processes in a small clinic, in: Proceedings of the 8th International Symposium on Human Factors in Organizational Design and Management (ODAM), June 22–25, Maui, HI, 2005. J.J. Saleem, E.S. Patterson, L. Militello, Impact of clinical reminder redesign on learnability, efficiency, usability, and workload for ambulatory clinic nurses, J. Am. Med. Inform. Assoc. 14 (2007) 632–640. J. Nielsen, Usability Engineering, Academic Press, San Diego, CA, 1993. P.S. Sockolow, K.H. Bowles, M.L. Rogers, K.E. Hand, J. George, Challenges and Facilitators to Nurse Use of a Guideline-based Nursing Information System: Recommendations for Nurse Executives, 2013 (in press). P.S. Sockolow, H.P. Lehmann, J.P. Weiner, Are nurses happier after an electronic health record is implemented at a nurse-managed practice for the elderly? Comput. Inform. Nurs. 27 (5) (2009) 330–331. X. Xu, R.A. Rocha, S.M. Bigelow, C.J. Wallace, T. Hanna, L.K. Roemer, Understanding nurses’ information needs and searching behavior in acute care settings, AMIA Annu. Symp. Proc. (2005) 839–843. E. Törnvall, L.K. Wahren, S. Wilhemsson, Advancing nursing documentation–an intervention study using patients with leg ulcer as an example, Int. J. Med. Inf. 78 (9) (2009) 605–617.