Feedback is important for any design process. Whether it is software or a new cocktail recipe, we rely on testers to ensure that the final product is acceptable. What happens, however, when the feedback received is misleading? Then what?; usually hours of soul-searching, head-scratching and frustration. Everyone who has designed software has gone through this at some point, so it was nice to see someone had actually taken the time to shed some light on why user-feedback is sometimes quite sincere and very misleading.
Morita and Cafazzo have done every software designer a solid by pointing out specific ways in which user feedback can throw off development efforts. In Challenges and Paradoxes of Human Factors in Health Technology Design, the authors describe three paradoxes they have encountered when interpreting user-feedback. They offer detailed descriptions of the paradoxes (abridged versions appear below).
The paradox of expertise: “Do As I Do, Not As I Say”
Subject-matter experts (nurses, physicians, allied health professionals, among other care providers and patients) are involved in the early stages of product development through interviews and focus groups. Their feedback forms the basis of system’s’ specifications. These experts are integrated into the design process because they are considered to bring domain knowledge that is otherwise not available to the design team.
Consequently, as experts, their interactions with the medical technology under development are influenced by their extensive knowledge and well-aligned, mature mental models.
The main premise is that experts will offer greater knowledge in defining product requirements, defining workflows, etc. However, the issue with this approach is that the health technology being developed now reflects only the interactions and constraints of a small percentage of the total users (often, only expert users) who will be interacting with it.
This is good, old expertitis. Experts do not do things in the same way that novices or those with intermediate-level skills do (2). Designing systems based only on expert users can result in the paradoxical effect of being inappropriate for those of lower skill levels. Consider what this means for decision support. Experts have background knowledge and rules of thumb that they have internalized and are not consciously aware of. I learned this lesson when I got my first apartment. My mother was a great cook and never (that I could ever tell) used a recipe. After a few weeks of eating take-out, I wanted a home-cooked meal. Asking for a recipe for one of my favorite dishes proved to be a complete waste of time—what I was told never worked.
Creating decision support systems based solely on experts assumes that those at lower skill levels will have equivalent clinical knowledge, history-taking ability, and physical examination skills. Nope.
The Paradox of Preference Versus Performance: How Could Someone Like Something They Cannot Use?
One would expect that when evaluating two possible designs, users would prefer the design in which they had greater success during testing. Oddly, that is not always the case, leaving the HFPs to conclude that the testing was somehow flawed, or they just disregard that user’s opinion altogether.
As design methods have evolved, more approaches have been made available to influence user behavior by making simple changes in the aesthetics of the device or by using a seemingly novel and engaging control interface. New features might drive users to prefer a particular design simply due to increased affinity for that experience.
In practice at Toronto General Hospital’s usability labs, cases have been observed where the color palette of a device had a greater influence on nursing preference than on its usability.
This is an unexpected finding, but now that I think about it, I have seen similar reactions to software in the past. Back in the days when PCs were new, a friend bought a check-writing program about which he waxed poetic. I considered getting a copy, but watching him, it took far longer to manage an account using the computer than manually. He loved it, but I could never figure out why. Aesthetics are important no matter what the domain. Of course, if users like the way your software looks, but are worse off by using it, you don’t have a sustainable product.
The Paradox of Choice: Less Is Often More
Choice consists of a mental decision-making process in which individuals have to judge merits among a range of options available and select one.
Within a health care perspective, designers can have the misconception that including more features in a product would be beneficial to caregivers and patients, who would now have a wider range of functionality and operational modes to use and more features to tailor their care.
Health care is a highly demanding work environment where caregivers are generally under extreme pressure, which is a perfect situation for excessive choices to become overwhelming and a nuisance at a minimum, and safety hazard at its worst. Adding more choice and options to a single user interface can create uncertainty and distraction to the user.
One thing I learned from building custom software is that users always want more features. Many they will never use or use rarely, but they want them there anyway. For something like an EHR system, which many different professionals will use, “too many choices” is an easy mistake to make. Improving such designs requires knowing what users need in specific situations, which is difficult without detailed workflow knowledge. Since clinical work consists of processes that consume and produce information, information systems that support clinical work must take processes and information into account. Systems without explicit process representations have no way of knowing (or, at least, it is difficult to know) what process is under way and what information is needed. As a result, there is a kitchen sink approach to navigation and UI. Everything is crammed in, and users need extensive training to master the software.
The authors offer the following advice for addressing these paradoxes.
The lesson to be learned from the paradoxes described in this paper is that to design health technology aligned with the needs of its final users, engineers and manufacturers must incorporate a gamut of UCD methods in the design process to gain a comprehensive and realistic understanding of the work domain and user constraints. Observational methods such as cognitive walkthroughs and usability testing provide an opportunity to gather information about how users actually use the technology. The data gathered through these two methods can help minimize the impact of the paradox of expertise and the paradox of preference versus performance, allowing designers to focus on tailoring the technology based on unbiased usage data.
Other methods such as interviews and concept mapping can be used to address the effects of the paradox of choice, creating opportunities for designers to identify the needs of each health care professional and organize the requirements into a manageable and tailored version of the technology.
While I agree that the steps suggested by the authors might be helpful, I think there are additional critical issues that should be addressed if clinical software design is to advance. First, there is a need for much better models of clinical work.
Clinical workflow analyses done in healthcare settings rely on flowcharts, and flowcharts cannot capture important workflow properties in a consistent manner. The information movements and feature-access behaviors captured using cognitive walk-throughs and usability testing must be recorded in a way that makes it easier for software developers to translate those findings into programming code. A more expressive workflow modeling language is required such as Colored Petri nets, BPMN or YAWL. The simple fact is clinical workflow analyses need to be much more quantitative–now they are almost exclusively qualitative.
Secondly, software designs must become more adaptable. Current data-centric clinical software has hardcoded workflows, and changing those workflows requires writing new programming code. Consequently, changing how the software works is difficult, costly, and time-consuming.
When systems are hard to change, then it makes sense to try and determine every possible way the system might be used in advance. Unfortunately, few systems intended for use by a wide-range of different people can be fully specified in advance. Software that supports processes must be amenable to change because processes change. This is a run-of-the-mill coupling and cohesion problem. Process support has to be decoupled from other system components so that it can be altered/changed/upgraded as needed, and the best way to decouple process support is via use of workflow technology.
The final issue arises from an underlying, untested assumption accepted by nearly everyone in and around HIT. That assumption is that the ideal system for supporting clinical work is a chart. There was a time when paper was all we had, so we had to make do, but not anymore. In hospitals and large multispecialty groups, with a few exceptions, everyone used the same paper chart. Naturally, the paper chart became the metaphor on which electronic systems are based. Thus, although there are many different types of clinical professionals electronic systems, like paper, attempt to accommodate everyone using the same interface. Why?
It is impossible to design one UI that every single person is going to be equally comfortable with. Yet, much of the UCD/usability work being done for EHR systems attempts to make these systems better while leaving hardcoded workflows and one-size-fits-all UIs in place (see A Usability Conundrum: Whether it is EHRs or Hospital Gowns, One Size Never Fits All…).
The choice paradox can be addressed in two ways: 1) by creating UIs that are role-based, and 2) by making the UI user-configurable. All clinicians do not do the same thing. Why should there be a single UI??? Why spend time agonizing over the ideal color, font, window position or other design element property when each user could be offered the ability to configure the UI to suit him/herself?
Consider this: How would UCD and usability efforts differ if UIs could be configured by users, workflows (information, tasks, steps, etc.) were explicit and could be adjusted through a graphical interface, and UI designs were role-based? All of these software features are possible…now… Why not try them (3)?
- Morita PP, Cafazzo JA Challenges and Paradoxes of Human Factors in Health Technology Design JMIR Human Factors 2016;3(1):e11
- National Research Council. How People Learn: Brain, Mind, Experience, and School: Expanded Edition. Washington, DC: The National Academies Press, 2000 End Fragment
- Is the Electronic Health Record Defunct?