NIST Usability Resources—A Goldmine for Developers

by Jerome Carter on February 1, 2012 · 1 comment

I have just finished rereading the NIST Draft EHR Usability Protocol (EUP).   I am even more impressed with its quality now that I have a new software project underway.   When developing software, creating an actionable set of requirements is difficult.   Working with users only solves part of the problem because naïve users (those who have never used a system of the type under development) have only a vague idea of what a system should do.    I have seen many HIT projects end in disaster because developers gave users what they thought they asked for.    The workflow assumptions built into an EHR are a good example of translation mishaps.

Every clinician has his/her own personal work habits.   Building a system by canvassing a group of clinicians for information about what they do, the steps involved, the information required at a given step, and other workflow-related issues will result in information that the developers must interpret  in order to create a working system. However, since what emerges will be a distillation of the commonalities between users, the final system may not be ideal for anyone consulted.   This is where the NIST information becomes helpful.

The EUP, along with the Guide to the Processes Approach for Improving the Usability of Electronic Health Records and Customized Common Industry Format Template for Electronic Health Record Usability Testing, provide developers with concrete data on common usability problems that can be used to complement and enrich data-gathering interactions with end users.   The guidance in these documents allows developers to uncover many workflow issues during design phases, when they are more easily (and cheaply) corrected, instead of relying predominantly on end-user test results.

While these documents are a boon for developers, they also hold value for anyone buying an EHR.   The expert review questions/checklists in the EUP  are easily converted into screening tools for EHR purchasers trying to evaluate products.   Even a small subset of the checklist questions, when morphed into an evaluation script, is likely to be a better screening tool than most potential buyers usually have.

It will be interesting to see how the federal push for EHR usability affects the marketplace.  Knowing that a product has usability issues and fixing them are two very different matters.     The usability evaluation protocol focuses on user interface issues. Poor usability at the user interface level could easily be a symptom of a deeper design issue.  In such cases, fixing the usability issue might require rewriting significant portions of the program.  Going back to our workflow example, consider how an EHR product with hard-coded workflow pathways differs from one that has a configurable workflow engine.   Correcting usability issues in the former would require significant coding, whereas in the latter it might only require a change in configuration settings.   Systems with layered, component-based architectures that allow for “separation of concerns” will have a competitive advantage.

Considering both the lag time before significant product changes may occur and the current push to have as many clinicians as possible using EHR systems by 2015, in three years will providers discover they’ve gone through expensive implementations with outdated products?

Whatever happens in the marketplace, NIST has provided EHR developers and EHR purchasers a goldmine of high quality resources for evaluating EHR products.  If you are in either category, read them—you won’t regret it.



Leave a Comment

{ 1 comment… read it below or add one }

The Usability People (@UsabilityPeople) July 22, 2013 at 3:52 PM

The Office of the National Coordinator for Health Information Technology (ONC) has released a set of new certification and meaningful use requirements for electronic health records (EHRs). These require that EHR vendors include evidence of user-centered design and user test results in their certification submission. To be able to obtain the ONC certification (and meaningful use funding) EHR vendors must follow a formal User Centered Design (UCD) process and perform summative usability testing on specific areas of the product.

User-Centered Design: Helping users become Effective, Efficient, and Satisfied

We recommend that EHR vendors follow the ISO 9241-11 standard. ISO-9241 Part 11: (1998) pertains to the extent to which a product can be used by specified users to achieve specified goals with Effectiveness (Task completion by users), Efficiency (Task on time) and Satisfaction (responded by user in term of experience) in a specified context of use (users, tasks, equipment & environments).

Summative testing

Summative usability testing is an industry standard usability methodology (see ) During the testing a trained user experience professional works with the EHR vendor to create a series of representative tasks and then measures the amount of time it takes to complete the task (time on task), how many and what type of errors occur (error rates), and satisfaction with the task interaction (user satisfaction). The results —as measured by things like task times, assisted and un-assisted completion rates and standardized satisfaction scores –can be benchmarked against current industry standards.

EHR developers can and should perform many iterations of user testing. These early iterations are often called Formative testing. We also recommend that an Expert (Heuristic) review be performed before any of the more formal usability testing. This is a cost-saving measure that can significantly reduce the number of iterations required and ultimately reduce the number of “items needed improvement” presented in the ONC submission. The submission that is ultimately provided for testing and certification may be the expression of a final iteration in which few areas for improvement would be identified.

How should the Results Presented?

Results from the summative study should be created and presented to the ONC using the Customized Common Industry Format (CCIF) Template for EHR Usability Testing (NISTIR 7742 see )These reports typically include an Executive Summary, an introduction, a method section, discussion of the results and any appendices?.

The following are required for submission to the ONC:
• Name and version of the product
• Date and location of the test
• Test environment
• Description of the intended users
• Total number of participants
• Description of participants: their experience and demographic characteristics
• Description of the user tasks that were tested
• List of the specific metrics captured during the testing for effectiveness, efficiency and satisfaction
• Data scoring
• Results of the test and data analysis
• Major test findings
• Identified area(s) of improvement(s)

For more information on the ONC Test Procedure for §170.314(g)(3) Safety-enhanced design see

Usability Starts with “U”

We can help you create summative and formative usability evaluations using the NISTIR 7742 Customized Common Industry Format Template for EHR Usability Testing.

Contact Us ( ) for more information about these and other User Experience services that we can provide to your health IT organization.


Previous post:

Next post: