Universal Implications, External Validity, and Thomas Kuhn

Pick one: EHRs lower costs/ EHRs do not lower costs; EHRs improve quality/ EHRs do not improve quality.   Each position has its proponents, and anyone who keeps up with happenings in the healthcare arena will invariably run into one of these positions.  Precepts from logic can help in explaining the disparity in viewpoints between the two camps.  Essentially, many problems occur due to missteps taken when generalizing from a specific case or research project.

On revisiting any recent controversy over the relationship between EHRs and quality or costs, one will find headlines of the form “EHRs (do /do not)  affect X.”   When considering such headlines from a logical standpoint, one should ask:  1) “What is the writer trying to imply?” and 2) “What does the writer wish me to infer?”  For example,  the statements, “We implemented an EHR and the quality of care improved” or “ We implemented an EHR and the quality of care did not improve,”  are statements that can be verified as true or false by examining the data on which the statements are based.  However, statements of the type, “EHRs improve quality “ or “EHRs do not improve quality,” move from the specific case,  verifiable using available data,  to the general case, and take on the aura of  universal implications (i.e. IF an organization implements an EHR, THEN quality improves).   This is where problems arise.

Universal Implications
In logic, statements such as, “IF an organization implements an EHR, THEN quality improves,” are called conditional statements or implications, and have the general form “IF P, THEN Q.”    A universal implication is one that applies to the entire class or set of objects under discussion.  While many mathematical proofs can be difficult, disproving a universal implication is quite simple; all that is required is a single counterexample.  Therefore, when research findings are offered as a universal implication, for example, “For all healthcare organizations, IF an organization implements an EHR, THEN quality improves,” opponents merely have to show a single counterexample to negate it!

Original Implication IF an organization implements an EHR, THEN quality improves. IF P, THEN Q
Universal Implication For all healthcare organizations, IF an organization implements an EHR, THEN quality improves
Negation An organization implements an EHR AND  quality does NOT improve P AND ~Q
Negation of the Universal Implication There exists  at least one healthcare organization that implemented an EHR AND quality did NOT improve.

(“~” = NOT)

As can be seen above, the negation of an implication is never another “IF P, THEN Q “statement; it is an AND statement of the form P AND ~ Q (1).    The negation of the universal implication, “For all healthcare organizations, IF an organization implements an EHR, THEN quality improves” is “There exists at least one healthcare organization that implemented an EHR AND quality did NOT improve.”   Thus, anyone who knows of a case in which quality improvements did not occur after an EHR implementation, will feel quite confident in using his/her counterexample to negate the universal implication. I believe the overuse (or misuse) of universal implications  is a major source of unproductive arguments over research findings.

External validity
External validity provides a second explanation for the disagreements over research findings. It is a concept taken from clinical research and clinical epidemiology.   External validity is the degree to which research done at a SPECIFIC location with a SPECIFIC set of study subjects, applies to ANY location and ANY set of study subjects.   The greater the disparity between the subjects of the original study and those to whom the results are being extrapolated, the greater the likelihood the findings will not apply.

Listed below are just a few factors that can explain how diverse healthcare facilities might demonstrate different  EHR implementation effects—even when using the same product.

Facility Type: academic/community hospital, solo practice/multispecialty group;  Size: 500 beds/30 beds;  Location : urban/rural
Available Resources Medicaid/commercial patient base, access to IT expertise, access to informatics expertise, budget reserves, private/public sector
Implementation processes/practices Professional project management, process/workflow analysis skills, training, potential employee pool
EHR system Usability, implementation process, customization, interoperability features, workflow adaptation capability
Others Corporate culture, leadership involvement, vision

Since there are currently no standards for judging a facility’s preparedness for EHR implementation nor any formal  means for determining what constitutes a successful  implementation, judging the outcomes of an implementation at two different facilities, even within the same health system or when using the same product, is problematic at best.   When many variables are involved, external validity becomes an issue because the interactions between variables is difficult to detect, understand, and control for.  Bottom line: Apples to apples comparisons are difficult under the best of circumstances as there are simply so many variables.

What is my take on all of this?  I see a bright future for clinical informatics because answering questions about the effects of EHRs will require adapting current study designs/methods and creating new ones.  In addition, new analytical tools are required to understand the mountains of data that will become available. Moreover, we need standards for determining EHR implementation success and measuring data accuracy.  The debates of EHR effects, despite their occasional rancor, are just a healthy sign that clinical informatics is maturing as a discipline.

Thomas Kuhn, in his landmark work, The Structure of Scientific Revolutions, offers an interesting example of how debates in an area of inquiry are a form of birth pangs.    Kuhn notes,

Being able to take no common body of belief for granted, each writer on physical optics felt forced to build his field anew from its foundations. In doing so, his choice of supporting observation and experiment was relatively free, for there was no standard set of methods or of phenomena that every optical writer felt forced to employ and explain. Under these circumstances the dialog of the resulting books was often directed as much to the members of other schools as it was to nature.

With the publication of Opticks, Isaac Newton provided the much-needed conceptual center that permitted all schools to compare and contrast their work; thereby, moving the entire field forward.

Ultimately, the wide-ranging debates over EHRs and their effects are just the beginning of a paradigm shift in clinical informatics and EHR research, and I am glad to be a part of it!

1.            Epp SS.  Discrete Mathematics with Applications: Third Edition. Belmont: Brooks/Cole; 2004.



Leave a Reply

Your email address will not be published. Required fields are marked *