The 2015 National Academy of Medicine’s report, Improving Diagnosis in Healthcare , called attention to diagnostic errors and their contribution to patient harm. Among the major themes highlighted in the report is the need to improve training in clinical reasoning among diagnosticians. Theoretically, diagnosticians who are aware of potential flaws in human decision-making might be better prepared to recognize and avoid pitfalls in their diagnostic reasoning, thereby reducing patient harm .
Diagnostic reasoning can be understood using the dual process theory of decision-making, a general model describing how humans make decisions . Dual process theory suggests two modes of reasoning – System 1 and System 2 – are used to make decisions. Deliberate, conscious, time-consuming and effortful judgments characterize System 2 reasoning. System 1 reasoning, in contrast, is primarily intuitive, subconscious, fast and relatively effortless . System 1 reasoning relies on heuristics (often called cognitive biases). These mental shortcuts allow experienced clinicians to arrive at decisions quickly with a reasonable degree of certainty in familiar situations. Using heuristics to make decisions is neither avoidable nor an inherently inappropriate method for clinical reasoning . While experienced clinicians may recognize their reliance on System 1 reasoning in clinical practice, the theories underlying decision-making are not formally taught in most medical school or residency curricula .
In clinical vignettes, cognitive errors contribute to overall diagnostic errors in as many as 37–77% of cases  leading some authors to suggest that understanding the etiology of these errors is paramount to reducing them . Others have argued that knowing about heuristics may not necessarily allow clinicians to avoid diagnostic errors . Consequently, some authors have suggested novel interventions (e.g. diagnostic checklists, differential diagnosis generators) to mitigate the risks posed by heuristic-driven reasoning , . However, the deliberate use of such tools requires awareness of situations at high risk for cognitive errors. It is not clear that diagnosticians currently possess this awareness, but some understanding of the role of heuristics in diagnostic reasoning may be helpful in reducing harm from diagnostic errors.
Discussion of diagnostic errors during morbidity and mortality (M&M) conferences represents an approach to reducing harm through structured analysis of the errors including identification of cognitive factors . Katz and Detsky suggest that M&M is an ideal venue to introduce discussion of cognitive errors . Fruitful discussions of diagnostic error require an environment that is open to honest discussion focused on patient safety rather than harmful criticism of individuals . Yet, discussions of harm arising from flawed diagnostic reasoning are uncomfortable for clinicians , . For example, loss of reputation and fear of incurring blame are two significant components that influence clinicians’ comfort discussing their own errors with colleagues . Recently, the promotion of both psychological safety (including a non-blaming atmosphere) and transparency when reporting and reviewing medical errors in public venues has received significant attention . While some institutions have deliberately revamped their M&M conferences to promote psychological safety , there remains an undercurrent of blame culture in medicine .
We aimed to better understand two obstacles to improving training in clinical reasoning for clinicians. First, little is known about how familiar clinicians are with heuristics operating during the diagnostic process. Second, it is not clear whether current culture promotes psychological safety when discussing diagnostic errors in public venues such as M&M. In anticipation of developing a curriculum on diagnostic errors at our institution, we wanted to better characterize these obstacles among a subset of pediatric faculty. We hypothesized that participants’ awareness of heuristic definitions would positively correlate with an ability to recognize heuristics operating in clinical vignettes. We also sought to describe participants’ comfort discussing their own diagnostic errors in public venues and the perceived barriers to such discussions.
Materials and methods
Study design and participants
An anonymous web-based survey was distributed by e-mail invitation to all faculty members of the Divisions of Critical Care, Emergency Medicine and Hospital Medicine within the Department of Pediatrics at our institution. These divisions were selected because they have mature tiered case review programs amenable to discussing cognitive errors alongside systems issues. Physicians and advanced-practice providers (APPs; i.e. nurse practitioners and physician assistants) were eligible. The survey was available from January 31 to March 31, 2017. E-mail invitations were distributed at the beginning of the enrollment period with e-mail reminders sent at 3 and 6 weeks after the survey opened. Full- and part-time clinical faculty were eligible to participate; we did not enroll trainees. The Local Institutional Review Board deemed the study exempt from review.
The survey consisted of three sections. The first two sections asked participants to select definitions of common heuristics (base rate neglect, outcome bias, search satisfying, anchoring, gambler’s fallacy) and to name heuristics operating in four general pediatric clinical vignettes (premature closure, posterior probability error, visceral bias, framing effect). For each definition and vignette, the correct answer, three foils, and an “I don’t know” option were provided. No correct answer in the definition questions was used as a correct answer for the vignettes. We used definitions for heuristics as described by Croskerry . We selected heuristics that commonly contribute to diagnostic errors discussed in M&M in our own clinical experience; data on the true frequency of diagnostic errors attributable to specific heuristics are lacking . To ensure that the vignettes accurately demonstrated the heuristic of interest, two investigators (JAG, SIZ) held two focus groups with representatives from each division prior to finalizing the survey instrument; focus group members did not participate in the survey.
The third section of the survey assessed participants’ comfort discussing diagnostic errors and diagnostic uncertainty with trusted colleagues in private and in public venues using a five-point Likert scale. Additionally, we asked participants to identify the three most significant barriers to discussing their own diagnostic errors in public.
We pilot tested the survey with general pediatric residents, chief residents and fellows in the specialties to whom the survey was ultimately distributed. The survey was administered using REDCap  and took less than an average of 10 min to complete.
Fisher’s exact and chi-squared (χ2) tests were used to compare survey completion by age, gender, years in practice and provider type. Descriptive statistics were used to evaluate demographic characteristics. Mean number of correct responses with standard deviation (SD) were calculated for the definition and clinical vignette items. “I don’t know” responses were categorized as incorrect. Tetrachoric and Spearman rank correlation coefficients were used to explore associations between the total number of correctly answered heuristic definitions relative to the total number of correctly identified heuristics in clinical vignettes. Fisher’s exact and χ2 tests were used to assess differences in the attitudinal items by years in practice (dichotomized to <6 years and ≥6 years), gender and provider type. Differences related to age were compared using the Kruskal-Wallis test. For items regarding comfort discussing diagnostic errors, we dichotomized the responses into “comfortable” (including ‘pretty’ and ‘very’ comfortable) and “uncomfortable” (‘not at all’, ‘a little’ and ‘somewhat’ comfortable) as efforts to improve comfort surrounding such discussions would likely target those who are less comfortable or at least ambivalent about such discussions.
There were 229 eligible faculty from the three divisions surveyed. Overall, 88 (38.4%) faculty members accessed the survey and provided demographic information. Eight surveys were excluded from all further analyses because participants only provided demographic information; an additional 10 were excluded because they were only partially complete. There were no significant differences with respect to age, gender, years in practice or provider type between those that provided complete information compared to those with truncated responses. The overall response rate was 30.6% . The response rate was significantly lower for APPs (9.0%) compared to physicians (44.3%) (p<0.001). Table 1 shows the demographics of participants. Only 15.9% endorsed receiving formal training in clinical reasoning beyond training received in medical school or, for APPs, during graduate training. All participants endorsing formal training were less than 50 years old. Most participants (84.3%) reported that they had been involved in a diagnostic error resulting in patient harm during their career.
Knowledge of heuristic definitions
No participant answered all five heuristic definition questions correctly; the mean number of correctly selected heuristic definitions was 1.60/5 (SD=1.13) (Figure 1). Reporting formal training in clinical reasoning did not significantly increase the likelihood of correctly selecting the heuristic term for any definition (p-values range from p=0.05 to 1.00). Age, gender, years in practice and provider type were not significantly associated with answer accuracy.
Identifying heuristics in vignettes
Only two participants selected the correct heuristic for all four vignettes; the mean number of correctly selected heuristics among the vignettes was 1.01/4 (SD=1.06) (Figure 2). Reporting formal training in clinical reasoning did not significantly increase the likelihood of correctly identifying the heuristic for any vignette (p=0.72–1.00). Age, gender, years in practice and provider type were not significantly associated with answer accuracy for any of the vignettes.
There was a low but significant correlation between scores on the definition and vignette portions of the survey (Spearman’s ρ=0.27, p=0.02).
Comfort surrounding discussion of diagnostic errors
The proportion of participants reporting being “pretty” or “very” comfortable discussing diagnostic errors was significantly lower for public compared to private discussions: 28.6% vs. 74.3%, respectively (p<0.01). The proportion of participants reporting being “pretty” or “very” comfortable acknowledging diagnostic uncertainty to themselves was not significantly different than acknowledging diagnostic uncertainty to colleagues: 82.9% vs. 75.7%, respectively (p=0.06). Age, gender, formal training and provider type were not significantly associated with comfort levels. Years in practice was significantly associated with comfort discussing errors in public and acknowledging uncertainty to colleagues (Table 2).
Participants identified the three most significant barriers to discussing their own diagnostic errors in public venues (Table 3). Over 50% of the participants indicated that the most significant barriers were their reputation as a provider being at stake (62.9%), and their knowledge-base (58.6%) and decision-making being judged (57.1%).
Pediatric faculty in this study demonstrated limited ability to identify heuristic definitions. Similarly, they were frequently unable to correctly name heuristics employed in clinical vignettes. We did find evidence to support our hypothesis that knowledge of heuristic definitions correlates with being able to identify heuristics in clinical vignettes, although the correlation is low. Pediatric faculty expressed significant discomfort discussing their own errors publicly in venues such as M&M despite being generally comfortable discussing diagnostic errors in private and acknowledging diagnostic uncertainty to themselves and their peers. This discomfort appears to arise from concerns about negative peer perceptions regarding clinical reputation and the diagnosticians’ clinical reasoning skills and knowledge.
Clinicians are central to efforts to reduce patient harm resulting from diagnostic errors. Success in these efforts may be dependent on two related conditions: (1) the diagnostician’s ability to recognize how diagnostic reasoning might be subconsciously influenced through reliance on heuristics; and (2) the willingness to discuss cognitive errors in public in order to identify solutions to avoid them in the future.
Pediatricians can appreciate their own diagnostic errors in hindsight. Singh and colleagues demonstrated that 54% of surveyed pediatricians admit to making 1–2 diagnostic errors per month, and 45% reported making 1–2 diagnostic errors per year that result in patient harm . However, reducing patient harm due to diagnostic error requires that diagnosticians recognize and avoid heuristic failures prior to committing them.
There are few studies investigating strategies to avoid, or mitigate the effects of, heuristics and results are mixed . Most studies have included trainees and have focused on clinical vignettes rather than real patient encounters. While some studies have demonstrated small but significant improvements in diagnostic accuracy when residents were instructed to rely on slow deliberate System 2 reasoning (through structured reflection) , , one study did not demonstrate a benefit with this approach . Sherbino and colleagues provided instruction on two common heuristics (search satisfying and availability) to senior medical students followed by training on cognitive forcing strategies to avoid heuristic errors. Despite immediate testing after the educational intervention, fewer than 50% of students successfully “debiased” themselves . Little work has been done to evaluate practicing clinicians’ familiarity with and ability to recognize heuristics. We conducted this study to begin to address that gap.
Participants in this study were unable to correctly name several defined heuristics or identify heuristics operating in several clinical vignettes. We found that formal training in diagnostic reasoning did not improve participants’ ability to identify heuristics. We did not collect information regarding the timing or content of this training so it is difficult to know whether more robust or more proximate education would have yielded improved accuracy. However, our results align with another study in which experts in diagnostic error were asked to identify heuristics (cognitive biases) perceived to be operating in eight clinical vignettes . Participants demonstrated no agreement when naming heuristics represented by the vignettes (κ=0.0–0.6).
The goal of introducing diagnosticians to concepts including heuristics and dual process theory is not to teach new vocabulary. Rather, improving diagnosticians’ awareness of these concepts aims to improve recognition of the risk of diagnostic errors arising from System 1 reasoning. Nemes-Walsh and Lee evaluated the effect of an educational intervention introducing topics including dual process theory and cognitive bias pertaining to diagnostic errors . This study demonstrated a slight improvement in responses on the National Patient Safety Foundation Reducing Diagnostic Errors: Strategies for Solutions Quiz and increased willingness to take a “diagnostic timeout” following the intervention. There was no indication, however, that fewer diagnostic errors occurred after the training. In a more rigorous study, residents completing a yearlong curriculum on cognitive biases and diagnostic errors demonstrated improvement in correctly naming heuristics . Additionally, they were frequently able to identify heuristics in standardized clinical vignettes and describe potential debiasing strategies. The modest yet significant association between correctly naming defined heuristics and identifying heuristics in clinical vignettes in the present study aligns with those findings. While mastery of vocabulary alone is unlikely to reduce diagnostic errors arising from use of heuristics, conceptual familiarity with this type of reasoning may improve recognition of high-risk clinical scenarios and warrants further study to determine whether these interventions reduce diagnostic errors in clinical encounters.
Diagnosticians in this study were generally comfortable speaking privately to peers about their diagnostic errors. That comfort did not persist when participants considered discussing diagnostic errors in public settings. This discomfort appears to stem from a general concern over how diagnosticians’ knowledge, decision-making and reputation are judged rather than the error simply being made public. Diagnosticians also reported a poorer self-perception of themselves as clinicians as a barrier. These findings suggest significant cultural changes are necessary to advance patient safety. We did not inquire about the fear of malpractice litigation which may also be a substantial barrier to openly discussing diagnostic errors.
We were encouraged by the finding that more experienced clinicians were significantly more comfortable with public discussions of their errors than their less experienced peers. These data imply that senior clinicians may serve as influential role models in the cultural shift required to share experiences publicly in efforts to reduce harm from diagnostic errors.
Interpretation of our study must be considered in light of some limitations. We experienced a low response rate from APPs and, as a pilot study, sampled a relatively narrow segment of our hospital’s pediatric faculty. Our findings may not be representative of faculty at other hospitals or providers in non-academic and outpatient settings. We studied a small number of heuristics. Croskerry lists over 30 heuristics encountered in clinical reasoning but data describing which heuristics are most common or significant in diagnostic reasoning are limited . Thus, we chose heuristics that our own clinical experience indicates are employed relatively frequently but acknowledge that they are not proven to be the most common culprits in diagnostic errors. We also acknowledge that our survey attempted to assess familiarity with heuristic concepts using definitions as a surrogate. It would be more useful to assess familiarity using more qualitative methods discussing vignettes with diagnosticians; this is an opportunity for future study.
Clinicians in our study were unable to match heuristic definitions or clinical vignettes exemplifying heuristics to the proper heuristic term. Attempts to foster a culture supportive of discussing diagnostic errors will require both improved awareness of the role heuristics play in the diagnostic process and better methods to promote psychological safety when sharing errors.
Balogh E, Miller BT, Ball J, Institute of Medicine (U.S.). Committee on Diagnostic Error in Health Care. Improving diagnosis in health care. Washington, DC: The National Academies Press, 2015. Google Scholar
Rencic J, Trowbridge RL, Jr, Fagan M, Szauter K, Durning S. Clinical reasoning education at US medical schools: results from a national survey of internal medicine clerkship directors. J Gen Intern Med 2017;32:1242–6. CrossrefPubMedWeb of ScienceGoogle Scholar
Graber ML, Sorensen AV, Biswas J, Modi V, Wackett A, Johnson S, et al. Developing checklists to prevent diagnostic error in Emergency Room settings. Diagnosis 2014;1:223–31. CrossrefPubMedGoogle Scholar
Riches N, Panagioti M, Alam R, Cheraghi-Sohi S, Campbell S, Esmail A, et al. The effectiveness of electronic differential diagnoses (DDX) generators: a systematic review and meta-analysis. PLoS One 2016;11:e0148991. Web of ScienceCrossrefPubMedGoogle Scholar
Cifra CL, Jones KL, Ascenzi JA, Bhalala US, Bembea MM, Newman-Toker DE, et al. Diagnostic errors in a PICU: insights from the morbidity and mortality conference. Pediatr Crit Care Med 2015;16:468–76. CrossrefWeb of ScienceGoogle Scholar
Woodward HI, Lemer C, Wu AW. An end to the witch hunts: responding to the defenders of blame and shame. A commentary on Collins, Block, Arnold and Christakis. Soc Sci Med 2009;69:1291–3. CrossrefWeb of SciencePubMedGoogle Scholar
Kaldjian LC, Forman-Hoffman VL, Jones EW, Wu BJ, Levi BH, Rosenthal GE. Do faculty and resident physicians discuss their medical errors? J Med Ethics 2008;34:717–22. PubMedWeb of ScienceCrossrefGoogle Scholar
Tad YD, Pierce RG, Pell JM, Stephan L, Kneeland PP, Wald HL. Leveraging a redesigned morbidity and mortality conference that incorporates the clinical and educational missions of improving quality and patient safety. Acad Med 2016;91:1239–43. CrossrefWeb of SciencePubMedGoogle Scholar
Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap) – a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Sci 2009;42:377–81. Google Scholar
American Association of Public Opinion Research. Standard definitions: final dispositions of case codes and outcome rates for surveys, 9th edition; 2016. Google Scholar
McDonald KM, Matesic B, Contopoulos-Ioannidis DG, Lonhart J, Schmidt E, Pineda N, et al. Patient safety strategies targeted at diagnostic errors: a systematic review. Ann Intern Med 2013;158:381–9. PubMedCrossrefWeb of ScienceGoogle Scholar
Monteiro SD, Sherbino J, Patel A, Mazzetti I, Norman GR, Howey E. Reflecting on diagnostic errors: taking a second look is not enough. J Gen Intern Med 2015;30:1270–4. CrossrefWeb of SciencePubMedGoogle Scholar
Mamede S, van Gog T, van den Berge K, Rikers RM, van Saase JL, van Guldener C, et al. Effect of availability bias and reflective reasoning on diagnostic accuracy among internal medicine residents. J Am Med Assoc 2010;304:1198–203. CrossrefWeb of ScienceGoogle Scholar
Norman G, Sherbino J, Dore K, Wood T, Young M, Gaissmaier W, et al. The etiology of diagnostic errors: a controlled trial of system 1 versus system 2 reasoning. Acad Med 2014;89:277–84. Web of ScienceCrossrefPubMedGoogle Scholar
Sherbino J, Yip S, Dore KL, Siu E, Norman GR. The effectiveness of cognitive forcing strategies to decrease diagnostic error: an exploratory study. Teach Learn Med 2011;23:78–84. CrossrefWeb of SciencePubMedGoogle Scholar
Zwaan L, Monteiro S, Sherbino J, Ilgen J, Howey B, Norman G. Is bias in the eye of the beholder? A vignette study to assess recognition of cognitive biases in clinical case workups. BMJ Qual Saf 2017;26:104–10. PubMedCrossrefWeb of ScienceGoogle Scholar
Nemes-Walsh JK, Lee AJ. Diagnostic errors: impact of an educational intervention on pediatric primary care. J Pediatr Healthcare 2017;32:10. Google Scholar
Reilly JB, Ogdie AR, Von Feldt JM, Myers JS. Teaching about how doctors think: a longitudinal curriculum in cognitive bias and diagnostic error for residents. BMJ Qual Saf 2013;22:1044–50. CrossrefWeb of SciencePubMedGoogle Scholar
About the article
Published Online: 2019-02-13
Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.
Research funding: None declared.
Employment or leadership: None declared.
Honorarium: None declared.
Competing interests: The funding organization(s) played no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the report for publication.