Jump to ContentJump to Main Navigation
Show Summary Details
More options …


Official Journal of the Society to Improve Diagnosis in Medicine (SIDM)

Editor-in-Chief: Graber, Mark L. / Plebani, Mario

Ed. by Argy, Nicolas / Epner, Paul L. / Lippi, Giuseppe / Singhal, Geeta / McDonald, Kathryn / Singh, Hardeep / Newman-Toker, David

Editorial Board: Basso , Daniela / Crock, Carmel / Croskerry, Pat / Dhaliwal, Gurpreet / Ely, John / Giannitsis, Evangelos / Katus, Hugo A. / Laposata, Michael / Lyratzopoulos, Yoryos / Maude, Jason / Sittig, Dean F. / Sonntag, Oswald / Zwaan, Laura

CiteScore 2018: 0.69

SCImago Journal Rank (SJR) 2018: 0.359
Source Normalized Impact per Paper (SNIP) 2018: 0.424

See all formats and pricing
More options …

Identifying error types in visual diagnostic skill assessment

Cécile J. Ravesloot / Anouk van der Gijp / Marieke F. van der Schaaf / Josephine C.B.M. Huige / Olle ten Cate
  • Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, The Netherlands
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
/ Koen L. Vincken / Christian P. Mol / Jan P.J. van Schaik
Published Online: 2017-06-05 | DOI: https://doi.org/10.1515/dx-2016-0033



Misinterpretation of medical images is an important source of diagnostic error. Errors can occur in different phases of the diagnostic process. Insight in the error types made by learners is crucial for training and giving effective feedback. Most diagnostic skill tests however penalize diagnostic mistakes without an eye for the diagnostic process and the type of error. A radiology test with stepwise reasoning questions was used to distinguish error types in the visual diagnostic process. We evaluated the additional value of a stepwise question-format, in comparison with only diagnostic questions in radiology tests.


Medical students in a radiology elective (n=109) took a radiology test including 11–13 cases in stepwise question-format: marking an abnormality, describing the abnormality and giving a diagnosis. Errors were coded by two independent researchers as perception, analysis, diagnosis, or undefined. Erroneous cases were further evaluated for the presence of latent errors or partial knowledge. Inter-rater reliabilities and percentages of cases with latent errors and partial knowledge were calculated.


The stepwise question-format procedure applied to 1351 cases completed by 109 medical students revealed 828 errors. Mean inter-rater reliability of error type coding was Cohen’s κ=0.79. Six hundred and fifty errors (79%) could be coded as perception, analysis or diagnosis errors. The stepwise question-format revealed latent errors in 9% and partial knowledge in 18% of cases.


A stepwise question-format can reliably distinguish error types in the visual diagnostic process, and reveals latent errors and partial knowledge.

Keywords: assessment; diagnostic errors; image analysis; image interpretation; perception; radiology education; visual diagnosis; visual expertise


  • 1.

    Bhargavan M, Sunshine JH. Workload of radiologists in the United States in 2002–2003 and trends since 1991–1992. Radiology 2005;236:920–31.CrossrefPubMedGoogle Scholar

  • 2.

    Bhargavan M, Kaye AH, Forman HP, Sunshine JH. Workload of radiologists in United States in 2006–2007 and trends since 1991–1992. Radiology 2009;252:458–67.Web of SciencePubMedCrossrefGoogle Scholar

  • 3.

    van der Gijp A, van der Schaaf MF, van der Schaaf IC, Huige JC, Ravesloot CJ, van Schaik JP, et al. Interpretation of radiological images: towards a framework of knowledge and skills. Adv Health Sci Educ Theory Pract 2014;19:565–80.PubMedCrossrefGoogle Scholar

  • 4.

    Nodine CF, Kundel HL, Mello-Thoms C, Weinstein SP, Orel SG, Sullivan DC, et al. How experience and training influence mammography expertise. Acad Radiol 1999;6:575–85.CrossrefPubMedGoogle Scholar

  • 5.

    Norman GR, Coblentz CL, Brooks LR, Babcook CJ. Expertise in visual diagnosis: a review of the literature. Acad Med 1992;67:S78–83.CrossrefPubMedGoogle Scholar

  • 6.

    Taylor PM. A review of research into the development of radiologic expertise: implications for computer-based training. Acad Radiol 2007;14:1252–63.PubMedCrossrefWeb of ScienceGoogle Scholar

  • 7.

    Gruen RL, Jurkovich GJ, McIntyre LK, Foy HM, Maier RV. Patterns of errors contributing to trauma mortality: lessons learned from 2594 deaths. Ann Surg 2006;244:371–80.PubMedGoogle Scholar

  • 8.

    Wechsler RJ, Spettell CM, Kurtz AB, Lev-Toaff AS, Halpern EJ, Nazarian LN, et al. Effects of training and experience in interpretation of emergency body CT scans. Radiology 1996;199:717–20.PubMedCrossrefGoogle Scholar

  • 9.

    Guly HR. Diagnostic errors in an accident and emergency department. Emerg Med J 2001;18:263–9.CrossrefGoogle Scholar

  • 10.

    Rhea JT, Potsaid MS, DeLuca SA. Errors of interpretation as elicited by a quality audit of an emergency radiology facility. Radiology 1979;132:277–80.PubMedCrossrefGoogle Scholar

  • 11.

    Gwynne A, Barber P, Tavener F. A review of 105 negligence claims against accident and emergency departments. J Accid Emerg Med 1997;14:243–5.PubMedCrossrefGoogle Scholar

  • 12.

    Hu CH, Kundel HL, Nodine CF, Krupinski EA, Toto LC. Searching for bone fractures: a comparison with pulmonary nodule search. Acad Radiol 1994;1:25–32.CrossrefPubMedGoogle Scholar

  • 13.

    Krupinski EA. Visual scanning patterns of radiologists searching mammograms. Acad Radiol 1996;3:137–44.PubMedCrossrefGoogle Scholar

  • 14.

    Drew T, Le-Hoa Vo M, Olwal A, Jacobson F, Seltzer SE, Wolfe JM. Scanners and drillers: characterizing expert visual search through volumetric images. J Vis 2013;13:pii: 3.CrossrefGoogle Scholar

  • 15.

    Renfrew DL, Franken Jr EA, Berbaum KS, Weigelt FH, Abu-Yousef MM. Error in radiology: classification and lessons in 182 cases presented at a problem case conference. Radiology 1992;183:145–50.CrossrefGoogle Scholar

  • 16.

    Pinto A, Acampora C, Pinto F, Kourdioukova E, Romano L, Verstraete K. Learning from diagnostic errors: a good way to improve education in radiology. Eur J Radiol 2011;78:372–6.CrossrefPubMedWeb of ScienceGoogle Scholar

  • 17.

    Donald JJ, Barnard SA. Common patterns in 558 diagnostic radiology errors. J Med Imaging Radiat Oncol 2012;56:173–8.PubMedCrossrefGoogle Scholar

  • 18.

    Donovan T, Litchfield D. Looking for cancer: expertise related differences in searching and decision making. Appl Cognit Psychol 2013;27:43–9.CrossrefGoogle Scholar

  • 19.

    Rubin GD, Roos JE, Tall M, Harrawood B, Bag S, Ly DL, et al. Characterizing search, recognition, and decision in the detection of lung nodules on CT scans: elucidation with eye tracking. Radiology 2015;274:276–86.CrossrefPubMedWeb of ScienceGoogle Scholar

  • 20.

    Kundel HL, Nodine CF, Carmody D. Visual scanning, pattern recognition and decision-making in pulmonary nodule detection. Investigat Radiol 1978;13:175–81.CrossrefGoogle Scholar

  • 21.

    Pecaric M, Boutis K, Beckstead J, Pusic M. A big data and learning analytics approach to process-level feedback in cognitive simulations. Acad Med 2017;92:175–84.CrossrefWeb of SciencePubMedGoogle Scholar

  • 22.

    Ravesloot CJ, van der Schaaf MF, van Schaik JP, ten Cate OT, van der Gijp A, Mol CP, et al. Volumetric CT-images improve testing of radiological image interpretation skills. Eur J Radiol 2015;84:856–61.Web of ScienceCrossrefPubMedGoogle Scholar

  • 23.

    Ravesloot CJ, van der Gijp A, van der Schaaf MF, Huige JC, Vincken KL, Mol CP, et al. Support for external validity of radiological anatomy tests using volumetric images. Acad Radiol 2015;22:640–5.Web of SciencePubMedCrossrefGoogle Scholar

  • 24.

    Morita J, Miwa K, Kitasaka T, Mori K, Suenaga Y, Iwano S, et al. Interactions of perceptual and conceptual processing: expertise in medical image diagnosis. Int J Hum Comp Stud 2008;66:370–90.CrossrefGoogle Scholar

  • 25.

    Sadler DR. Beyond feedback: developing student capability in complex appraisal. Assess Eval High Educ 2010;35:535–50.CrossrefGoogle Scholar

  • 26.

    Black P, William D. Assessment and classroom learning. Assess Educ Princ Pol Pract 1998;5:7–74.Google Scholar

About the article

Corresponding author: Cécile J. Ravesloot, MD, Radiology Department, University Medical Center Utrecht, E01.132, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands, Phone: +31887556689, Fax: +31302581098

aCécile J. Ravesloot and Anouk van der Gijp are joint first authors.

Received: 2016-09-01

Accepted: 2017-04-26

Published Online: 2017-06-05

Published in Print: 2017-06-27

Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.

Research funding: SURF Foundation (Grant Number: TTL 11.0269).

Employment or leadership: None declared.

Honorarium: None declared.

Competing interests: The funding organization(s) played no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the report for publication.

Citation Information: Diagnosis, Volume 4, Issue 2, Pages 93–99, ISSN (Online) 2194-802X, ISSN (Print) 2194-8011, DOI: https://doi.org/10.1515/dx-2016-0033.

Export Citation

©2017 Walter de Gruyter GmbH, Berlin/Boston.Get Permission

Comments (0)

Please log in or register to comment.
Log in