Direct observation of depression screening: identifying diagnostic error and improving accuracy through unannounced standardized patients

Alan SchwartzORCID iD:, Steven Peskin, Alan Spiro and Saul J. Weiner



Depression is substantially underdiagnosed in primary care, despite recommendations for screening at every visit. We report a secondary analysis focused on depression of a recently completed study using unannounced standardized patients (USPs) to measure and improve provider behaviors, documentation, and subsequent claims for real patients.


Unannounced standardized patients presented incognito in 217 visits to 59 primary care providers in 22 New Jersey practices. We collected USP checklists, visit audio recordings, and provider notes after visits; provided feedback to practices and providers based on the first two visits per provider; and compared care and documentation behaviors in the visits before and after feedback. We obtained real patient claims from the study practices and a matched comparison group and compared the likelihood of visits including International Classification of Diseases, 10th Revision (ICD-10) codes for depression before and after feedback between the study and comparison groups.


Providers significantly improved in their rate of depression screening following feedback [adjusted odds ratio (AOR), 3.41; 95% confidence interval (CI), 1.52–7.65; p = 0.003]. Sometimes expected behaviors were documented when not performed. The proportion of claims by actual patients with depression-related ICD-10 codes increased significantly more from prefeedback to postfeedback in the study group than in matched control group (interaction AOR, 1.41; 95% CI, 1.32–1.50; p < 0.001).


Using USPs, we found significant performance issues in diagnosis of depression, as well as discrepancies in documentation that may reduce future diagnostic accuracy. Providing feedback based on a small number of USP encounters led to some improvements in clinical performance observed both directly and indirectly via claims.

  • 1.

    Weiner SJ, Schwartz A. Directly observed care: can unannounced standardized patients address a gap in performance measurement? J Gen Intern Med 2014;29:1183–7.

  • 2.

    Stange KC, Zyzanski SJ, Jaen CR, Callahan EJ, Kelly RB, GillandersWR, et al. Illuminating the ‘black box’. A description of 4454 patient visits to 138 family physicians. J Fam Pract 1998;46:377–89.

  • 3.

    National Center for Quality Assurance. Available from:

  • 4.

    Agency for Healthcare Research and Quality. CAHPS Clinician & Group Surveys. Available at: Last accessed May 10, 2014.

  • 5.

    Weiner S, Schwartz A. Contextual errors in medical decision making: overlooked and understudied. Acad Med 2016;91:657–62.

  • 6.

    Weiner S, Schwartz A, Weaver F, Goldberg J, Yudkowski R, Sharma G, et al. Contextual errors and failures in indivdualizing patient care. Ann Int Med 2010;153:69–75.

  • 7.

    Schwartz A, Weiner S, Binns-Calvey A. Unannounced standardize patient assessment of the roter interaction analysis system: the challenge of measuring patient-centered communication. Jt Comm J Qual Patient Saf 2013;39:83–8.

  • 8.

    Schwartz A, Weiner S, Binns-Calvey A. Comparing announced with unannounced standardized patients in performance assessment. Jt Comm J Qual Patient Saf 2013;39:83–8.

  • 9.

    Schwartz A, Weiner S, Weaver F, Yudkowsky R, Sharma G, Binns-Calvey A, et al. Uncharted territory: measuring costs of diagnostic errors outside the medical record. BMJ Qual Saf 2012;21:918–24.

  • 10.

    Luck J, Peabody JW. Using standardised patients to measure physicians’ practice: validation study using audio recordings. Br Med J 2002;325:679.

  • 11.

    Peabody JW, Luck J, Glassman P, Dresselhaus TR, Lee M. Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality. J Am Med Assoc 2000;283:1715–22.

  • 12.

    Krane NK, Anderson D, Lazarus CJ, Termini M, Bowdish B, Chauvin S, et al. Physician practice behavior and practice guidelines: using unanounced standardized patients to gather data. J Gen Int Med 2009;24:53–6.

  • 13.

    Peabody J, Luck J, Glassman P, Jain S, Hansen J, Spell M, et al. Measuring the quality of physician practice by using clinical vignettes: a prospective validation study. Ann Intern Med 2004;141:771–80.

  • 14.

    Culver JO, Bowen DJ, Reynolds SE, Pinsky LE, Press N, Burke W. Breast cancer risk communication: assessment of primary care physicians by standardized patients. Genet Med 2009;11: 735–41.

  • 15.

    Srinivasan M, Franks P, Meredith LS, Fiscella K, Epstein RM, Kravitz RL. Connoisseurs of care? Unannounced standardized patients’ ratings of physicians. Med Care 2006;44:1092–8.

  • 16.

    Fiscella K, Franks P, Srinivasan M, Kravitz RL, Epstein R. Ratings of physician communication by real and standardized patients. Ann Fam Med 2007;5:151–8.

  • 17.

    Glassman P, Luck J, O’Gara E, Peabody J. Using standardized patients to measure quality: evidence from the literature and a prospective study. Jt Comm J Qual Patient Saf 2000;26:644–53.

  • 18.

    Luoma JB, Martin CE, Pearson JL. Contact with mental health and primary care providers before suicide: a review of the evidence. Am J Psychiatry 2002;159:909–16.

  • 19.

    Schwartz A, Weiner SJ. Collecting and using hidden quality data to enhance value-based care. Horizon Blue Cross Blue Shield of New Jersey’s 6th Annual Value-Based Program Summit. Iselin, NJ, 2018. Available at: Accessed 3/1/2020.

Purchase article
Get instant unlimited access to the article.
Price including VAT
Log in
Already have access? Please log in.

Journal + Issues

Diagnosis aims at answering the question how diagnosis determines the quality of medical care. It focuses on how diagnosis can be advanced, how it is taught, and how and why it can fail, leading to diagnostic errors. The journal welcomes both fundamental and applied works, improvement initiatives, opinions, and debates to encourage new thinking on improving this critical aspect of healthcare quality.