Multisite assessment of emergency medicine resident knowledge of evidence-based medicine as measured by the Fresno Test of Evidence-Based Medicine

Context: Evidence-based medicine (EBM) is the application of scientific evidence while treating a patient. To date, however, there is very little evidence describing how residents in emergency medicine understand and incorporate EBM into practice. Objectives: The aim of this study was to determine EBM theoretical and quantitative knowledge in emergency medicine residents in community hospital-based training programs. Methods: A sample of emergency medicine residents from nine hospitals was enrolled to complete a cross-sectional assessment of EBM skills from April 2021 through June 2021. Performance on the Fresno Test of Evidence-Based Medicine (FTEBM) was assessed utilizing descriptive statistics, t tests, and one-way analysis of variance. Results: A total of 50.8% (124/244) of current emergency medicine residents completed the FTEBM during the study period. No significant difference on FTEBM scores was noted between the different types of medical degrees (DO vs. MD) (p=0.511), holding an advanced research degree (p=0.117), or between each postgraduate year of training (p=0.356). The mean score of those residents who rated their knowledge of EBM as average or higher was 36.0% (32.8 – 39.1%). The mean score of those residents who rated their programs as having an “ average ” or higher institutional focus on EBM was 34.9% (32.2 – 37.6%). Conclusions: Participating emergency medicine residents show an incomplete understanding of EBM both in theory and applied computations despite rating themselves as having an average understanding. Emergency medicine residencies would be well suited to implement a standardized EBM curriculum that focuses on longitudinal reinforcement of key concepts needed for the practicing physician.

Methods: A sample of emergency medicine residents from nine hospitals was enrolled to complete a cross-sectional assessment of EBM skills from April 2021 through June 2021. Performance on the Fresno Test of Evidence-Based Medicine (FTEBM) was assessed utilizing descriptive statistics, t tests, and one-way analysis of variance. Results: A total of 50.8% (124/244) of current emergency medicine residents completed the FTEBM during the study period. No significant difference on FTEBM scores was noted between the different types of medical degrees (DO vs. MD) (p=0.511), holding an advanced research degree (p=0.117), or between each postgraduate year of training (p=0.356). The mean score of those residents who rated their knowledge of EBM as average or higher was 36.0% (32. .1%). The mean score of those residents who rated their programs as having an "average" or higher institutional focus on EBM was 34.9% (32.2-37.6%). Conclusions: Participating emergency medicine residents show an incomplete understanding of EBM both in theory and applied computations despite rating themselves as having an average understanding. Emergency medicine residencies would be well suited to implement a standardized EBM curriculum that focuses on longitudinal reinforcement of key concepts needed for the practicing physician.
Keywords: emergency medicine; evidence based medicine; medical education.
Evidence-based medicine (EBM) is defined as the judicious use of the current evidence in making decisions about the healthcare of individual patients [1]. Research has shown that utilizing an evidence-based strategy to treat patients reduces medical errors, increases individualized patient care, and supports the application of best practices in the clinical setting [2,3]. However, many learners in undergraduate and graduate medical education lack the proficiency needed to adequately interpret and utilize EBM in treating patients despite it being a core competency according to many professional societies [4,5].
According to the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Emergency Medicine (ABEM), EBM is a key milestone for those in emergency medicine graduate medical education [6]. The practice-based performance improvement milestone notes that a resident should be competent in performance improvement to optimize emergency department function and self-learning at the time of graduation. Competency in EBM has been defined as the ability for a practitioner to develop structured clinical questions, identify the available literature to answer questions, critically appraise and apply the methodology and results of the literature to clinical practice, and evaluate the approach undertaken during a literature search [7]. Assessing competency in EBM can be completed with the use of the Fresno Test of Evidence-Based Medicine (FTEBM). The FTEBM is a validated assessment tool that evaluates a learner's ability to frame a research question, search for evidence in the literature, understand the hierarchy of evidence, and interpret magnitude and validity as well as basic statistical and methodological concepts [8].
It has been reported that medical students have rated their knowledge of evidence-based principles higher than objectively measured outcomes on a standardized examination [9]. Research has also shown that senior-level medical students demonstrate competence in only about half of all evidence-based medical topics that are needed in graduate medical education [10]. Without having a baseline knowledge from undergraduate medical education, those in graduate medical education may not reach the highest levels of competency based upon the ACGME milestones. The discrepancy between perceived and objectively measured competence in EBM in medical students has necessitated the systematic study of emergency medicine residents' knowledge of EBM. The objective of this study was to determine basic EBM theoretical and quantitative knowledge in a cross-sectional sample of emergency medicine residents.

Setting
The Foundation for Osteopathic Emergency Medicine (FOEM) research network is comprised of a group of ACGME emergency medicine residency training programs across the United States. A total of nine sites agreed to participate after initial contact and disseminated the enrollment survey and the FTEBM to their residents. The sites with the FOEM research network include Kingman Regional Medical Center, McLaren Macomb, Saint Barnabas Medical Center, Sunrise Medical Center, Trinity Health System, Conemaugh Memorial Medical Center, Ohio Health, Geisinger Medical Center, and Ascension Macomb.

Study design
All study procedures were approved by the Kingman Regional Medical Center Institutional Review Board and designated the approval number KHI-0204. Sixteen emergency medicine residencies in the United States were contacted in February 2021 to determine their interest in participating. Nine (56.3%, 9/16) residencies agreed to participate. Written informed consent was obtained from each participant prior to data collection. Participating institutions were provided with physical copies of the enrollment surveys, and the FTEBM were provided to each site's primary investigator following agreement to participate as a research site. Residencies enrolled PGY-1 through PGY-4 residents between April 1, 2021 and July 30, 2021. At the time of enrollment, residents were provided with an hour to complete the assessment before returning it to the site primary investigator.

Fresno test of evidence-based medicine (FTEBM)
The FTEBM is a questionnaire that consists of 12 items related to EBM principles that measure the understanding and practical application of research concepts such as forming a question, finding evidence to answer that question, internal and external validity, and computing basic clinical statistics. The highest total score that could be obtained is 128. Each section has the following maximum scores: Writing Clinical Questions=6, Information Sources=6, Search Strategies=8, Clinical Study Design=12, Literature Relevancy=12, Internal Val-idity=24, Magnitude and Significance=12, Diagnostic Accuracy=20, Risk Reduction=16, Statistical Significance=4, Diagnostic Study Design=4, and Prognostic Study Design=4. Performance by question type (theoretical or quantitative) was also assessed: theoretical included questions 1-7, 11, and 12, while quantitative included questions 8, 9, and 10. Upon return of the surveys and FTEBM to the primary investigator, a standardized Fresno grading system was utilized to assess the completed FTEBM by two trained evaluators. Each evaluator was trained by the primary investigator of the project, who also resolved any scoring questions that arose during grading.

Statistical analysis
Data were analyzed utilizing SPSS statistics version 27 (IBM Corp., Armonk, NY). The demographic results of the participating residents were assessed utilizing descriptive statistics. Total performance, performance on the theoretical questions, and performance on the quantitative questions were assessed based on postgraduate year, medical degree, sex, and additional educational degree attainment by the independent-samples t-test. As a baseline for improvement across residencies, we calculated the proportion of residents who performed above the 50% criterion on questions. Individual items on the FTEBM were then assessed for improvement over the years of residency training via an analysis of variance with polynomial contrasting. Statistical significance was set at a p≤0.05.

Results
Of the 124 submitted FTEBMs for grading, only 51.6% (64/ 124) had all 12 questions answered. A total of 48.4% (60/ 124) of FTEBMs was submitted with one or more question left unanswered. The most common question left unanswered was quantitative question 9 (risk reduction) (36.3% (45/124)) (  Table 2). A minority of the residents held an additional degree at the Master's level and above (29/124, 23.4%) ( Table 2). Residents most frequently reported an "average" ability (72/124, 58.1%) to assess the quality of medical evidence compared with their peers ( Table 2). The overall mean score in those residents who reported an "average" or "above average" baseline knowledge in EBM was 36.0% (46.1/128) (32.8-39.1%). No correlation was noted with the self-perceived knowledge in EBM and overall test scores on the FTEBM (T b= 0.021; p=0.768). The residency program's focus on EBM was most often rated as "average" (41/124, 33.1%) or "above average" (41/124, 33.1%) by the participating residents ( Table 2). The overall mean score on the FTEBM of residents rating their institutions as having an "average" or "above average" focus on EBM was 34.9% (44.7/128) (32.2-37.6%). No correlation was noted with the perceived institutional     Table 3). When assessed by question type (theoretical or quantitative), no significant differences based upon resident demographics were detected (Table 3).

Discussion
This study represents the largest independent assessment of emergency medicine residents' knowledge of evidencebased practice in the literature to date. A recent systematic review on the development of EBM curriculums for emergency medicine noted several studies assessing a resident's knowledge on topics commonly covered in EBM but that few utilized a validated assessment tool [11]. Conducted in 2010, Friedman et al. [12] sought to describe the impact of EBM on the change in patient-directed care by emergency medicine residents. Although the authors did not assess baseline knowledge, they did describe how following a search and review of the literature, the residents altered the course of care for 16.3% of patients. While the data presented in this manuscript suggest an improvement of literature search skills over the course of residency training, it appears that additional work could be completed by faculty to further train residents on how to interpret the papers that are sourced.
In a group of residents at a single training institution, Mohr et al. [13] reported a baseline score on the FTEBM of 82.3%. In comparison, this graduate medical education program represents an academic health center program that is usually assumed to be more replete in EBM didactics compared to a community hospital-based residency program such as ours. The highest score among residencies with  more than 10 participants in our study was 48.0%. Bentley et al. [14] estimate that scores start low in emergency medicine residents and then increase linearly from PGY-1 through PGY-4 with a starting point of 25% and endpoint of 77%. Based upon the current study and prior literature, it appears that emergency medicine residents have a limited understanding of the basic theoretical and statistical concepts that are needed to effectively understand and practice EBM. Although no overall differences in FTEBM scores were observed based on postgraduate years, more experienced residents did show an increase in knowledge on searching for clinical literature (PGY-3 outperforming PGY-1 residents by 12.3% points) and on study design (PGY-3 residents outscoring PGY-1 residents by 12.7% points). Unlike many of the items on the FTEBM, these items represent the initial phases of clinical evidence. It is likely that residents garner a more ad hoc search strategy as they learn to utilize available online resources over the course of their training. Although residents learn to better identify literature, the remainder of the items on the FTEBM do not show training-year performance-related increases.
Thus, while the residents may be able to locate literature, it is unlikely that they are able to evaluate the scope of the reported findings based upon statistical analysis.
Residency programs may be well served to increase the amount of time spent teaching both research-focused concepts and biostatistics in order to better prepare trainees to incorporate the principles of EBM into future practice. Although not addressed in the current study, when asked about attitudes toward research, residents indicate little perceived impact of EBM on their medical training [15]. These attitudes represent a concerning barrier toward future training.
Based upon these results and the current body of literature, the open question that remains is what educational activities can be implemented during residency to improve understanding of the core concepts of EBM. Although sparse in the literature, several interventions have attempted to improve knowledge of research appraisal. Validated instructional techniques for postgraduates include dedicated journal clubs [16], individual research methodology workshops [17], and full curricula associated with monthly instruction [18]. Unfortunately, only a minority of graduate medical education programs have an established EBM curriculum [19].

Limitations
Although this was a cross-sectional survey of a diverse group of emergency medicine residents from across the nation, it was not encompassing of all residents in training.
Overall study findings could be altered by including a larger sample size of residents and by including more MDs for a better representation of this population. Residents were not excluded from the study if they had previous knowledge or had previously taken the FTEBM as part of their education or during another research project. The EBM curriculum at each institution was also not included in the original data that was gathered. These two limitations could have altered the results because residents may have had a prior understanding of the concepts on the examination or may have taken the examination at a previous point in their academic training.

Conclusions
The surveyed emergency medicine residents show an incomplete understanding of the basic concepts in EBM as tested by the FTEBM. No difference in knowledge was seen based on the postgraduate level of training, on the different medical school degrees, or if a resident held an advanced degree in addition to that of medicine. A heightened focus on teaching the basic theoretical and statistical concepts of research is needed if residents are truly expected to not only meet the specific milestones prior to graduation but also effectively treat patients with an eye toward EBM.
Research funding: This project was supported by the Foundation for Osteopathic Emergency Medicine Research Network David A. Kuchinski Memorial Grant. Author contributions: All authors provided substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; all authors drafted the article or revised it critically for important intellectual contact; all authors gave final approval of the version of the article to be published; all authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.