An analysis of publication trends of orthopedic surgery residency graduates in relation to academic achievement

Context: Traditionally, the Accreditation Council for Graduate Medical Education (ACGME) requires residency programs to implement research and other scholarly activities into their training curriculum. Encouraging residents to publish during residency is believed to promote research throughout their careers; however, no study has attempted to quantify research productivity among orthopedic surgery residents before, during, and after residency. Objectives: To determine whether publishing in peerreviewed journals during orthopedic residencies was an indicator of continued academic achievement after graduation. Methods: This study was observational in nature and employed a cross-sectional design. We examined whether research outcomes during orthopedic residency was associated with academic advancement or continued research involvement after residency. We identified 201 orthopedic residency programs on the Doximity website and randomly selected 50 to include in our sample. Of these programs, graduate rosters for 31 programs were located and subsequently included. Of the 341 graduates identified, we recorded the number of peer-reviewed publications, H-indices, fellowships, and whether the graduate pursued a career in private practice or academia. Results: Orthopedic residency graduates from 31 programspublished a total of 1923 peer-reviewedmanuscripts. On average, residents had a total of 5.6 publications and an h-index of 3.2. Residents entering academia and pursuing fellowships had a significantly higher total number of publications, higher number of first-author publications, and greater H-indices compared to those who did not enter academia or pursue a fellowship. Conclusions: Increased research productivity was associated with continued academic pursuits and an increased likelihood of pursuing fellowship training after residency. Research experiences, such as written abstracts, conference presentations, and peer-reviewed publications, are considered key benchmarks of a medical student’s residency application [1, 2]. Residency program directors may utilize objectivemeasures to compare oneapplicant to thenext, and previous literature has shown that program directors view research as an important component of one’s application and may increase the likelihood of successful matching into competitive programs [3]. For example, a study of 134 surveys returned by individuals at general surgery programs, of which 120 were program directors, found that 89.5% of respondents considered research involvement nearly all the time when making acceptance decisions [4]. In the field of orthopedic surgery, applicants with prior research experience had an increased likelihood of successfully matching into a residency program compared to applicants with limited research experiences [5]. Value may be placed on different types of research experiences (e.g., abstracts, conference, or poster presentations); however, applicants with research experiences in the form of peer-reviewed *Corresponding author: Marvin Carr, BS, OMS III, Office of Medical Student Research, Oklahoma State University Center for Health Sciences, 1111 W 17th St, Tulsa, OK 74107, USA, E-mail: marvin.carr@okstate.edu. https://orcid.org/0000-00019370-5883 J. Michael Anderson, BS, OMS IV, Samuel Shepard, BS, OMS III, James Hobbs, BS, OMS III, Corbin Walters, BS, OMS IV and Austin L. Johnson, BS, OMS IV, Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK, USA. https://orcid.org/0000-0002-1903-5735 (J. Anderson). https://orcid.org/0000-0003-4447-354X (S. Shepard). https://orcid.org/0000-0002-8600-6096 (C. Walters) Matt Vassar, PhD, Department of Psychiatry and Behavioral Sciences, OklahomaState University Center for Health Sciences, Tulsa, OK, USA. https://orcid.org/0000-0003-2859-6152 J Osteopath Med 2022; 122(4): 195–202 Open Access. © 2022 Marvin Carr et al., published by De Gruyter. This work is licensed under the Creative Commons Attribution 4.0 International License. publications may be more highly sought after because this productivity may lead to continued research output during residency training [6]. Since its conception in 1994, the Accreditation Council for Graduate Medical Education (ACGME) [7] has placed research requirements on both residency programs and their residents. These requirements mandate that residency programs teach residents “basic principles of scientific inquiry, including how research is designed, conducted, evaluated, explained to patients, and applied to patient care,” and residents must then engage in scholarly activities as part of their training [8]. Despite having these scholarly requirements, a previous study demonstrated that residency programs often fall short of meeting such requirements [9]. For example, a survey evaluating 217 program directors of internal medicine programs determined that only two programs met the requirements of scholarly activity, while others failed to demonstrate scholarly requirements, inherently becoming a common cause for citation from the ACGME [9]. In the realm of orthopedic surgery, a common shortcoming and perceived area for improvement among review committees includes “insufficient resident and/or faculty participation in research and scholarly activity in the areas of experimental design, hypothesis testing, and other research methods [10, 11].” Although efforts have been made to determine the level of adherence to ACGME research requirements in other medical specialties, to our knowledge, no study has correlated research productivity of orthopedics residency graduates with continued academic success or analyzed whether those with higher research output elected to pursue a fellowship upon completion of residency training. Additionally, with the United States Medical Licensing Exam (USMLE) Step 1 and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) board examination transitioning to a pass/fail grading system in 2022 [12], research productivity may become even more influential on the residency match process, as well as serve as an objective measure for program directors to utilize as a means of assessing an applicant’s future academic achievements. Therefore, we present our findings of current research practices and publication trends of orthopedic surgery residency graduates to determine (1) if research efforts made during medical training (student, resident during residency and postgraduate, fellowship and postgraduate, academia, and private practice) predicted future academic achievements; and (2) if higher research output during residency correlated with the pursuit of fellowship training or a career in academia. Methods Primary and secondary objectives Our study was cross-sectional in design and adhered to an a priori protocol [13]. Our primary objective was to determine whether a relationship exists between research output—in the form of accepted peerreviewed journal publications—during medical training (before, during, and after residency training), with further academic achievement and continued research output postresidency training. Secondarily, we aimed to assess the relationship between research productivity and continued training (e.g., fellowship training in a specialized area of orthopedic surgery) as well as pursuit of a career in academia. Additionally, we assessed whether secondary study characteristics of fellowship, gender, and authorship position were associated with continued academic success. All data in our study consisted of publicly available records. Thus, our studywas not subject to Institutional Review Board supervision per the US Department of Health and Human Services’ Code of Federal Regulations [14]. Training department and residency graduate selection We filtered residency programs A–Z utilizing the reputation ranking from the Doximity navigator of 2019–2020 (https://residency. doximity.com/) and exported the data to a Microsoft Excel spreadsheet. We utilized Microsoft Excel’s random number generator to randomly select 50 orthopedic surgery programs from the Doximity list of accredited orthopedic surgery programs. Google was utilized to collect public records of residency graduation placement, graduate rosters, fellowship match records, and other pertinent alumni information. All data collected per resident were during the years 2013, 2014, and 2015. This selection process was based on that utilized by Yang et al. [15].

selected 50 to include in our sample. Of these programs, graduate rosters for 31 programs were located and subsequently included. Of the 341 graduates identified, we recorded the number of peer-reviewed publications, H-indices, fellowships, and whether the graduate pursued a career in private practice or academia. Results: Orthopedic residency graduates from 31 programs published a total of 1923 peer-reviewed manuscripts. On average, residents had a total of 5.6 publications and an h-index of 3.2. Residents entering academia and pursuing fellowships had a significantly higher total number of publications, higher number of first-author publications, and greater H-indices compared to those who did not enter academia or pursue a fellowship. Conclusions: Increased research productivity was associated with continued academic pursuits and an increased likelihood of pursuing fellowship training after residency.
Research experiences, such as written abstracts, conference presentations, and peer-reviewed publications, are considered key benchmarks of a medical student's residency application [1,2]. Residency program directors may utilize objective measures to compare one applicant to the next, and previous literature has shown that program directors view research as an important component of one's application and may increase the likelihood of successful matching into competitive programs [3]. For example, a study of 134 surveys returned by individuals at general surgery programs, of which 120 were program directors, found that 89.5% of respondents considered research involvement nearly all the time when making acceptance decisions [4]. In the field of orthopedic surgery, applicants with prior research experience had an increased likelihood of successfully matching into a residency program compared to applicants with limited research experiences [5]. Value may be placed on different types of research experiences (e.g., abstracts, conference, or poster presentations); however, applicants with research experiences in the form of peer-reviewed publications may be more highly sought after because this productivity may lead to continued research output during residency training [6].
Since its conception in 1994, the Accreditation Council for Graduate Medical Education (ACGME) [7] has placed research requirements on both residency programs and their residents. These requirements mandate that residency programs teach residents "basic principles of scientific inquiry, including how research is designed, conducted, evaluated, explained to patients, and applied to patient care," and residents must then engage in scholarly activities as part of their training [8]. Despite having these scholarly requirements, a previous study demonstrated that residency programs often fall short of meeting such requirements [9]. For example, a survey evaluating 217 program directors of internal medicine programs determined that only two programs met the requirements of scholarly activity, while others failed to demonstrate scholarly requirements, inherently becoming a common cause for citation from the ACGME [9]. In the realm of orthopedic surgery, a common shortcoming and perceived area for improvement among review committees includes "insufficient resident and/or faculty participation in research and scholarly activity in the areas of experimental design, hypothesis testing, and other research methods [10,11]." Although efforts have been made to determine the level of adherence to ACGME research requirements in other medical specialties, to our knowledge, no study has correlated research productivity of orthopedics residency graduates with continued academic success or analyzed whether those with higher research output elected to pursue a fellowship upon completion of residency training. Additionally, with the United States Medical Licensing Exam (USMLE) Step 1 and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) board examination transitioning to a pass/fail grading system in 2022 [12], research productivity may become even more influential on the residency match process, as well as serve as an objective measure for program directors to utilize as a means of assessing an applicant's future academic achievements. Therefore, we present our findings of current research practices and publication trends of orthopedic surgery residency graduates to determine (1) if research efforts made during medical training (student, resident during residency and postgraduate, fellowship and postgraduate, academia, and private practice) predicted future academic achievements; and (2) if higher research output during residency correlated with the pursuit of fellowship training or a career in academia.

Primary and secondary objectives
Our study was cross-sectional in design and adhered to an a priori protocol [13]. Our primary objective was to determine whether a relationship exists between research output-in the form of accepted peerreviewed journal publications-during medical training (before, during, and after residency training), with further academic achievement and continued research output postresidency training. Secondarily, we aimed to assess the relationship between research productivity and continued training (e.g., fellowship training in a specialized area of orthopedic surgery) as well as pursuit of a career in academia. Additionally, we assessed whether secondary study characteristics of fellowship, gender, and authorship position were associated with continued academic success. All data in our study consisted of publicly available records. Thus, our study was not subject to Institutional Review Board supervision per the US Department of Health and Human Services' Code of Federal Regulations [14].

Training department and residency graduate selection
We filtered residency programs A-Z utilizing the reputation ranking from the Doximity navigator of 2019-2020 (https://residency. doximity.com/) and exported the data to a Microsoft Excel spreadsheet. We utilized Microsoft Excel's random number generator to randomly select 50 orthopedic surgery programs from the Doximity list of accredited orthopedic surgery programs. Google was utilized to collect public records of residency graduation placement, graduate rosters, fellowship match records, and other pertinent alumni information. All data collected per resident were during the years 2013, 2014, and 2015. This selection process was based on that utilized by Yang et al. [15].

Eligibility
To ensure that the correct graduate was included, we utilized the advanced author filter search option on Scopus. Each author was searched utilizing multiple search terms, including full name, institution, residency program, fellowship program, and field of interest (e.g., sports medicine, hand, trauma). Names were then searched via Google Scholar and Scopus by identical terms to compare publication history to assure a precise record. Of note, it is possible that the surnames of residents may have changed over the course of their training. Therefore, it is possible that we may have missed one or more publications during our search; however, the use of multiple search terms provided a robust, stepwise process that exhausted all efforts to accurately identify potential publications for each resident included in our sample.
Training M.C. and J.H. completed in-person training prior to data extraction to discuss, in detail, the study design, protocol, extraction form, and key data points to extract. A Google Forms extraction sheet was developed and pilot-tested by M.C. and J.H. during training. To pilot test the extraction form, M.C. and J.H. independently extracted data for 10 randomly selected graduates within our sample, and then the results were compared and the disagreements were reconciled.

Data analysis
Extraction data were sorted to represent the three following cohorts of time: preresidency (undergraduate and medical school education); during residency (5 years in length in the United States); and postresidency (including fellowship training if appreciated). Each period was extended by 6 months to capture publications that were likely completed within the preresidency period or during the residency period but may not have been published until afterward (the residency period consists of research published during residency training and the 6 months after residency training). The results were presented as percentages and frequencies. To examine the relationship between publication periods (before, during, and after residency), a Pearson product-moment correlation coefficient was utilized. An independent sample test was utilized to compare the mean publications of those who pursued a career in academic medicine or entered private practice, and publication outputs between those who pursued fellowship training and those who did not. Binary logistic regression was utilized to evaluate the relationship between career type (academic or private practice), first-author publications before residency, and fellowship while controlling for gender. Linear regression was utilized to evaluate the relationship between academic success (quantified by the h-index) and the number of first-author publications prior to residency training, also controlling for gender. Analyses were performed utilizing STATA 15.1 (StataCorp, LLC, College Station, TX).

Results
Of the 50 randomly selected orthopedic residency programs, 31 (62.0%) were included in our analysis, and 19 were excluded after a list of residency graduates were not obtained ( Figure 1). Our sample included 341 orthopedic residency graduates, the majority of which were males (295, 86.5%) who graduated from an allopathic medical school (330, 96.8%) and of which went into private practice (264, 77.4%). Additional graduate characteristics can be found in Table 1. Altogether, these 341 graduates published a total of 1923 publications during the selected time interval, with a mean of 5.6 (SD, 7.2) publications per graduate. Graduates had a mean of 0.6 (SD, 2.1) publications before residency, 1.9 (SD, 2.8) during residency and 3.1 (SD, 5.5) after residency. Graduates who completed residency training in 2013 produced a total of 557 (mean, 5.4; SD, 6.6) publications, graduates from 2014 produced 700 (mean, 6.3; SD, 8.1) publications, and graduates from 2015 produced 666 (mean, 5.3; SD, 6.7) publications. The most common type of publications were observational studies (863/1923, 44.9%), and the most common journal to publish in was the Journal of Bone and Joint Surgery (n=120; Table 2). Graduates had a mean h-index of 3.2 (SD, 4.0).
The Pearson correlation demonstrated a weak correlation between the number of publications obtained before and during residency (r=0.24) and between publications before and total publications after residency (r=0.21). Publishing research before residency was moderately correlated with an increase in h-index (r=0.45). Residents who pursued a fellowship had more publications (t 341 =−3.0, p=0.003) and more first-author publications (t 341 =−3.5, p=0.0128) than residents who did not pursue fellowship training. Residents with a higher h-index were more likely to have pursued a fellowship than those that did not participate in a fellowship (t 341 =−3.0, p=0.003). A post-hoc analysis was performed utilizing a Pearson chi-squared test, which showed a significant relationship between gender and first-author publication after residency (X2=5.7, p < 0.05); 60% of women graduates were the first author on publications after residency compared to 42.0% of their male counterparts. Males and females entered into academia (66/295, 22.4%; 11/46, 23.9%, respectively), as well as fellowships (274/295, 92.9%; 43/46, 93.5%, exclusively), at similar rates. Residents that had a higher total number of first-author publications were more likely to enter academic medicine than those with lower total publications (t 341 =−6.1, p < 0.001) ( Table 3).
Our final logistic regression model examined the relationship between first-author publications before residency, fellowship training, and gender and whether the graduate went into academic medicine. No significant correlations were found for graduates who published one or more articles before residency (adjusted odds ratio [aOR], 2.1; 95% CI, 0.6-7.4). No significant correlations were observed between graduates pursuing fellowships Table : Characteristics of , , and , orthopedic residency graduates (n = ).

Discussion
Results from our study suggest that the number of peerreviewed publications produced by orthopedic surgery residents was associated with a higher likelihood of pursuing fellowship training and a higher likelihood to enter into academia. This finding is similar to studies of other surgical specialties [16,17]. DeLong et al. [16] evaluated whether 624 plastic surgery residents who matched into fellowship training had an increased number of publications than residents who did not. Their analyses showed that research publications before and during residency correlated with an increase in both academic productivity and matching into fellowships [16]. Similarly, in the field of urology, 49 urology residents from the Mayo Clinic correlated a significant increase of those pursuing academic medicine with one or more publications prior to residency [17]. These findings among plastic surgery and urology residents coincide with our current analysis in orthopedic residents.
In the present study, we showed a mean of 5.6 publications per resident with a continuing increase from preresidency, during residency, and postresidency. Our results are consistent with previous studies, emphasizing the importance of research and matching into fellowships, specifically accepted publications in journals with high impact factors [18,19].  [18]. Another study of 130 orthopedic surgery residents found a mean number of total publications greater for those choosing a career in academia (4.8) than those who chose a nonacademic career (2.4) [19]. Taken together, these findings, when combined with ours, suggest that increased research productivity may be a necessary component for matching into fellowships and pursuit of a career in academic medicine.
We also compared several measures of research production by gender. In our study, females and males had a nearly equal number of publications, h-indices, and number of first-author publications. A rather interesting finding from our study was a significant relationship between gender and first-author publications postresidency. We found that 60.0% of the females in our sample were first authors after residency compared to only 42.0% of the *represents p Value < ., SD, standard deviation. Table : Binary logistic regression was utilized to analyze the relationship between career type (academic or private practice) and first-author publications before residency and fellowship, controlling for gender (n = ).

Characteristic
No. (%) uOR (% CI) aOR (% CI) males in our sample. A nonsignificant but important finding from our study is that male and female resident graduates entered into academics at nearly equal rates, 22.4% and 23.9% respectively. Ence et al. [20] found that, in orthopedic surgery, gender was not independently associated with an increased probability of senior academic rank. These authors also reported no statistically significant difference in m-indices (h-index divided by academic career duration in years) between men and women overall [20]. A recently published review comparing gender of orthopedic faculty found no significant difference in research productivity except at the assistant professor level [21]. Similar findings have been corroborated in a study comparing research productivity based on the gender of faculty in gastroenterology residencies [22]. A review of 34 different medical specialties comprising of 9952 academic physicians found no differences in research productivity throughout the careers of academic surgeons [23]. When comparing mean h-indices by gender, most surgical specialties have significant differences; one explanation for this finding is a significantly lower representation in surgical specialties rather than less research production [24]. The disproportionate percentage of total residency positions held by females in orthopedics has seen improvements, going from 11% in 2005 to 14.0% in 2016 [24]. Even after this increase, orthopedics still has the lowest proportion of female residents of any specialty. With a similar makeup in our sample (13.5% female), our findings add to the growing body of literature illustrating how little gender influences research production and academic success, specifically in surgical specialties. Similarly, with the USMLE and COMLEX board examinations transitioning to a pass/fail grading system, research productivity may become an increasingly important component for applicants matching into orthopedic surgery residency [12,25]. More than two-thirds of orthopedic program directors utilize USMLE Step 1 scores to screen the number of applicants necessary for review [26]. While the number of applicants placing into orthopedic surgery has remained steady over time-National Resident Matching Program (NRMP) match data suggest 80.0% placement from 2007 to 2014-abstracts reported by orthopedic applicants who matched more than doubled from 3.0 in 2007 to 6.7 in 2014 (p=0.04) [27]. Conversely, this number increased less drastically for all other specialty applicants, going from 2.2 in 2007 to 4.2 in 2014 (p=0.004) [27]. DePasse et al. [27] predicted a plateau in the mean USMLE scores, which we now see in the form of pass/fail scores beginning in January 2022. Additionally, another review utilizing NRMP data from 2006 to 2014 of matched US seniors was significantly greater than unmatched US seniors (p=0.035) for research production [28,29]. For academic residency programs, particular research metrics may be useful during residency selection if the goal is to recruit trainees with more research experience who could translate this experience into independent lines of research. Future studies should evaluate the effect of this change on the research outcomes of medical students from a cohort of students before and after this change.
Our study has both strengths and limitations. Regarding its strengths, we performed all data extraction in duplicate with masked extractors. This practice is considered the gold standard for conducting systematic reviews and is recommended by the Cochrane Collaboration [30]. We also performed a random sample of orthopedic surgery residency programs to improve the generalizability of our findings. We had 31 of 50 residency programs in our sample, a response rate of 62.0%; thus, we feel as though our data accurately portray the current state of affairs with orthopedic surgery specifically pertaining to gender representation. We utilized a multi-step process for obtaining resident data, including contacting programs, which is more robust than what has been done in previous studies, which limited their data collection efforts to examining websites and excluding programs that did not have websites with the pertinent information. Regarding its limitations, we performed a crosssectional study, and our results should be interpreted within the time frame of our sample and should not be generalized to other time periods. It is also possible that our random sample resulted in sampling bias. It is also possible that the surnames of residents may have changed over the course of the study, and we may have missed these changes during data collection. Our random generator includes few AOA-approved programs, potentially decreasing the representation of orthopedic graduates and academic achievement. Further investigation specifically for AOA-approved programs should be pursued.

Conclusions
Research productivity during residency training significantly correlates with the pursuit of academic achievement via fellowship or academic medicine. The present study represents an average of 5.6 publications, with the majority being observational reviews and published in highimpact-factor journals. Additionally, increased research output, especially first-author publications, significantly showed a positive correlation with further academic pursuits. Although increased research output showed further academic pursuits, 77.4% (n=264) of graduates pursued private practice vs 22.6% (n=77) entering academia. Contrary to the current literature, our study supports similar numbers of publications, h-indices, and number of firstauthor publications among women and men, thus illustrating that the only difference between men and women in orthopedic residencies is the rate at which they are accepted. This study provides a comprehensive appreciation of research productivity among orthopedic residents preresidency, during residency, and postresidency as it relates to additional academic pursuits, which may be useful for medical students and orthopedic residents in preparation for the application cycle.
Research funding: None reported. Author contributions: All authors provided substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; M.C., J.M.A., S.S., and M.V. drafted the article or revised it critically for important intellectual content; all authors gave final approval of the version of the article to be published; and all authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Competing interests: None reported.