Key factors for residency interview selection from the National Resident Matching Program: analysis of residency Program Director surveys, 2016 – 2020

Context: As the number of medical school graduates continues to outpace the available residency training positions, applying for residency in the United States has become a highly competitive process, often associated with a low rate of selection and invitation for interview. The National Resident Matching Program (NRMP) Program Director survey provides data assessing factors considered by Program Directors (PD) in selecting and inviting candidates for interview. Assessing the evolution of these factors over time is e ﬃ cacious to inform and guide prospective applicants toward improving preparation for residency application. Objectives: We aim to synthesize NRMP data showing factors that PDs reported and rated as important in their decision to select and invite applicants for interview. Methods: Data from residency PD surveys from 2008 to 2021 were accessed, but after applying inclusion/exclusion criteria, only the data from 2016 to 2020 were reviewed and analyzed. The NRMP survey reports provided two metrics that characterized PDs ’ evaluation of the residency factors for interview, namely, “ percent citing factor ” and “ average rating ” on a 0 to 5 Likert-type scale. These two metrics were combined into an aggregate measure of importance (AI), and another measure of relative importance (RI) was constructed from normalizing the AI of each individual factor to the sum of the AI within each survey year. Results: The top ranked factors were United States Medical Licensing Examination (USMLE) Step 1/Comprehensive Osteo-pathic Medical Licensing Examination (COMLEX) Level 1, Letter of Recommendation (LOR) in the specialty, Medical Student Performance Evaluation (MSPE/Dean ’ s Letter), and USMLE Step 2 Clinical Knowledge (CK)/COMLEX Level 2 Cognitive Exam (CE) score, any failed attempt in USMLE/COMLEX, and perceived commitment to specialty. Factors rising in importance were Audition Elective/Rotation Within Your Department, Personal Statement (PS), Perceived Commitment to Specialty, Perceived Interest in Program, LOR in the Specialty, Other Life Experience, and Personal Prior Knowledge of the Applicant. Factors with declining importance were Interest in Academic Career, Awards or Special Honors in Basic Sciences, Graduate of Highly Regarded US Medical School, Awards or Special Honors in Clinical Clerkships, Lack of Gaps in Medical Education, Awards or Special Honors in Clerkship in Desired Specialty, and Consistency of Grades. Compared to the 2021 PD survey, our ﬁ ndings show continued predictive consistency, particularly related to specialty and program commitment. Conclusions: The factorsidenti ﬁ edfortheselectionofmedical school graduates for interview into a residency program reveal that PDs move toward a more integrated approach. Speci ﬁ cally, PDs are placing increasing emphasis on factors that border on subjective qualities more so than the more traditional, quantitative, and objective metrics. Medical students and educators need to continually apprise themselves of the NRMP data to inform students ’ preparation endeavors throughout medical school to strengthen their application portfolios and enhance their competitiveness for the matching process.

and outside (international medical graduates) the United States.Hence, the number of medical students has been growing in recent years without attendant increases in residency slots at the same pace.Consequently, the number of students applying for residency consistently surpasses the number of postgraduate training positions available.In 1976, there were 16,112 first-year residency (PGY1) positions available with 16,728 active applicants-this represented a deficit gap of about 3.8 %.As of 2021, these numbers have grown to 35,194 and 42,508, for available positions and active applicants, respectively [1].This represented a deficit gap of approximately 20.8 %, compared with 3.8 % in the earlier decades.Increasingly, the transition process has become highly competitive and ever-exhausting, and the intensity of the process and the accompanying disappointments it engenders in many applicants have become concerning [2].
In response to pressure from this rising competition and to boost the likelihood of getting into a residency, graduates are submitting an increasing number of applications over time [3].Relative to the year 2005, the average number of applications submitted by US medical students has increased by approximately 130 %, that is, from 30.3 to 72.8 per applicant in 2021 [3].The perceived competition has resulted in many applicants hedging a successful match by submitting larger numbers of applications.Although this strategy may feel intuitive and attractive, it can be costly and unaffordable to less financially privileged applicants [4].Moreover, this tactic may have diminishing benefits, and success may vary by specialty [5].
With an ever-increasing number of applications and already limited resources and time for programs to adequately review applications [4], Program Directors (PDs) are forced to implement filters to distill the number of applications before a full review.Despite this practice, more than half of PDs spend less than 10 minutes reviewing each of the resultant applications even after the filtration [6].Therefore, for applicants, obtaining interviews in a congested system can be a difficult endeavor.
To our knowledge, although there is growing literature guiding PDs in this evolving process [7][8][9][10], comparable resource or study is limited for prospective applicants.The National Resident Matching Program (NRMP) publishes data from surveyed PDs informing students on factors considered for interview selection.Although this provides valuable information, it is limited as a snapshot of the present status and warrants a longitudinal synthesis that would provide more comprehensive information to guide student application decisions.To give a time-dependent, longitudinal perspective to residency applicants, we conducted a retrospective analysis of national surveys of PDs [11,12] over the period 2016 to 2020 to identify prevailing trends and dominant factors that PDs considered for the invitation of applicants for residency interviews.By being well informed of the specific factors impacting selection decisions, early-career students would be able to tailor their preparation and application accordingly to obtain interviews in an increasingly competitive atmosphere.

Methods
This study was a retrospective review of secondary data from publicly available datasets.The data were obtained from the biannual publication of PD surveys by the NRMP.They came as de-identified and reported in aggregate, so no human subject research concerns were applicable to qualify for an Institutional Review Board (IRB) review.
The PD surveys for the period 2016 to 2020 on factors considered during the interview selection process for residency were accessed from the NRMP's website and reviewed.PD response rates computed as (number of respondents)/(total number of surveys sent), declined with rates of 1,435/3,599 (39.9 %), 1,333/4,546 (29.3 %), and 924/5,143 (18 %) for the years 2016, 2018, and 2020 [1,12], respectively.Data collection and flow chart analyses for this study are shown in Figure 1.Over the period under review, 33 factors were at play for consideration by the PDs in Reports identified from NRMP database: 3 (2016, 2018, 2020) Aggregate Importance of the individual factors from NRMP reports 2016, 2018, and 2020 calculated: Percent Citing Factor X Average Rating; all 33 factors included in the analysis; median plotted in Figure 2 Factors identified to be rising in importance, Figure 3(A 2).Utilizing calculated AI values, the relative importance (RI) of each factor was derived.RI was then utilized to identify factors that showed a significant increase or decrease in importance (see Figure 3).assessing the applicant pool and making their final decision for invitation for interview.
To analyze the perceived importance of the factors as a composite of the two PD survey metrics ("percent citing factor" as important and "average rating on a Likert-type scale"), a measure called Aggregate Importance (AI) for each individual factor within their respective surveys was created.Each AI was calculated by multiplying the percentage of PDs citing the use of each factor by its corresponding average Likert rating (scale: 1=not at all important, through 5=very important) serving as a weighted score.This was done to ensure that variations from the two different measures reported for each factor were balanced out as a single aggregate index.For example, the measure of importance of a factor assigned a low percentage citation, yet had a relatively high Likert score or vice versa.Mean AI of each factor among them was then calculated, allowing us to determine the order of importance of the AIs based on the magnitude of the means (Figure 2).
Trend analyses were performed to identify appreciating and depreciating factors over the study period.To adjust for time-dependent trends, each year's data was internally normed/ modified to offset year-to-year variations, likely due to fluctuations in the PD response rates.For this norming, the relative importance (RI) for each factor was obtained by dividing the individual factor's AIs by the sum of all factors' AI in the survey year, multiplied by 100 %.To observe meaningful time-dependent changes, the RIs were subjected to a linear regression analyses to identify possible increasing or decreasing trends in the slopes.We identified rising and declining factors as those with a slope >0.1 and <−0.1, respectively, and with a corresponding R 2 >0.8. Figure 3 shows the factors and their corresponding time-dependent changes.
A follow-up content analysis was conducted on the resulting subset of factors from the initial 33 into logical categories that were underpinned by a common defining characteristic.Those were factors found to be significant from the quantitative analysis.

Rank of aggregate importance for factors considered for interview selection
Utilizing the AIs of possible values of 0 through 5, values greater than or equal to 3 signified higher importance with  Most of these factors could be categorized to "professional characteristics, and performance and commitment to specialty/program."In general, this group echoes the theme, "commitment to specialty" identified in the AI≥3.0 group.

Time-dependent trends
From the linear regression, utilizing the RIs of all 33 factors, time-dependent trends for rising and declining factors were obtained.Seven factors with a slope>0.1 and R 2 >0.8 showed a rising trend (see data table in Figure 3A), and another seven factors with a slope<−0.1 and R 2 >0.8 exhibited a declining trend (see data table in Figure 3B).
Based on the shared innate characteristics, these 14 factors can be grouped into four general categories (Table 1): Category I includes factors reflecting "Relationship of Individual to Specialty," Category II "Relationship of Program to Individual," Category III "Innate Personal Characteristics," and Category IV "Academic Performance." Category I ("Relationship of Individual to Specialty") was constituted by the two factors, perceived commitment to specialty and LOR in the specialty, which relate to demonstrating one's dedication toward the applied specialty.Category II ("Relationship of Program to Individual") may relate to program preference toward inviting known candidates and those whose applications suggest that they are likely to rank their program highly.Category III ("Innate Personal Characteristics") consists of factors that offer additional insights or information that may relate to Categories I/II, depending on one's unique background and the content of one's Personal Statement.Meanwhile, Category IV ("Academic Performance") relates to academics, specifically experiences from within the field of medicine and evidence of superior scholastic abilities.
In general, Categories I-III are constituted by factors that cannot be easily quantified utilizing standardized metrics, instead relying on relatively subjective assessments as interpreted by individual faculty within a program and as interpreted between programs.These factors rely largely on characteristics existing outside of a candidate's scholastic control, with reliance on intrinsic qualities and experiences.Together, these describe personal characteristics and other knowledge of the applicants.

Rising factors
Among the seven factors displaying a rising trend (see Figure 3A and its associated data table), Audition elective/ rotation within your department had the sharpest increase based on slope, followed closely by Personal Statement.It is also noted that, with the exception of Other Life Experiences, all other six factors among this group have stronger mean RI of >3.0, with the LOR in the specialty having the highest mean RI.These metrics suggest that these seven rising factors also have a greater weighted importance among the 33 factors.In categorization, all seven factors fall into Categories I-III (Table 1).

Declining factors
Seven factors showed a declining trend (see Figure 3B and its associated data table).Consistency of grades showed the sharpest decline based on slope, followed by Awards or special honors in clerkship in desired specialty.In addition, these seven factors collectively have weaker mean RIs, ranging from 1.03 (the lowest importance: Awards or special honors in basic sciences) to 2.98 (Consistency of grades).Together, these metrics suggest these seven declining factors also have a lesser weighted importance among the 33 factors.In categorization, all seven factors fall into Category IV (Table 1).

Discussion
Utilizing the aggregated analyses considering Percent Citing Factor with their respective Average Rating released from NRMP PD surveys, we determined over the period covered by this study (2016-2020) the factors that PDs considered in inviting applicants for residency interviews.Our ranking order analyses identified factors that are most valued by the PDs in the initial selection process, which were those providing a construct of academic performance, endorsements, and commitment to specialty.
The time-dependent trending studies identified increasing and decreasing trends among the 33 factors that PDs considered for inviting applicants during 2016-2020.We showed seven factors of an increasing trend with Audition elective/rotation within your department and Personal statement have the highest rate of increase.Furthermore, among the seven increasing factors, Perceived commitment to specialty and LOR in the specialty had the highest RI.Overall, these factors were associated with an underlying construct of applicant personal characteristics and knowledge that could be described as not easily amenable to the traditional, objective, and quantifiable metrics for assessment.This is in marked contrast with the characteristic of factors that demonstrated decline in our study.Seven factors were identified to have a declining trend, with Consistency of grades exhibiting the sharpest decline.In fact, all these declining factors are related to academics, including mostly recognition of superior scholastic abilities.In other words, the factors in decline are thematically opposite to those seen rising in valuation and provides evidentiary support for one another.Together, we observe a theme consistent of evaluating and selecting an applicant in a more holistic approach [13].
Among all the trending factors, it is not surprising that LORs have increasingly served to engender a more personalized perspective of an applicant.Such attestation provides independent yet focused testimony that may not be otherwise discerned [14,15].Although selecting LOR writers that will speak highly of an applicant is important, it may be wise to limit the number of writers from outside of the applied specialty to intentionally create a networking effect within a desired specialty and signal commitment.Additionally, a chair letter may be an additional opportunity to demonstrate commitment, which is a factor not yet measured by the NRMP.Reliance on indirect appraisals can be circumvented via audition rotations (ARs).As one of the fastest rising factors, it allows a more comprehensive and direct evaluation by programs, which is particularly important for osteopathic students [16].Furthermore, ARs can offer applicants not only an opportunity to express explicit interest in particular programs but also may communicate geographical preference and specialty commitment.
While carrying out this study, our team also substantiated the disparity in PDs assessing USMLE and COMLEX results from applicants.At the beginning of 2020, the factors pertaining to the USMLE/COMLEX were assessed separately for the first time by the NRMP.Prior to 2020, the "USMLE Step 1/COMLEX-USA Level 1" was the most reported factor that the PDs considered.Meanwhile, in 2020, while Step 1 remained the most cited, Level 1 was ranked number 18. Therefore, parcellation of these factors seen in 2020 onwards yields valuable insight on the inequitable weighting of these examinations by PDs and an inadvertent measurement bias in prior surveys.This is despite the American Medical Association's (AMA) unanimous vote in 2018 to officially recognize these examinations as equal, promoting their acceptance at all US residency programs [17,18].Despite this inequity, for PDs who do take it into consideration, similar values of importance are assigned [12].
Although osteopathic students should be viewed now (and always) as equals to their allopathic colleagues, this may not be so and may be slow to change without a catalyst.Despite tremendous progress, a glass ceiling may exist for osteopathic candidates as allopathic candidates appear to be given preference in competitive specialties, even at present [19].Accordingly, among all specialties, 37 % of PDs 'never' or 'seldom' consider an osteopathic senior for an interview, a figure approaching 100 % in some fields (Figure 3 Summary of Program Interviewing and Ranking Activities.Sub figure: Percentages of Programs Interviewing Candidates by Applicant Type, 2021) [1].Given the ongoing concerns for systemic discrimination [20], osteopathic medical graduates should be considered within programs' Diversity, Equity, and Inclusion protection.

Limitations
While many interesting insights may be derived from this study, there were several limitations.Each year, the response rate to the PD surveys was relatively low, resulting in populations of PDs (responding vs. nonresponding) with potential bias due to lack of representativeness of the respondent sample.As a survey, it may be riddled with inherent selection bias.Participating PDs may be more inclined to favor some factors than nonrespondent PDs.Additionally, factor emphasis may differ by specialty and should be a topic of future research to benefit students with interests in specialty areas.

Conclusions
The factors that residency Program Directors consider as most important for their decision to invite applicants to interview evolve over time.Although some important factors maintained their traditional recognition, increasingly many nontraditional factors are rapidly gaining recognition of importance.Evolving values will continue to shape the future and attendant shifts in the importance of these factors in postgraduate medical education.Understanding these changes will be imperative to maximize a medical student's success in obtaining an interview as a first step toward entry into residency (postgraduate medical education).

Figure 3 :
Figure 3: Factors considered among residency program directors trending in increasing (A) and decreasing (B) importance.Factors demonstrating increasing and decreasing importance over time can be seen in panels (A) and (B), respectively.The mean and slope for each factor is indicated next to their respective plotted line.

Table  :
Rising and declining factors identified for the period, -.