A retrospective and correlative analysis of academic and nonacademic predictors of COMLEX level 1 performance

Context: National licensing exams (NLEs) including the Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 1 evaluate student achievement. Scores have historically been utilized to stratify medical student applicants for residency. Grade point average (GPA), number of practice questions completed, and performance on practice exams have been shown to be predictive of NLE performance. Test anxiety and acute stress have been shown to negatively impact NLE performance. The role of study behaviors and other nonacademic factors in COMLEX Level 1 performance is unknown. Objectives: This study aims to evaluate academic and nonacademic factors and to correlate them with COMLEX Level 1 performance. Additional analysis is conducted to associate COMLEX Level 1 performance with academic and nonacademic factors when controlling for GPA. Methods: An anonymous online survey was administered to third(OMS III) and fourth-year (OMS IV) osteopathic medical students at Kansas City University that had completed the COMLEX Level 1 examination. In total, 72 students responded to the survey. Survey results were linked to student records of GPA and COMLEX Level 1 scores, resulting in 59 complete responses for analysis. Independent-sample t-tests and linear ordinary least squares regression were utilized to analyze the results. Results: Themajority of participants are male (62.7%) and OMS III (98.3%) with an average age of 27.14 ± 2.58 (mean ± standard deviation). Further demographic data reveal hours per week spent for personal time during dedicated study (n=46, 19.7 ± 18.53), hours of sleep per night during dedicated study (7.34 ± 0.92), and money spent on board preparation ($1,319.12 ± $689.17). High ($1,600–$3,000), average ($1,000–$1,500), and low ($100–$900) spenders do not statistically differ and COMLEX Level 1 performance is not related to the number of resources utilized (F statistics <1; p>0.05). Pearson correlations reveal a statistically significant relationship between COMLEX Level 1 scores with GPA (0.73, p<0.001), number of practice exams completed (0.39, p<0.001), number of questions completed (0.46, p<0.001), number of weeks of study (0.55, p<0.001), and preparation cost (0.28, p<0.05). The regression analysis revealed thatmoney spent on board preparation, number of questions completed, and time spent studying accounted for 75.8% of the variance in COMLEX Level 1 scores after controlling for GPA. Conclusions: The data show the association ofmoney spent on board preparation, numbers of questions competed, and time spent studying with a student’s COMLEX Level 1 score. Additionally, these results highlight the amount of money students spend on extracurricular materials to prepare for COMLEX Level 1, yet the data show that the number of resources that students utilized is not related to a student’s COMLEX Level 1 performance.

student achievement and mastery [1]. COMLEX specifically aims to assess a medical student's knowledge of biomedical science, clinical understanding, and osteopathic manipulative medicine (OMM). Furthermore, the role of these exams has expanded to include the assessment of applicants for residency [2]. Success on NLEs can largely be attributed to academic achievement and study habits [1,[3][4][5][6]. In one study, the grade point averages (GPAs) of osteopathic medical students (n=187) that completed the COMLEX Level 1 were found to correlate with COMLEX Level 1 scores (0.81) [1]. Important nonacademic factors that may influence NLE performance include financial investment and personal well-being [7]. Psychological variables, such as resilience, may also influence NLE performance [8]. Past research has shown that medical student (n=93) test anxiety was inversely correlated with United States Medical Licensing Examination (USMLE) Step-1 scores (b=−0.24, p=0.01) [8].
Previous findings have highlighted the strong positive correlation between GPA prior to and during medical school with NLE scores [1,3,4,9,10]. OMM course grades (n=217) have also been found to correlate with COMLEX level 1 performance (r 2 =0.28) [11]. Medical College Aptitude Test (MCAT) scores have shown variable success in predicting NLE scores [6,9]. School-specific factors, such as curriculum [4], or unevaluated individual difference factors [12], such as motivation, may play a role in the variability in the relationship between MCAT and NLE scores. Regarding study strategies, research has shown that retrieval-based strategies, spaced practice, and number of practice questions completed [7], as well as performance on the UWorld question bank and performance on practice exams [13,14], are predictive of NLE success. A specific study found that there was a strong positive correlation (r=0.66 to 0.74, p<0.001) between students' scores (n=2,414) on practice exams and the actual NLE [13]. Most studies assessing factors that affect NLE performance only address the prediction of USMLE scores. This illustrates the need for additional research around factors that are associated with a student's COMLEX performance. USMLE and COMLEX performance are moderately to strongly correlated [2,15]; nevertheless, it cannot be assumed that factors associated with USMLE success will generalize to COMLEX success.
Internet use for academic purposes has also been found to correlate with NLE success [16,17]. In one such study, medical student (n=170) use of study resources on Facebook was positively correlated (r=0.956) with NLE performance [17]. There are numerous colloquial blog posts and unpublished resources providing recommendations on how to study for USMLE and COMLEX. These anecdotal sources have fueled student commentary on medical school curricula, perceived inadequate preparation by medical educators for NLEs, and mental health [18]. Medical students face financial, interpersonal, and emotional stress. Burnout in medical school is an additional concern. In one such study, a significant amount of medical students and residents (n=381) reported experiencing above-normal psychological distress (41.5%) or high to very high psychological distress (9.2%) [19]. Previous studies have demonstrated an unclear relationship between burnout and academic achievement; one study looking at first-year medical students (n=277) found student engagement with studies, not burnout, to be associated with academic achievement [20]. A strong negative relationship does exist between test anxiety and NLE performance [7,8], yet it remains to be seen whether general anxiety and depression in medical students relates to NLE performance.
Resilience can be defined as a personal characteristic that describes an individual's ability to adapt to stress, trauma, adversity, or tragedy [21,22]. Indications of resiliency can be organized into four categories: hardiness, benefit finding, thriving, and posttraumatic growth [23]. Hardiness can be described as having a strong commitment to self, viewing change as a challenge, and being able to find meaning in life [24]. Benefit finding is being able to find the positive or personal growth within challenges [25]. Thriving is the ability to return to a level of functioning at or above baseline prior to a stressor [26]. Posttraumatic growth is a combination of being able to adapt and grow from traumatic events, which can include finding new or greater meaning in life as well as forming closer relationships [27]. When examined through the scope of medical education, medical and psychology students (n=560) who demonstrated greater resilience had lower rates of distress (p<0.001) [28]. The lower rates of distress in a resilient medical student may counteract test anxiety and allow for greater performance on NLEs. This aligns with the hardiness category as an indicator of resiliency. However, it is unknown how resilience and NLE performance are correlated.
Academic achievement, measured by GPA, plays a substantial role in the prediction of NLE scores. It is unknown how study behaviors and nonacademic factors correlate with COMLEX scores. Therefore, the primary aim of the current study is to evaluate these factors and correlate them with COMLEX scores. An additional aim of the study is to identify if specific factors, such as money spent on extracurricular NLE preparation, may explain additional variation in COMLEX scores after controlling for GPA. We hypothesized that resiliency and study behaviors identified by previous studies (e.g., number of questions completed) would be positively correlated with COMLEX performance.

Methods
This study was reviewed and approved by the Institutional Review Board at Kansas City University under 45 CFR 46.104 Exempt Category (d-2 Criteria #i). The IRB number for this study is: 1480505-1. No additional external or internal funding was utilized to conduct this study. Research subjects were informed about the study through an online study information sheet (SIS) that served as the first page of the online survey. The SIS outlined the research investigators, title of the study, commitment from the research participant, probability of harm, how results will be utilized, benefits of participation in the research study, how to withdraw or decline participation, and contact information for the principal investigator and the IRB at Kansas City University. The SIS was the first page of the survey. Clicking on the link for the study from the recruitment materials gave the research participant access to the SIS and the online survey. Participants had to click that they acknowledged the SIS to continue to the survey. No compensation was provided to research participants.
Participants for this study included primarily third-year (OMS III) osteopathic medical students who had completed the COMLEX Level 1 examination in the 2019 academic year by the time the study was conducted. Fourth-year (OMS IV) osteopathic medical students were also eligible if they met this criteria. Participants were recruited utilizing student email distribution lists and via Kansas City University Facebook student groups. An estimated 390 osteopathic medical students had the opportunity to take the survey via email.
The anonymous online survey was developed by the authors and was administered through Qualtrics (Qualtrics, Provo, UT) over a fourweek period in January 2020. The single recruitment email and private social media advertisement provided a link to click on to participate, which navigated prospective participants to a consent page and the survey. Participants provided their student identification numbers to allow for student records pertaining to GPA and national licensing board scores to be linked with the data collected in the survey. An independent analyst, who was not involved in the study, linked the data sets together and removed student identification numbers, resulting in an anonymous data set. The independent analyst was a staff member who serves as the manager of assessment for Kansas City University.
Responses were obtained from 72 medical students. Thirteen participants in the sample did not provide their student identification number, prohibiting the independent analyst from being able to link their academic performance data to survey responses, and subsequently their inclusion in statistical analyses. The final sample consisted of 59 participants. There were 37 male (62.7%) and 22 female (37.3%) participants. Among the sample, 98.3% (n=58) were OMS III students with only one participant reporting as an OMS IV student. The average age of the sample was 27.14 ± 2.58 (mean ± standard deviation; range: 24-36).
Research participants were surveyed for information regarding demographics, study behavior, and nonacademic variables such as preferred specialty of interest, financial burden, and resilience. Questions were in binary, multiple-choice, or free-text format with programmed follow-ups dependent on participant answers. Example survey questions included: how many hours per week did you take for personal time and recreation during dedicated board study, how many practice exams did you complete, and how much money did you roughly spend on board preparation materials. In regard to financial burden, participants were also asked to estimate to the best of their ability the total amount of money spent on preparation materials. Resilience was evaluated by the Connor Davidson-Resilience 10-item Scale (CD-RISC 10), with the highest score of 50 [29]. The scale utilizes a modified five-point Likert scale, with 1 corresponding to "not true at all" and 5 corresponding to "true nearly all the time." Higher overall scores indicate greater levels of resilience. See Appendix, Supplementary Material for a full copy of the survey.
When participants were asked about the hours spent on personal time during NLE preparation, some did not report engaging in that behavior. Only the participants who provided an estimate of hours spent for personal time were utilized for analysis (Table 1). Similarly, one participant did not complete the CD-RISC and two participants did not report the dollar amount spent on preparation materials. All analyses were run with complete case data. Independent sample t-tests were conducted to determine demographic differences in academic and nonacademic variables. Pearson correlations were conducted to evaluate relationships between study variables. A regression was conducted to determine if financial, psychological, and study habit variables were associated with a student's COMLEX score after controlling for GPA. A p value of 0.05 was considered statistically significant. Statistical analyses were completed utilizing SPSS software version 26.

Results
Preferred specialties among the participants were queried and stratified into two groups: low-competitiveness (LC) and high-competitiveness (HC). This stratification was conducted utilizing established competitiveness groupings based on National Resident Matching Program (NRMP) charting outcomes match data, which were modified from previously published criteria [30]. The moderate and high competitiveness specialties were combined into the HC group. Over half (52.5%) of the sample indicated an interest in a high-competitive specialty, whereas 39% reported interest in a low-competitive specialty, 3.4% in multiple specialties, and 5.1% were undecided. Descriptive statistics for academic and nonacademic variables are presented in Table 1. The average GPA for the final sample was 3.33 ± 0.37. The average COMLEX Level 1 score was 598.4 ± 78.7, and the goal pretest COMLEX Level 1 score was 587.97 ± 38.33. Further demographic data reveal money spent on board preparation ($1,319.12 ± $689.17), hours of sleep per night during dedicated study (7.34 ± 0.92), hours per week spent for personal time during dedicated study (n=46, 19.7 ± 18.53), and CD-RISC 10 (40.53 ± 4.24). Independent samples t-tests were conducted to determine differences in academic and nonacademic variables between the LC and HC groups. There was a statistically significant difference between these groups in the number of practice exams completed [t(52)=−3.12, p<0.01]. Those in the HC group (6.81 ± 2.36) completed on average approximately two additional practice exams compared to the LC group (4.78 ± 2.35). However, this difference in practice exams did not influence overall COMLEX Level 1 performance. Pearson correlations between academic and nonacademic factors are presented in Table 2.
An ordinary least squares regression was conducted to determine the contribution of academic and nonacademic factors in the prediction of COMLEX Level 1 scores after controlling for GPA. GPA was entered as a predictor in the first model. GPA, number of questions completed, money spent on board preparation, and number of weeks of study were entered as predictors into the second model. Overall, the first model with GPA was statistically significant in accounting for 53.2% (adjusted R 2 ) of the variance in COMLEX Level 1 scores [F (1, 55) Table 3 for regression results.
Because money spent on board preparation was found to be a statistically significant predictor of COMLEX scores (β=0.21, p<0.01), the quantity of resources utilized was   assessed. Three student spending categories were stratified as high ($1,600-$3,000), average ($1,000-$1,500), and low ($100-$900) spenders. These three groups did not statistically differ in the average number of resources (content or question banks) they reported (F statistics <1; p>0.05), and the number of resources utilized was not related to COMLEX performance (Pearson correlations ranging from −0.11 to 0.13, p>0.05). See Table 4 for exam preparation resources by spending group based on answers to survey questions.

Discussion
Medical students prepare for NLEs in different ways and with different motivations. In this study, we surveyed OMS III and OMS IV osteopathic medical students to determine correlations between academic variables, such as GPA and COMLEX Level 1 scores, with nonacademic variables such as study habits and resilience. On average, the medical students in our sample spent greater than $1,000 on board preparation materials, yet the number of resources purchased did not correlate with higher COMLEX Level 1 scores. In addition, students pursuing highly competitive residencies take, on average, two additional practice exams, yet these additional practice exams do not correlate with higher COMLEX Level 1 exam scores.
Results indicated that there were no statistically significant relationships between resilience, amount of sleep, and amount of free time spent with performance on the COMLEX Level 1 exam. Resiliency scores in the sample were generally high, indicating that many students subjectively endorsed having resilient traits. This may be a feature of the sample or indicative of medical students' ability to persist in the face of many challenges. Sleep and amount of free time varied in the sample and may speak more to individual differences around responsibilities outside of medical school (such as family obligations) or to the nature of the sample size. These variables should be further evaluated for their relationship with NLE performance as well as other factors such as motivation and confidence. After controlling for GPA, three factors were able to explain a significant amount of the variance in COMLEX Level 1 scores. These additional factors were the number of questions completed, number of weeks of study, and money spent on board preparation. Retrieval practice, spaced practice, and self-testing have been shown to be successful study techniques because their use improved student retention of study material and enhanced student learning [31][32][33]. In one such study, long-term retention and application of materials were improved (p<0.0001) in medical students (n=47) that utilized repeated testing [33]. The results of this study corroborate these other studies by demonstrating that time spent studying and the number of questions completed are significant factors associated with a student's COMLEX Level 1 performance.
Money spent on board preparation is also found to be an additional factor associated with a student's COMLEX Level 1 performance. This result is potentially problematic and may place students that are socioeconomically disadvantaged in an unfair situation. The data indicate that the high spending group ($1,600-$3,000) did not utilize more resources for studying than other groups. The high spending group utilized less total content resources, on average, when compared to the other spending groups. This may indicate that the high spending group purchased fewer, more expensive resources. For example, all of the students in the high spending group utilized the study resource UWorld, whereas 88% of students from the low spending group utilized UWorld. Overall, the cost of NLE preparation may disproportionately impact socioeconomically disadvantaged medical students. More research is needed to tease apart the difference in board preparation investment, to assess where the money from the high spending group is going, and to evaluate how NLE preparation cost impacts career development outcomes such as residency match success and long-term debt burden.
The findings in this study are limited in a few different ways. The final sample size is small, and the results from this study may not be generalizable to other medical student populations. Students with higher levels of resilience and academic performance may have been more likely to respond to the survey than other students. Indeed, the average COMLEX Level 1 score and GPA reported in the survey results are greater than the average for the entire class, leading to sampling bias and response/nonresponse bias. Despite the small sample size, our results share trends with other similar studies on key factors such as COMLEX and USMLE correlation and with the number of questions completed during NLE preparation predicting NLE performance [34,35]. Specific prior study results revealed that attempts by medical students (n=274) at more practice questions was significantly associated with USMLE Step 1 scores [35]. Several of the factors surveyed, including financial investment and study behaviors, are reported in a subjective fashion. Students reported information retrospectively and may have misremembered or guessed how they studied or the amount they spent on resources.
The finding that money spent on NLE preparation is associated with a student's COMLEX Level 1 exam performance prompts future research. It is not known whether students felt adequately prepared for the NLEs by the medical school curriculum or if students are motivated to purchase outside board preparation materials based on feelings of anxiety to increase their NLE performance. It also remains to be seen whether the transition of the COMLEX Level 1 and USMLE Step 1 to pass/fail grading will change the amount of money that students spend on board exam preparation. It may be that COMLEX Level 2 CE and USMLE Step 2 CK become more important for postgraduate residency candidacy. Further studies could evaluate how resilience, financial investment, and other factors relate to performance on these NLEs. Students take COMLEX Level 2 CE and USMLE Step 2 CK later in medical school, so they likely have a better understanding of their career plans. Future studies could also assess how clinical experience has influenced student resilience.
Our data indicate that time spent studying and the number of questions completed are essential factors associated with COMLEX Level 1 performance. A potential intervention that may operationalize this finding is the practice of spaced retrieval. It is entirely possible that the introduction of spaced retrieval practice to students as a study technique may enhance student performance on NLEs by giving students an effective study strategy to interweave NLE practice questions within the standard medical curriculum. In addition, the use of self-testing and spaced repetition would allow medical students to study for a longer period of time and to see more practice questions prior to taking an NLE, all of which may improve a student's NLE performance. These study techniques represent the philosophical opposite of cramming and have the added benefit of enhancing long-term retention of the studied material [36]. Additionally, spaced-retrieval practice has been shown to protect memory against acute stress, which may be an additional positive effect to potentially enhance NLE performance [37]. Informing medical students of spaced retrieval practice early in their career would be a simple and inexpensive way to potentially enhance student performance on NLEs.

Conclusions
Noncognitive factors, such as psychological traits, time management skills, and socioeconomic status, are increasingly being elucidated as relevant to scholarly performance. This is especially pertinent given the evolving landscape of medical education and heightened emphasis on student well-being. This study preliminarily highlights money spent, practice questions completed, and amount of time spent studying as being associated with a student's COM-LEX Level 1 performance. Furthermore, amount of sleep, amount of free time, and resiliency are not correlated with performance on the COMLEX Level 1. Lastly, despite the amount of money that students spent on extracurricular NLE preparation resources, the number of extracurricular resources utilized is not correlated with increased COMLEX Level 1 performance. In conclusion, this research study suggests that one possible inexpensive operationalization of these results would be the inclusion of spaced retrieval practice in NLE preparation. Further research is required to identify how these and other factors should be accounted for in osteopathic medical education curricula and resource allocation.
Acknowledgments: The authors would like to acknowledge Andrea Hanson, BA (Manager of Assessment at Kansas City University) for help in deidentifying the dataset. The authors would also like to acknowledge Monika Koirala, BS, OMS II (College of Osteopathic Medicine at Kansas City University) for her help in revising the study for publication. Research funding: None reported. Author contributions: All authors provided substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; all authors drafted the article or revised it critically for important intellectual content; all authors gave final approval of the version of the article to be published; and all authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Competing interests: None reported.
Informed consent: Research subjects were informed about the study through an online study information sheet (SIS) that served as the first page of the online survey. The SIS outlined the research investigators, title of the study, commitment from the research participant, probability of harm, how results will be used, benefits of participation in the research study, how to withdraw or decline participation, and contact information for the principal investigator and the IRB at Kansas City University. The study information sheet was the first page of the survey. Clicking on the link for the study from the recruitment materials gave the research participant access to the SIS and the online survey. Participants had to click that they acknowledged the study information sheet to continue to the survey. Ethical approval: This study was reviewed and approved by the Institutional Review Board at Kansas City University under 45 CFR 46.104 Exempt Category (d-2 Criteria #i). The IRB number for this study is: 1480505-1.