Does the Medical College Admission Test Predict Global Academic Performance in Osteopathic Medical School?

Individuals applying for admission in AOA-approved osteopathic medical schools approved by the American Osteopathic Association provide their scores on the Medical College Admission Test (MCAT) as a standard application requirement. Most allopathic medical schools in the United States also require MCAT results from their applicants. The MCAT was originally developed in 1928 and has received regular updates since that time.¹ Today's applicants are using MCAT's fifth revision, which was introduced in 1991.¹ Usually taken in the year prior to medical school application, the MCAT is composed of four subtests: biological sciences (biology MCAT), physical sciences (physical MCAT), verbal reasoning (verbal MCAT), and a writing sample (written MCAT).

When selecting applicants for admission to medical school, admissions committees most often evaluate a candidate's MCAT scores in combination with his or her cumulative undergraduate grade point average (total UGPA) and undergraduate science GPA (science UGPA) in addition to performing an interpersonal evaluation using an interview process.^2,3 Although there may be some differences among institutions with regard to how strongly MCAT scores are weighted, these scores have long been an important part of the admissions process.

Studies have shown that MCAT performance has positive predictive value for academic performance in allopathic medical schools.^4-8 The criteria used to judge academic performance in allopathic medical school often include scores from the United States Medical Licensing Examination (USMLE) and the Medical Council of Canada Qualifying Examination (MCCQE) Part II.

The USMLE is divided into three “steps.” Step 1 tests candidates for comprehension of the basic sciences considered necessary to practice medicine. At the time our study was conducted, the questions in Step 2 involved only the application of medical knowledge and skills, assessing whether students can practice medicine with safety and competence. Currently, this step also has a standardized patient component. Finally, Step 3 provides a conclusive assessment of graduates to ensure they are prepared for the unsupervised practice of medicine. Similarly, the MCCQE Part II is designed to measure the clinical competence of postgraduate candidates who wish to practice medicine in Canada. The MCCQE Part II also includes problems in medicine, pediatrics, and obstetrics and gynecology as well as other disciplines judged by the MCC to be integral to well-rounded competence in general healthcare practices.

Low MCAT scores are associated with academic difficulty in the first 2 years of medical school.^4,9,10 High MCAT scores are linked to better performance on the USMLE.^4-7,11,12 Studies have also shown that higher clerkship performances are associated with strong MCAT scores.^13,14

Little has been reported about the MCAT and its predictive value for student performance in osteopathic medical schools, however. The results that have been reported measure performance using scores on the Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA),¹⁵ a national performance examination with three “levels” of testing, each designed to test students' knowledge of osteopathic medicine and the clinical skills considered necessary to practice osteopathic medicine. Level 1 of COMLEX-USA is a multiple-choice examination of basic science knowledge relevant to medical problems, usually taken at the end of the second year of osteopathic medical school. Level 2 of COMLEX-USA is a multiple-choice examination of clinical concepts and principles involved in medical problem-solving, usually taken in the fourth year of osteopathic medical school (COMLEX-USA Level 2-CE [Cognitive Evaluation]). A relatively new assessment model, COMLEX-USA Level 2-PE (Performance Evaluation), analyzes candidates' interactions with standardized patient cases. As with USMLE's Step 2, this patient-centered component was added to the examination after the study period. Level 3 of COMLEX-USA is the final part of the series, usually taken in the first year of postgraduate medical education.

In a study at the West Virginia School of Osteopathic Medicine in Lewisburg, Baker and colleagues¹⁵ found no significant correlation between students' MCAT scores and their later performance on COMLEX-USA Level 1. Evans and colleagues¹⁶ described a positive correlation between MCAT scores and COMLEX-USA Level 2 performance at Oklahoma State University College of Osteopathic Medicine (OSU-COM). Meoli and colleagues¹⁷ described positive relationships between MCAT scores and COMLEX-USA Level 1 and Level 2 performance at the University of Medicine and Dentistry of New Jersey–School of Osteopathic Medicine in Stratford. As of May 2004, no published studies described the relationship between MCAT scores and overall academic performance in osteopathic medical school.

The purpose of the present study was to examine the value of preadmission criteria including MCAT scores, total UGPA, and science UGPA in predicting global academic performance for osteopathic medical students. We did not evaluate results from COMLEX-USA Level 3 in the current study. Performance was measured using the following data:

• GPA in basic science (basic GPA)

• GPA in clinical clerkship (clinical GPA)

• cumulative GPA (total GPA)

• national board scores on COMLEX-USA Level 1 and Level 2

Methods

Subjects were 434 osteopathic medical students at OSU-COM who either graduated or were expected to graduate between the years 1999 and 2003. Of these subjects, 150 (34.6%) were women and 284 (65.4%) were men with a mean (SD) age of 30.33 (5.25) years. The Class of 1999 comprised 92 students; Class of 2000, 86; Class of 2001, 90; Class of 2002, 84; and Class of 2003, 82.

The dataset included 12 students who did not graduate because they were either dismissed or voluntarily withdrew from OSU-COM. Of these subjects, the Class of 1999 lost 8 students, the Class of 2000 lost 2, and the classes of 2002 and 2003 each lost 1 student. Data were obtained from the longitudinal applicant and enrollee datasets maintained by the college, and contained no personal identifiers. The OSU Center for Health Sciences' institutional review board determined that this study was exempt from the informed consent requirement.

The five dependent (performance) variables representing global measures of academic performance were: (1) basic GPA, (2) clinical GPA, (3) total GPA, (4) COMLEX-USA Level 1 score, and (5) COMLEX-USA Level 2 score. The six independent (predictor) variables were: (1) total UGPA, (2) science UGPA, and scores from the MCAT subtests: (3) biology MCAT, (4) physical MCAT, (5) verbal MCAT, and (6) written MCAT. The written MCAT subscore is measured using a scale that ranges from a low score of J to a high score of T. These scores were recoded for the purposes of the present study to numerical values ranging from 1 to 11, thus creating a discrete variable falling along a quantitative continuum.

The linear associations between pairs of dependent and independent variables were examined by calculating Pearson product moment correlation coefficients. Separate standard, multivariate linear regression analyses were conducted for each of the five dependent variables using two models: (1) total UGPA with the four MCAT subtests, and (2) science UGPA with the four MCAT subtests. The regression analyses were conducted to assess the relative importance of each of the independent variables to other independent variables within the set (ie, through calculation of beta [β] coefficients the relative contribution of each independent variable to the prediction of the dependent variable) and also to assess the combined effect of the set of five independent variables in predicting each dependent variable (ie, through calculation of R², the amount of variation in the dependent variable explained by the set of independent variables). Analyses were performed using SPSS statistical software (version 11.0; SPSS Inc, Chicago, Ill) with each analysis being evaluated for violations of assumptions (eg, nonlinearity). The Bonferroni adjustment was used to account for the inflation of type I error rate that may occur when multiple statistical tests are performed simultaneously.

Results

Table 1 presents the descriptive statistics (ie, mean [SD]) and Pearson product moment correlation coefficients between the pairs of dependent and independent variables. Total UGPA, science UGPA, and biology MCAT were all significantly correlated with each of the general performance variables except clinical GPA. Physical MCAT was significantly correlated with basic GPA, total GPA, and COMLEX-USA Level 1. Verbal MCAT was significantly correlated only with COMLEX-USA Level 2, whereas written MCAT was significantly correlated only with total GPA. These correlation coefficients were low. For example, the largest correlation between an MCAT subtest and a performance variable was r=0.22 (eg, physical MCAT and COMLEX-USA Level 1), which accounted for a small proportion of the total variation in the outcome (ie, r²=0.048).

Table 2 presents results from the separate multiple regression analyses of the five performance variables, predicted on the basis of total UGPA and scores from the four MCAT subtests. The different sample sizes for each analysis reflect the minimum number of subjects available for that particular analysis (ie, as a result of students not graduating). Four of the five regression models were statistically significant (P<.01), but the overall proportion of variance (R²) was low, ranging from only 0.12 (COMLEX-USA Level 1) to 0.18 (basic GPA and total GPA). The regression model predicting clinical GPA was not statistically significant.

The β coefficients for total UGPA were significant for all performance variables measured. For all five regression models, the β coefficients of total UGPA were of greater magnitude than those of the MCAT subtests, suggesting the greater predictive importance of total UGPA. Biology MCAT was a modest but significant predictor of basic GPA (β=.14). Physical MCAT significantly predicted COMLEX-USA Level 1 score, but also to a modest degree (β=.15). Verbal MCAT significantly predicted COMLEX-USA Level 2 score (β=.21). Written MCAT was not a statistically significant predictor of any performance variable. No MCAT subtest emerged as a significant predictor of either clinical GPA or total GPA.

Additional analyses were conducted using science UGPA, rather than total UGPA, as a predictor of academic performance. Results were compared with those obtained using total UGPA. As shown in Table 1, the correlations between the performance variables and total UGPA were slightly stronger than those with science UGPA, with the exception of COMLEX-USA Level 1. Table 3 presents a summary of results from regression analyses testing models predicting academic performance on the basis of the MCAT subtests combined with either total UGPA or science UGPA. In comparing the β coefficients, total UGPA again was modestly better than science UGPA in predicting all performance variables except clinical GPA and COMLEX-USA Level 1. In addition, the regression models combining total UGPA with MCAT subtests displayed a slightly greater proportion of variance in the performance measures than did regression models including science UGPA, again, with the exception of clinical GPA and COMLEX-USA Level 1.

Comment

In this study, total UGPA appeared to be the strongest predictor of global academic performance in osteopathic medical school. Overall, MCAT score appears to have limited value in predicting academic performance. We found a small difference between total UGPA and science UGPA in predicting overall academic performance, with total UGPA having only slightly stronger predictive value.

There are mixed findings regarding the strengths of using UGPA and MCAT score to predict medical school performance. Kulatunga-Moruzi and Norman¹⁸ found that UGPA had the most utility in predicting academic and clinical performance, a finding that is further supported by our results. However, other studies suggest that MCAT score is a better predictor of academic performance than UGPA. Wiley and Koenig⁴ found that MCAT scores had a slightly higher correlation (r=0.615-0.67) with academic performance when compared with UGPA (r=0.54-0.58). Veloski and colleagues⁵ reported that science MCAT was a better predictor of performance on the National Board of Medical Examiners Part I (a predecessor of the present USMLE Step 1) than UGPA. Swanson and colleagues¹² found that MCAT score was a better predictor of USMLE Step 1 performance than UGPA. Basco and colleagues⁷ showed that MCAT score was more strongly related to USMLE Step 1 performance than science UGPA. In addition, Julian⁸ recently reported that MCAT score is substantially better than UGPA in predicting performance on USMLE Steps 1, 2, and 3, meaning there is virtually no need to use UGPA to predict these scores in the future.

While biology MCAT and physical MCAT were significant predictors of basic GPA as well as COMLEX-USA Level 1 and Level 2, they were not significant predictors of clinical GPA. Of note, none of the MCAT subtests were found to be significantly correlated with clerkship performance alone (Table 1), nor were any found to be a significant predictor in combination with UGPA (Table 2). The high average GPA in clerkships at OSU-COM (mean, 3.79) may limit the ability to discriminate high performers from low performers. Another limitation could be the result of the nonstandardized clerkship examinations given in each rotation. It is important to note that OSU-COM does not use COMLEX-USA shelf examinations in clinical clerkship testing, differing from many allopathic medical schools that use standardized USMLE shelf examinations, a practice that may influence clerkship grades.

While written MCAT was found to correlate significantly, albeit modestly, with total GPA (Table 1), it was not found to be a significant predictor of any of the five performance variables when combined with the other MCAT subtests and total UGPA in the regression models (Table 2). This finding suggests limited predictive value for the written MCAT.

Conversely, verbal MCAT was found to be both a significant correlate (Table 1) and a significant predictor (Table 2) within the regression model of COMLEX-USA Level 2 scores. Though MCAT score was previously found to be correlated with COMLEX-USA Level 2 performance, this result was from a study¹⁶ that looked at only one graduating class (also at OSU-COM) and used average values of MCAT scores rather than differentiating between MCAT subtests as specified in the present study design. Therefore, it is difficult to compare directly the results of these studies. In contrast with the results of the regression analyses used in our study, Hojat and colleagues¹⁴ reported that written MCAT results were more closely associated with clinical competence and class rank than with basic science performance as measured by basic GPA. Kulatunga-Moruzi and Norman¹⁸ noted that verbal MCAT was useful in predicting communication skills on the MCCQE Part II. Roth and colleagues¹⁹ similarly reported that the most highly predictive factor for USMLE Step 2 performance was verbal MCAT (r=0.33). Daugherty and colleagues¹³ also suggested that verbal MCAT may have predictive value for identifying poor preclinical performers who would do better in clerkships.

The findings of the present study are limited in several ways. First, the dataset is comprised of academic records from a single osteopathic medical institution across a 5-year time span, thus limiting our ability to generalize the results across the osteopathic medical profession. Second, this dataset (as with all data taken from medical students) is subject to self-selection bias (ie, comprised only of individuals selected using the predictor variables), and includes a small number of students (2.8%) who showed poor performance on the general academic measures. In such cases, where there is a restricted and homogeneous sample, statistical theory suggests that a restriction in the range of possible values of normally distributed variables (and many nonnormally distributed variables) may occur. This range restriction, in turn, would reduce correlations that may otherwise be seen in unrestricted populations (eg, all MCAT examinees). The issue of restriction in range was analyzed by calculating corrected correlations between the dependent and independent variables using a standard formula.²⁰ However, corrected correlations were found to show a nearly 10% increase in absolute value above those presented in Table 1. This increase suggests that restricted range is not a substantial issue in this dataset. Lastly, the variance seen in MCAT scores may be the result of students' test-taking abilities.

Biology MCAT and physical MCAT were significantly correlated with most general academic performance variables, with the notable exception of clinical GPA. However, when compared in the regression analyses, biology MCAT emerged as the stronger predictor of most outcomes. These findings are likely the result of the degree of overlap between the two subtests (r=0.46). The extent of the overlap did not invalidate the regression analyses (ie, through multicollinearity), and the nature of the commonalities (eg, content, test-taking approach) is beyond the scope of this paper.

Results from this study raise several important questions:

• What role should each MCAT subtest serve in helping admissions committees at osteopathic medical schools chose among applicants for admission to their programs?

• How should the MCAT results be weighted relative to UGPA and preadmission interview performance?

• What if MCAT performance is predictive not only of success for applicants who achieve high scores but also of risk for specific “adverse events” (eg, course failure, failure of national board licensing examinations, or voluntary withdrawal prior to graduation) for applicants receiving low scores?

Our results indicate that MCAT scores have limitations when used to predict academic success. Though Swanson and colleagues¹² found a much larger variance (at r=0.615, the variance was 38%), our findings (at r=0.22, the variance was 4.8%) are substantially lower.

It is unclear whether the results of our study were unique to osteopathic medical education. More studies are needed to answer these questions, particularly through integrative approaches, such as examining multiple variables predictive of academic success with larger samples drawn from multiple schools and diverse geographic regions over time. Using a larger dataset that includes a higher number of unsuccessful students from other osteopathic medical schools may shed more light on this topic.