The income gap between specialists and primary care physicians and among specialists is well established, but the drivers of this difference are not well delineated. Using the Community Tracking Study (CTS) Physician Survey, we sought to isolate and compare premiums paid to physicians for specialization and the proportion of time spent on offices visit rather than procedures. We divided medical subspecialties according the proportion of Medicare billing for Evaluation and Management (E&M) codes for the specialty as a whole. We report substantial differences in income across physician specialty, and over 70 percent of the difference in income remained controlling for factors that may confound the relationship between income and specialty including gender, location and type of practice, and hours. We note a large variation in premiums for specialization: 11.3–46.8 percent above family medicine after controlling for confounders. Classifying medical subspecialties by E&M billing as procedural versus non-procedural specialties revealed clear income differences. Controlling for confounders, procedural medical specialties earned 37.5 percent more than family medicine, as compared with 15.3 percent for non-procedural medical specialties. This analysis suggests that differences in physician income and resulting incentives are a direct consequence of the payment structure itself, rather than compensation for additional years of training or a reflection of different underlying demographics.