A hierarchical Bayesian model of pitch framing

2.1 PITCHf/x data

In 2006, the TV broadcast company Sportvision began offering the PITCHf/x service to track and digitally record the full trajectory of each pitch thrown using a system of cameras installed in major league ballparks. During the flight of each pitch, these cameras take 27 images of the baseball and the PITCHf/x software fits a quadratic polynomial to the 27 locations to estimate its trajectory (Sidhu and Caffo 2014). This data is transmitted to the MLB Gameday application, which allows fans to follow the game online (Fast 2010). In addition to collecting pitch trajectory data, an MLB Advanced Media employee records game-state information during each pitch. For instance, he or she records the pitch participants (batter, catcher, pitcher, and umpire) as well as the outcome of the pitch (e.g, ball, swinging strike, hit), the outcome of the at-bat (e.g. strikeout, single, home run), and any other game action (e.g. substitutions, baserunners stealing bases). The PITCHf/x system also reports the approximate vertical boundaries of the strike zone for each pitch thrown. Taken together, the pitch tracking data and game-state data provide a high-resolution pitch-by-pitch summary of the game, available through the MLB Gameday API.

Though our main interest in this paper is to study framing effects in the 2014 season, we collected all PITCHf/x data from the 2011 to 2015 regular season. In Section 2.2, we use the data from the 2011 to 2013 seasons to select the function of pitch location f^u(x, z) from Equation 1. We then fit our model using the 2014 data and in Section 3.1, we assess our model’s predictive performance using data from 2015. In the 2014 season, there were a total of 701,490 pitches, of which 355,293 (50.65%) were taken (i.e. not swung at) and of these taken pitches, 124,642 (35.08%) were called strikes. Rather than work with all of the taken pitches, we restrict our attention to those pitches that are “close enough” to home plate to be “frameable.” More precisely, we first approximate a crude “average rule book strike zone” by averaging the vertical strike zone boundaries recorded by the PITCHf/x system across all players and all pitches, and then focused on the N = 308,388 taken pitches which were within one foot of this approximate strike zone. In all, there were a total of n_U = 93 umpires, n_B = 1010 batters, n_C = 101 catchers, and n_P = 719 pitchers.

2.2 Adjusting for pitch location

Intuitively, pitch location is the main driver of called strike probability. The simplest way to incorporate pitch location into our model would be to include the horizontal and vertical coordinates (x, z) recorded by the PITCHf/x system as linear predictors so that fu(x,z)=θxux+θzuz, where θxu and θzu are parameters to be estimated. While simple, this forces an unrealistic left-to-right and top-to-bottom monotonicity in the called strike probability surface. Another simple approach would be to use a polar coordinate representation, with the origin taken to be the center of the approximate rule book strike zone. While this avoids any horizontal or vertical monotonicity, it assumes that, all else being equal, the probability of a called strike is symmetric around this origin.

Such symmetry is not observed empirically, as seen in Figure 2, which divides the plane above home plate into 1′′ squares whose color corresponds to the proportion of pitches thrown in the 3 year window 2011–2013 that pass through the square that are called strikes. Also shown in Figure 2 is the average rule book strike zone, demarcated with the dashed line, whose vertical boundaries are the average of the top and bottom boundaries recorded by the PITCHf/x system. If the center of the pitch passes through the region bound by the solid line, then some part of the pitch passes through the approximate strike zone. This heat map is drawn from the umpire’s perspective so right handed batters stand to the left of home plate (i.e. negative X values) and left-handed batters stand to the right (i.e. positive X values). We note that the bottom edge of the figure stops 6 inches off of the ground and the left and right edges end 12 inches away from the edges of home plate. Typically, batters stand an additional 12 inches to the right or left of the region displayed. Interestingly, we see that the empirical called strike probability changes rapidly from close to 1 to close to 0 in the span of only a few inches.

Figure 2:

Heat map of empirical called strike probabilities, aggregate over the 3-year window 2011–2013. The boundary of the approximate 2014 rule book strike zone is shown in dashed line. If the center of the pitch passes through the region bounded by the solid line, some part of the pitch passes through the approximate strike zone. Red = 100% called strike probability, white = 50%, and blue = 0%.

Rather than specifying a explicit parametrization in terms of the horizontal and vertical coordinates, we propose using a smoothed estimate of the historical log-odds of a called strike as an implicit parametrization of pitch location. This is very similar to the model of Judge et al. (2015), who included the estimated called strike probability as a covariate in their probit model.

Figure 3 plots the spatial distribution of taken pitches broken down by the batter and pitcher handedness. Once again, the plots are drawn from the umpires’ perspective so that a right handed batter stands to the left side of the figure and vice versa. We see immediately that the spatial distribution of taken pitches varies considerably with the combination of batter and pitcher handedness. When the batter and pitcher are of the same handedness, we see a decidedly higher density of “low and outside” pitches near the bottom corner of the average rule book strike furthest away from the batter. In contrast, in the matchup between left-handed batters and right-handed pitchers, we see a higher density of pitches thrown to the outside edge of the strike zone further away from the batter. The differences in spatial distribution of pitches seen in Figure 3 motivate us to use a separate smoothed estimate of the historical log-odds of a called strike for each combination of batter and pitcher handedness.

Figure 3:

Kernel density estimate of pitch location based on batter and pitcher handedness. Figures drawn from the umpires perspective so right-handed batters stand to the left of the displayed strike zone. Darker regions correspond to a higher density of pitches thrown to those locations.

Inspired by Mills (2014), we fit generalized additive models with a logistic link to the data aggregated from 2011 to 2013, one for each combination of pitcher and batter handedness, hereafter referred to as the “hGAMs” or “historical GAMS.” These models express the log-odds of a called strike as a smooth function of the pitch location. Figure 4 shows the hGAM forecasted called strike probabilities. Interestingly, we see that for right handed pitchers, the corresponding hGAMs called strike probability surfaces very nearly align with the average rule book strike zone. For left-handed pitchers, however, the hGAMs forecast a high called strike probability several inches to the left of the average rule book strike zone. This is perhaps most prominent for the matchup between right-handed batters and left-handed pitchers. For each taken pitch in 2014 dataset, we used the appropriate hGAM to estimate the historical log-odds that the pitch was called a strike. We then use these estimates as continuous predictors in our model, so that potential player effects and count effects may be viewed as adjustments to these historical baselines.

Figure 4:

hGAM forecasts based on batter and pitcher handedness. Red = 100% called strike probability, white = 50%, and blue = 0%.

2.3 Bayesian logistic regression models

Before fully specifying our models, we label the 93 umpires u₁,…,u₉₃. Consider the ith called pitch and let y_i = 1 if it is called a strike and y_i = 0 if it is called a ball. Let h_i be a vector of length four, encoding the combination of batter and pitcher handedness on this pitch and let LO_i be a vector of length four, containing three zeros and the estimated log-odds of a strike from the appropriate historic GAM based on the batter and pitcher handedness. Letting x_i and z_i denote the PITCHf/x coordinates of this pitch, we take fu(xi,zi)=hi⊤Θ0u+LOi⊤ΘLOu, where ΘLOu is a vector of length four recording the partial effect of location and Θ0u is a vector of length four containing an intercept term, one for each combination of batter and pitcher handedness. Finally, let u(i) denote which umpire called this pitch. To place all of the variables in our model on roughly similar scales, we first re-scale the corresponding historical GAM estimates for each combination of batter and pitcher handedness to have standard deviation 1.

Finally let CO_i, CA_i, P_i and B_i be vectors encoding the count, catcher, pitcher, and batter involved with this pitch, and let ΘCOu,ΘCAu,ΘPu, and ΘBu be vectors containing the partial effect of count, catcher, pitcher, and batter on umpire u. For identifiability, we specify a single catcher, Brayan Pena, and count, 0–0, as baseline values. We can re-write the model from Equation 1 as

We are now ready to present several simplifications of this general model of gradually increasing complexity. We begin first by assuming that the players and count have no effect on the log-odds of a called strike (i.e. that ΘCOu,ΘCAu,ΘPu, and ΘBu are all equal to the zero vector for each umpire). This model, hereafter referred to as Model 1, assumes that the only relevant predictor of an umpire’s ball/strike decision is the pitch location but allows for umpire-to-umpire heterogeneity. We model, a priori,

The vector μ_LO is taken to be the vector of standard deviations of the hGAM forecast for each combination of batter and pitcher handedness. In this way, Model 1 centers the prior distribution of the log-odds of a strike at the hGAM forecasted log-odds. We may interpret the parameters τ02 and τLO2 as capturing the umpire-to-umpire variability in the intercept and location effects and we may view Θ₀ and Θ_LO as the mean intercept and location effects averaged over all umpires. By placing a further level of prior hierarchy on Θ₀ and Θ_LO, we render the Θ0u’s and ΘLOu’s dependent, both a priori and a posteriori. In this way, while we are fitting a separate model for each umpire, these models are “mutually informative” in the sense that the estimate of umpire u’s intercept vector Θ0u will, for instance, be “shrunk” towards the average of all umpires’ intercept vectors by an amount controlled by τ02 and τLO2. Further priors on the hyper-parameters σ02 and σLO2 introduce dependence between the components of Θ0u and ΘLOu as well, enabling us to “borrow strength” between the four combination of batter and pitcher handedness.

While Model 1 essentially estimates a separate called strike probability surface for each umpire, it entirely precludes the possibility of player or count effects. We now consider two successive expansions of Model 1. In Model 2, we incorporate both catcher and count effects that are assumed to be constant across umpires. That is, in Model 2 we assume that all of the the ΘCOu’s are all equal to some common value Θ_CO and all of the ΘCAu’s are equal to some common value Θ_CA. Similarly, in Model 3 we augment Model 2 with constant pitcher effects and constant batter effects. A priori, we model

and consider similar, zero-mean spherically-symmetric Gaussian priors for Θ_CA, Θ_P and Θ_B, while retaining the same prior specification on the Θ0u’s and ΘLOu’s. Though they elaborate on Model 1, Models 2 and 3 still represent a vast simplification to the general model in Equation 1 as they assume that there is no umpire-to-umpire variability in the count or player effects. This leads us to consider Model 4, which builds on Model 2 by allowing umpire-specific count and catcher effects, and Model 5, which includes umpire-specific batter and pitcher effects and corresponds to the general model in Equation 1 We model

and consider similarly structured prior hierarchies for ΘCAu,ΘBu,ΘPu in Models 4 and 5. Throughout, we place independent Inverse Gamma(3,3) hyper-priors on the top-level variance parameters σ02,σLO2,σCO2,σCA2,σP2 and σB2.

It remains to specify the hyper-parameters τ02,τLO2,τCO2,τCA2,τP2 and τB2 which capture the umpire-to-umpire variability in the intercept, location, count, and player effects. For simplicity, we fix these hyper-parameters to be equal to 0.25 in the appropriate models. To motivate this choice, consider how two umpires would call a pitch thrown at a location where the historical GAM forecasts a 50% called strike probability. According to Model 2, the difference in the two umpires’ log-odds of a called strike follows a N(0,2(τ02+τCO2+τCA2)) distribution, a priori. Taking τ02=τCO2=τCA2=0.25 reflects a prior belief that there is less than a 10% chance that one umpire would call a strike 75% of the time while the other calls it a strike only 25% of the time. For simplicity, we take τLO2=τB2=τP2=0.25 as well.

3.1 Predictive performance

We fit each model in Stan (Carpenter et al. 2016) and ran two MCMC chains for each model. All computations were done in R (versions 3.3.2 and later) and the MCMC simulation was carried out in RStan (versions 2.14.1 and later) on a high-performance computing cluster. For each model, after burning-in the first 2000 iterations, the Gelman-Rubin R^ statistic for each parameter was <1.1, suggesting convergence. We continued to run the chains, after this burn-in, until each parameter’s effective sample size exceeded 1000. For Models 1 and 2, we found that running the sampler for 4000 total iterations was sufficient while for Models 3, 4, and 5, we needed 6000 iterations. The run time of these samplers ranged from just under an hour (Model 1) to 50 h (Model 5).

Figure 5:

Comparative box plots of 30 catcher effects sorted by the posterior mean of their partial effect on the log-odds scale.

Using the simulated posterior draws from each model, we can approximate the mean of the posterior predictive distribution of the called strike probability for each pitch in our 2014 dataset. Table 1 shows the misclassification and mean square error for Models 1–5 over all pitches from 2014 and over two separate regions, as well as the error for the historical GAM forecasts. Region 1 consists of all pitches thrown within 1.45 inches on either side of the boundary of the average rule book strike zone defined in Section 2.1. Since the radius of the ball is about 1.45 inches, the pitches in Region 1 are all “borderline” calls in the sense that only part of the ball passes through the strike zone but are, by rule, strikes. Region 2 consists of all pitches thrown between 1.45 and 2.9 inches outside the boundary of the average rule book strike zone. These pitches miss the strike zone by an amount between one and two ball’s width, and ought to be called balls by the umpire. To compute misclassification error, we used 0.5 as the threshold for a strike.

We see that Models 1–5 outperform the historical GAMs overall and in both Regions 1 and 2. This is hardly surprising, given that the hGAMs were trained on data from 2011 to 2013 and the other models were trained on the 2014 data. Recall that Model 1 only accounted for pitch location. As we successively incorporating count and catcher (Model 2), and pitcher and batter (Model 3) effects, we find that the overall error drops. Finally, Model 5 has the best performance across the board. This is entirely expected as Model 5 given the tremendous number of parameters.

Of course, we would be remiss if we assessed predictive performance only with training data. Table 2 compares such out-of-sample predictive performance, by considering pitches from the 2015 season for which the associated batter, catcher, pitcher, and umpire all appeared in our 2014 dataset.

Now we see that Model 3 has the best out-of-sample performance overall and in Regions 1 and 2. The fact that Models 4 and 5 have worse out-of-sample performance, despite having very good in-sample performance is a clear indication that these two over-parametrized models have overfit the data. One could argue, however, that comparing predictive performance on 2015 data is not the best means of diagnosing overfitting. Roegelle (2014), Mills (2017a), and Mills (2017b) have documented year-to-year changes in umpires’ strike zone enforcement ever since Major League Baseball began reviewing and grading umpires’ decisions in 2009. In Appendix B, we report the results from a cross-validation study, in which we repeatedly re-fit Models 1–5 using 90% of the 2014 data and assessing performance on the remaining 10%, that similarly demonstrates Model 3’s superiority.

Model 3’s superiority over Models 1 and 5 reveals that although accounting for player effects can lead to improved predictions of called strike probabilities, we cannot reliably estimate an individual catchers catcher effects on individual umpires with a single season’s worth of data.

3.2 Full posterior analysis

We now examine the posterior samples from Model 3 more carefully. Figure 5 shows box plots of the posterior distributions of catcher effects on the log-odds scale for the catchers with the top 10 posterior means, the bottom 10 posterior means, and the middle 10 posterior means.

Figure 6:

Fifty percent and 90% contours for called strike probability for the Bumgarner-Puig-Posey matchup for different umpires in different counts.

We see that there are some catchers, like Hank Conger and Buster Posey, whose posterior distributions are entirely supported on the positive axis, indicating that, all else being equal, umpires are more likely to call strikes when they are caught by these catchers as opposed to the baseline catcher, Brayan Pena. On the other extreme, there are some catchers like Tomas Tellis with distinctly negative effects. As we would expect, catchers who appeared very infrequently in our dataset have very wide posterior distributions. For instance, Austin Romine caught only 61 called pitches and his partial effect has the largest posterior variance among all catchers. It is interesting to see that all of the catcher effects, on the log-odds scale, are contained in the interval [−1.5, 1.5], despite the prior placing nearly 20% of its probability outside this interval. The maximum difference on the log-odds scale between the partial effects of any two catchers is 3, with high posterior probability. For context, a change of 3 in log-odds corresponds to a change in probability from 18.24% to 81.76%. As it turns out, the posterior distribution of each count effect is also almost entirely supported in the interval [−1.5, 1.5], on the log-odds scale. This would seem to suggest that catcher framing effects are comparable in magnitude to the effect of count. We explore this possibility in much greater detail in Appendix C.

Armed with our simulated posterior draws, we can create posterior predictive strike zones for a given batter-pitcher-catcher-umpire matchup. Suppose, for instance, that Madison Bumgarner is pitching to the batter Yasiel Puig, with Buster Posey catching. Figure 6 shows the 50% and 90% contours of the posterior predictive called strike probability for two umpires, Angel Hernandez and Mike DiMuro, and an average umpire in a 2–0 and 0–2 count. Note, if the center of the pitch passes within the region bounded by the dashed gray line in the figure, then some part of the ball passes through the average rule book strike zone, shown in gray. Puig is a right-handed batter, meaning that he stands on the left-hand side of the approximate rule book strike zone, from the umpire’s perspective.

Across the board, Hernandez’s contours enclose more area than the average umpire’s contours and DiMuro’s contours enclose less area than the average umpire’s. For instance, on a 2–0 count, Hernandez’s 50% contour covers 4.37 sq. ft., the average umpire’s covers 3.87 sq. ft., and DiMuro’s covers 3.53 sq. ft. The contours on a 0–2 pitch are much smaller, indicating that all else being equal, these umpires are less likely to call a strike on an 0–2 pitch than on a 2–0 pitch. Each of the 50% contours extend several inches to the left or inside edge of the approximate rule book strike zone. At the same time, the same contours do not extend nearly as far beyond the right or outside edge of the strike zone. This means that, Hernandez, DiMuro, and the average umpires are more likely to call strikes on pitches that miss the inside edge of the strike zone than they are on pitches that miss the outside edge by the same amount. Even the 90% contour on a 2–0 count extend a few inches past the inside edge of the strike zone, implying that Hernandez, DiMuro, and the average umpire will almost always call a strike that misses the inside edge of a the strike zone so long as it is not too high or low. Interestingly, we see that the leftmost extents of DiMuro’s and the average umpire’s 90% contours on a 0–2 pitch nearly align with the dashed boundary on the inside edge. A pitch thrown at this location will barely cross the average rule book strike zone, indicating that at least on the inside edge, DiMuro and umpires on average tend to follow the rule book prescription, calling strikes over 90% of the time.

The same is not true at the top, bottom, or outside edge. For instance, the rightmost extent of average umpires’ 90% contour on a 0–2 pitch lies several inches within the outside edge of the strike zone. So in the space of about four and a half inches, the average umpires’ called strike probability drops dramatically from 90% to 50%, despite the fact that according to the rule book these pitches should be called a strike.

Figure 6 is largely consistent with the empirical observation that Hernandez tends to call a much more permissive strike zone than DiMuro: Hernandez called 42.67% of taken pitches strikes (1624 strikes to 2182 balls) and DiMuro called 39.92% of taken pitches strikes (1220 strikes to 1836 balls). On 2–0 pitches, Hernandez’s strike rate increased to 51.44% (71 strikes to 67 balls) and DiMuro’s increased to 48.31% (57 strikes to 67 balls).

4.1 Catcher aggregate framing effect

Looking at Table 4, it is tempting to say that Miguel Montero is the best framer. After all, he is estimated to have saved the most expected runs relative to the baseline catcher. We observe, however, that Montero received 8086 called pitches while Conger received only 4743. How much of the difference in the estimated number of runs saved is due to their framing ability and how much to the disparity in the called pitches they received?

A naive solution is to re-scale the RS estimates and compare the average number of runs saved on a per-pitch basis. While this accounts for the differences in number of pitches received, it does not address the fact that Montero appeared with different players than Conger and that the spatial distribution of pitches he received is not identical to that of Conger. In other words, even if we convert the results of Table 4 to a per-pitch basis, the results would still be confounded by pitch location, count, and pitch participants.

To overcome this dependence, we propose to integrate f(ca, ξ) over all ξ rather than summing f(ca, ξ) over 𝒫ca. Such a calculation is similar to the spatially aggregate fielding evaluation (SAFE) of Jensen, Shirley, and Wyner (2009). They integrated the average number of runs saved by a player successfully fielding a ball put in play against the estimated density of location and velocity of these balls to derive an overall fielding metric un-confounded by dispraise in players’ fielding opportunities. We propose to integrate f(ca, ξ) against the empirical distribution of ξ and define catcher ca’s “Catcher Aggregate Framing Effect” or CAFE to be

The sum in Equation 2 may be viewed as the number of expected runs catcher ca saves relative to the baseline if he participated in every pitch in our dataset. We then re-scale this quantity to reflect the impact of his framing on 4000 “average” pitches. We opted to re-scale by CAFE by 4000 as the average number of called pitches received by catchers who appeared in more than 25 games was just over 3992. Of course, we could have easily re-scaled by a different amount.

Once again, we can use our simulated posterior samples of the Θ^u’s to simulate draws from the posterior distribution of CAFE. Table 5 shows the top and bottom 10 catchers ranked according to the posterior mean of their CAFE value, along with the posterior standard deviation, and 95% credible interval for their CAFE value. Also shown is the a 95% interval of each catchers marginal rank according to CAFE.

We see that several of the catchers from Table 4 also appear in Table 5. The new additions to the top ten, Christian Vazquez, Martin Maldonado, Chris Stewart, and Francisco Cervelli were ranked 13th, 17th, 18th and 19th according to the RS metric. The fact that they rose so much in the rankings when we integrated over all ξ indicates that their original rankings were driven primarily by the fact that they all received considerably fewer pitches in the 2014 season than the top 10 catchers in Table 4. In particular, Vazquez received 3198 called pitches, Cervelli received 2424, Stewart received 2370, and Maldonado received only 1861.

Interestingly, we see that now Hank Conger ranks ahead of Miguel Montero according to the posterior mean CAFE, indicating that the relative rankings in Table 4 was driven at least partially by disparities in the pitches the two received than by differences in their framing effects. Though Conger emerges as a slightly better framer than Montero in terms of CAFE, the difference between the two is small, as evidenced by the considerable overlap in their 95% posterior credible intervals.

We find that in 95% of the posterior samples, Conger had anywhere between the largest and 11th largest CAFE. In contrast, we see that in 95% of our posterior samples, Tomas Tellis’s CAFE was among the bottom 3 CAFE values. Interestingly, we find much wider credible intervals for the marginal ranks among the bottom 10 catchers. Some catchers like Koyie Hill and Austin Romine appeared very infrequently in our dataset. To wit, Hill received only 409 called pitches and Romine received only 61. As we might expect, there is considerable uncertainty in our estimate about their framing impact, as indicated by the rather wide credible intervals of their marginal rank.

4.2 Year-to-year reliability of CAFE

We now consider how consistent CAFE is over multiple seasons. We re-fit our model using data from the 2012 to 2015 seasons. For each season, we restrict attention to those pitches within one foot of the approximate rule book strike zone from that season. We also use the log-odds from the GAM models trained on all previous seasons so that the model fit to the 2012 data uses GAM forecasts trained only on data from 2011 while the model fit to the 2015 data uses GAM forecasts trained only on data from 2011 to 2014. When computing the values of CAFE, we use the run values given in Table 3 for each season. There were a total of 56 catchers who appeared in all four of these seasons. Table 6 shows the correlation between their CAFE values over time.

In light of the non-stationarity in strike zone enforcement across seasons, it is encouraging to find moderate to high correlation between a player’s CAFE in one season and the next. In terms of year-to-year reliability, the autocorrelations of 0.5–0.7 place CAFE on par with slugging percentage for batters. Interestingly, the correlations between 2012 CAFE and 2013 CAFE and the correlation between 2013 CAFE and 2014 CAFE are >0.7, but the correlation between 2014 CAFE and 2015 CAFE is somewhat lower, 0.58. While this could just be an artifact of noise, we do note that there was a marked uptick in awareness of framing between the 2014 and 2015 seasons, especially among fans and in the popular press. One possible reason for the drop in correlation might be umpires responding to certain catcher’s reputations as elite pitch framers by calling stricter strike zones, a possibility suggested by Sullivan (2016).

5.1 Extensions

There are several extensions of and improvements to our model that we now discuss. While we have not done so here, one may derive analogous estimates of RS and CAFE for batters and pitchers in a straightforward manner. Our model only considered the count into which a pitch was thrown but there is much more contextual information that we could have included. For instance, Rosales and Spratt (2015) have suggested the distance between where a catcher actually receives the pitch and where he sets up his glove before the pitch is thrown could influence an umpire’s ball-strike decision making. Such glove tracking data is proprietary but should it be become publicly available, one could include this distance along with its interaction with the catcher indicator into our model. In addition, one could extend our model to include additional game-state information such as the ball park, the number of outs in the half-inning, the configuration of the base-runners, whether or not the home team is batting, and the number of pitches thrown so far in the at-bat. One may argue that umpires tend to call more strikes late in games which are virtually decided (e.g. when the home team leads by 10 runs in the top of the ninth inning) and easily include measures related to the run-differential and time remaining into our model. Expanding our model in these directions may improve the overall predictive performance slightly without dramatically increasing the computational overhead.

More substantively, we have treated the umpires’ calls as independent events throughout this paper. Chen et al. (2016) reported a negative correlation in consecutive calls, after adjusting for location. To account for this negative correlation in consecutive calls, we could augment our model with binary predictors encoding the results of the umpires’ previous k calls in the same at-bat, inning, or game. Incorporating this Markov structure to our model would almost certainly improve the overall estimation of called strike probability and may produce slightly smaller estimates of RS and CAFE. At this point, however, it is not a priori obvious how large the differences would be or how best to pick k. It is also well-known that pitchers try to throw to different locations based on the count, but we make no attempt to model or exploit this phenomenon. Understanding the effect of pitch sequencing on umpires’ decision making (and vice-versa) would also be an interesting line of future research.

We incorporated pitch location in a two-step procedure: we started from an already quite good generalized additive model trained with historical data and used the forecasted log-odds of a called strike as a predictor in our logistic regression model. Much more elegant would have been to fit a single semi-parametric model by placing, say, a common Gaussian process prior on the umpire-specific functions of pitch location, f^u(x, z) in Equation 1. We have also not investigated any potential interactions between pitch location, player, and count effects. While we could certainly add interaction terms to the logistic models considered above, doing so vastly increases the number of parameters and may require more thoughtful prior regularization. A more elegant alternative would be to fit a Bayesian “sum-of-trees” model using Chipman, George, and McCulloch (2010)’s BART procedure. Such a model would likely result in more accurate called strike probabilities as it naturally incorporates interaction structure. We suspect that this approach might reveal certain locations and counts in which framing is most manifest.

Finally, we return to the two pitches from the 2015 American League Wild Card game in Figure 1. Fitting our model to the 2015 data, we find that Eric Cooper was indeed much more likely to call the Keuchel pitch a strike than the Tanaka pitch (81.72% vs 62.59%). Interestingly, the forecasts from the hGAMs underpinning our model were 51.31% and 50.29%, respectively. Looking a bit further, had both catchers been replaced by the baseline catcher, our model estimates a called strike probability of 77.58% for the Keuchel pitch and 61.29% for the Tanaka pitch, indicating that Astros’ catcher Jason Castro’s apparent framing effect (4.14%) was slightly larger than Yankee’s catcher Brian McCann’s (1.30%). The rather large discrepancy between the apparent framing effects and the estimated called strike probabilities reveals that we cannot immediately attribute the difference in calls on these pitches solely to differences in the framing abilities of the catchers. Indeed, we note that the two pitches were thrown in different counts: Keuchel’s pitch was thrown in a 1–0 count and Tanaka’s was thrown in a 1–1 count. In 2015, umpires were much more likely to call strikes in a 1–0 count than they were in a 1–1 count, all else being equal. Interestingly, had the Keuchel and Tanaka pitches been thrown in the same count, our model still estimates that Cooper would be consistently more likely to call the Keuchel pitch a strike, lending some credence to disappointed Yankees’ fans’ claims that his strike zone enforcement favored the Astros. Ultimately, though, it is not so clear that the differences in calls on the two pitches shown in Figure 1 specifically was driven by catcher framing as much as it was driven by random chance.