New metrics for evaluating home plate umpire consistency and accuracy

2.1 Rectangular metrics

A natural definition for the established strike zone is the smallest rectangular region containing all the strikes. Figure 2 shows an example of these established strike zones for left- and right-handed batters for a particular MLB game. Any called ball inside these rectangles is inconsistent. For a given game, we define the one-rectangle inconsistency index I_R1 as

Figure 2:

The smallest rectangle containing all the strikes. Balls are drawn as blue circles, and strikes as red diamonds. Note that the center of the circle or diamond must lie on or inside the rectangle to be considered inside the rectangular region.

In the game shown in Figure 2, out of 110 called balls, 2 were inconsistent to left-handed batters and 13 were inconsistent to right-handed batters, so I_R1 = (2 + 13)/110 ≈ 0.136.

While the one-rectangle inconsistency index is natural to define and easy to compute, it is highly sensitive to a single outlying strike call. For example, in the right-handed plot in Figure 2, if we remove the lowest strike at (0.26, 1.27), the lower border of the established strike zone moves up to the next-lowest strike, eliminating three inconsistent balls. Had this single low strike been called a ball, the index I_R1 would have been (2 + 10)/110 ≈ 0.109 instead of (2 + 13)/110 ≈ 0.136.

Another weakness of the one-rectangle index is that it can fail to account for multiple bad strike calls in the same location. Again using the right-handed plot in Figure 2, we see that eliminating the strike at (−0.93, 2.94) has no effect on I_R1, because the resulting rectangle will still enclose the same number of called balls. While this strike call seems inconsistent given the five called balls around px = −0.6, the measure I_R1 fails to reflect this inconsistency.

We can mitigate these limitations in the one-rectangle inconsistency index by using more rectangles. As with the definition of I_R1, the first rectangular region is the smallest one that contains all the strikes; that is, it is the rectangle determined by the smallest and largest values of both px and pz for the set of called strikes. The second rectangle is then determined by the second-smallest and second-largest values of these coordinates. Continuing in this manner, taking the ith smallest and ith largest coordinates, we can form rectangles R1,R2,…,Rn for both left- and right-handed hitters, for some choice of n. (Once i becomes large enough to exhaust all of the called strikes, take R_i to be the empty set.) Let s(i) be the number of called balls inside the two R_i’s. Define the n-rectangle inconsistency index as

The rectangles used to calculate I_R10 are shown in Figure 3. Versus left-handed batters, rectangle R₁ is the only rectangle containing called balls. However, versus right-handed batters, R₁ contains 13 called balls (as above), R₂ contains 11, R₃ contains 7, R₄ contains 1, and the remaining rectangles contain none. Therefore,

Figure 3:

Successive rectangles enclose inconsistent balls. The more inconsistent called balls lie within more rectangles.

Notice that, in this example, I_R10 = I_R9 = ⋯ = I_R4, illustrating that the value of this index eventually stabilizes, given enough rectangles. In practice, 10 rectangles is plenty for most MLB game data sets.

Since R_i+1 ⊆ R_i, inconsistent balls are weighted according to how many rectangles they are contained in. Balls that are inconsistent due to a single outlying strike will only have weight 1, while egregiously bad ball calls will lie inside several rectangles, increasing their contribution to I_Rn. Therefore, compared to the one-rectangle index, the n-rectangle index is less sensitive to a single outlying strike call.

Furthermore, when several bad strike calls lie in the same location, the n-rectangle index will reflect this inconsistency, since the successive rectangles will not shrink until these strike calls are exhausted. In Figure 3, the called balls around px = −0.6 are weighted more heavily because of the strike at (−0.93, 2.94), in contrast to the situation in Figure 2, where we saw that this strike had no effect on the one-rectangle index.

2.2 Convex hull metrics

Since rectangles are used to construct the inconsistency index I_Rn, it measures inconsistency under the assumption (stipulated in the rules of baseball) that the true strike zone is a rectangle. However, as we will see in Section 3, strike zones in practice tend to be rounded at the corners. In this section we will introduce inconsistency measures that relax the assumption of a rectangular zone. Instead, we assume that a consistent zone will have the property that any pitch landing between two called strikes will also be a called strike. In other words, we assume that the established strike zone is convex.

Given a discrete set P⊆ℝ2 representing the locations of called strikes during a game, there is a natural geometric definition for the established strike zone, namely, the convex hull of P. We can define the convex hull S as the intersection of all closed half planes that contain P:

where Hlc denotes the complement of the open half-plane bounded by the line l.

Using the convex hull as our established strike zone, we can define the convex hull inconsistency index I_CH analogously to the one-rectangle inconsistency index. Now an inconsistent ball is one that lies within the convex hull of strikes, and I_CH is given by

For example, see Figure 4. There were five inconsistent balls versus left-handed batters, and one versus right handed batters, out of a total of 118 called balls. Therefore I_CH = (5 + 1)/118 ≈ 0.051.

Like the one-rectangle index, the convex hull inconsistency index can fail to account for multiple bad strikes in the same location. It can also be unaffected by outlying strikes, depending on their location. For example, in Figure 4 versus right-handed batters, the strike at (−0.01, 1.31) has no effect on I_CH; removing this point would shrink the convex hull without changing the number of called balls enclosed. However, this call seems inconsistent, given its proximity to several called balls. The problem is that a vertex of the convex hull can lie in a region populated by called balls, yet fail to enclose any. Creating smaller convex hulls inside the first (as we did to define I_Rn) will not address this issue.

To account for this phenomenon, we can use the locations of called balls to define a called-ball region. Instead of counting called balls within the established strike zone, we can measure the area of the overlap between the called-ball region and the convex hull of strikes.

Figure 4:

The established strike zone is the convex hull of called strikes.

Unlike the established strike zone, the called-ball region will typically not be convex, or even simply connected. Given a set Q⊆ℝ2 representing the locations of called balls during a game, and given some radius α > 0, define

where Bx,αc denotes the complement in the plane of the open disk of radius α centered at the point x. The region X, which will serve as our called-ball region, is called the α-convex hull of Q (Pateiro-López and Rodrıguez-Casal 2010). Note that the α-convex hull is not convex, in general.

Let a_L and a_R be the areas of the intersection of the convex hull of called strikes and the α-convex hull of called balls, for left-handed and right-handed batters, respectively. We define the α-convex hull inconsistency index I_ACH to be a weighted average of these two areas. Let n_L be the number of called pitches thrown to left-handed batters, and let n_R be the number of called pitches to right-handed batters. Then

Figure 5 shows the ball and strike calls for the same game as Figure 4, along with the α-convex hull of called balls, using α = 0.7. In this case, I_ACH = 0.127. Notice that the called strike at (−0.01, 1.31) now has a significant effect on I_ACH, since it causes a large region of overlap between the convex hull and the α-convex hull, which contains the nearby called balls.

Figure 5:

The established strike zone is the convex hull of called strikes (in red), and the called-ball region is the α-convex hull of balls (in blue), where α = 0.7.

One limitation to this choice of inconsistency metric is that there is no canonical choice for the constant α. Figure 6 illustrates the issues involved. If the radius α is too small, the α-convex hull will contain isolated points and small disconnected regions. Large values of α (such as α = 0.9 in this example) will produce a single, simply-connected α-convex hull, making the called-ball region completely cover the established strike zone. Generally speaking, the larger the value of α, the tougher the metric I_ACH is on the umpires.

Figure 6:

Six different called-ball regions (α-convex hulls) for different choices of α.

In the analysis that follows, we have chosen to use α = 0.7, based largely on qualitative inspections of various game examples, as in Figure 6. A correlation analysis can lend some empirical support to this choice. Table 1 gives the pairwise correlations for I_ACH computed over all 2017 regular season games using six different values of α between 0.4 and 0.9, in increments of 0.1. For these six values, the greatest correlation is between α = 0.6 and α = 0.7, and we observe that once α exceeds 0.7, the correlations between adjacent values begin to decrease. These results confirm the observation that choosing α in the range 0.6 ≤ α ≤ 0.7 tends to give similar measures of I_ACH.

2.3 Statistical properties of the inconsistency metrics

All four inconsistency measures are sensitive to a single outlying called strike, and the n-rectangle and α-convex hull indices are sensitive to an egregiously bad called ball in the middle of the strike zone. This sensitivity is by design, to avoid penalizing umpires who make slightly inconsistent calls as much as those who make clearly bad calls. However, a consequence of this feature is that these metrics are also sensitive to the number of pitches called. As the number of called pitches increases, the chances that an umpire will make an egregious call increases, and once such a call is made, the inconsistency index will remain high.

Figure 7 investigates the association between number of pitches called and inconsistency index. The top row shows scatter plots of the four indices versus number of pitches called in the game, for all regular-season games with between 50 and 300 called pitches. (Only 2 of 2425 games in our sample fall outside this range.) For each index, there is a slight discernible upward trend. The correlation coefficient r is approximately 0.2 in all four cases.

Figure 7:

Scatterplots of the four inconsistency indices, I_R1, I_R10, I_CH, and I_ACH, versus the number of called pitches in the game. The top row shows the data for all games with between 50 and 300 called pitches, while the bottom row includes only games with high inconsistency indices. The blue curves show the smoothed conditional means.

The smoothed density estimates in Figure 8 show that the distributions of I_R1, I_R10, I_CH, and I_ACH are all skewed right. Such skewness may be a feature of the metrics, or it may indicate that major league umpires are, on the whole, very good at calling games consistently. To assess sensitivity in the tails of these distributions, the second row of scatter plots in Figure 7 considers only “high-inconsistency games,” where I_R10 + I_ACH > 0.3. For these games, and for this range of pitches called, there does not appear to be a strong association between the inconsistency measures and the number of called pitches (|r| < 0.1 in all cases).

Figure 8:

Smoothed density estimates of the four inconsistency indices computed on 2423 games.

In this section we have considered two simple metrics, I_R1 and I_CH, along with extensions I_R10 and I_ACH, respectively, which attempt to address deficiencies in the simple metrics. Figure 8 shows that the tails of the distributions of the extended metrics are substantially thicker, suggesting that these metrics are better at differentiating between higher levels of inconsistency.

The pairwise correlation coefficients for the four metrics are given in Table 2. The two extended metrics I_R10 and I_ACH are correlated, but not very strongly, indicating that they measure different aspects of inconsistency. Some of the difference may be due to the shape of the strike zone that an umpire tends to call. For example, among the 79 umpires who called at least 20 games behind home plate in 2017, Chad Whitson was the 17th most consistent umpire when measured using I_ACH, but ranked 51st when measured using I_R10. The methods that we will present in Section 3.1 reveal that Whitson’s zone tends to be quite rounded at the corners, rather than rectangular, so it is not surprising that he ranks higher when judged using the convex hull metrics. By comparison, Pat Hoberg, whose zone is somewhat less rounded, has the 6th best I_R10, but ranks 34th according to I_ACH. See Figure 9.

Figure 9:

Zone tendencies for Chad Whitson and Pat Hoberg. Whitson is more consistent according to I_ACH, while Hoberg is more consistent according to I_R10. The method for constructing these contours is discussed in Section 3.1.

As a compromise, in some of our analysis, we use the sum I_R10 + I_ACH as a general measure of inconsistency. Notice that I_R10 typically takes larger values than I_ACH, so the 10-rectangle metric is effectively weighted more than the α-convex hull metric in this sum. Over all 2423 games, the mean of the sum I_R10 + I_ACH is 0.191 with standard deviation 0.142, and its median is 0.156.

3.1 Kernel density estimation

Let s(x, y) be the two-dimensional probability density function describing the distribution of called strikes in the plane. That is, s(x, y) gives the density of the probability that a called pitch will cross the plate at location (x, y), given that the pitch is called a strike. In order to describe the consensus zone, we would like to compute the reverse conditional probability, that is, the probability density f(x, y) that a called pitch will be called a strike, given that it crosses that plate at location (x, y). Let s^(x,y) be a two-dimensional kernel density estimate computed on the (px, pz) coordinates of called strikes, and let c^(x,y) be a two-dimensional kernel density estimate computed on the coordinates of all called pitches. Then by Bayes’ theorem, an estimate for f(x, y) is given by

where p^ is the proportion of called pitches that are strikes. The 50% contour of f^(x,y) will then be the border of the consensus zone.

Figure 10 shows the smooth contours produced using this method, along with the discrete approximation of the method described in (Roegele 2018). In addition to improving the resolution of the zone boundary, the kernel density estimation method will work well for smaller samples of pitches. In particular, the kernel density estimation method can produce 50% contours for each MLB umpire’s calls over the course of a season, which can be used to describe season-long tendencies.

Figure 10:

The smooth curves are the boundaries of the consensus zones computed using kernel density estimation. The shaded squares show the result of computing the consensus zone using the discrete method of (Roegele 2018).

For example, let X be the region in the plane bounded by the 50% contour for a particular umpire, and let C be the consensus zone described above. The symmetric difference (X∖C) ∪ (C∖X) is the set of all points lying in one zone and not in the other, and its area D_S measures the extent to which the umpire’s zone deviates from the consensus zone. To illustrate the extent to which zones can conform to the consensus zone, Figure 11 shows the contour zones for the umpires with the greatest and least values of D_S for the 2017 season, along with the consensus zone.

Figure 11:

Measured by symmetric difference with the consensus zone (in gray), Stu Scheurwater had the most conforming zone, while Rob Drake’s zone was the least conforming.

3.2 Contour-based zone accuracy and size

Given any pair of closed curves 𝒵l,𝒵r representing the boundaries of strike zones versus left- and right-handed batters, and any set of called pitches, let A𝒵 denote the proportion of correctly-called pitches, based on the strike zones 𝒵l and 𝒵r. Let C_l and C_r be the 2017 consensus zones computed using the kernel density estimation method, and let R = R_l = R_r be the rule-book rectangle described above. For each MLB umpire who called at least 20 games behind home plate in 2017, we compute the umpire’s consensus accuracy A_C and rule-book accuracy A_R.

For each umpire, the individual contour zones yield a convenient measure of an umpire’s zone size S. It has been suggested (Roegele 2017) that accuracy measurements can function as a proxy for zone size, since inaccurate umpires would tend to have larger strike zones. Our data and measurements do not provide evidence for this assertion. Figure 12 illustrates the associations between these two measures of accuracy, A_C and A_R, along with zone size S. The accuracy measures A_C and A_R are only moderately correlated, and neither is strongly associated with zone size S. The correlation between A_C and S is even weaker if the influential observation with the largest zone (Doug Eddings) is removed.

Figure 12:

Pairwise correlations of A_C, A_R, and S, for each full-time umpire over the 2017 season. The curves down the diagonal show the distributions of each measure. The scatterplots in the bottom row indicate that measures of accuracy are not strongly associated with zone size.

4.1 Correlation and outliers

Ideally, any new metric should evaluate phenomena that previous metrics ignore, since strongly associated measures can yield redundant information. However, it is reasonable to expect that the best umpires are the best at several aspects of the job, so there are likely to be associations among different performance measurements. Figure 13 summarizes the pairwise correlation coefficients for the above measures on all MLB umpires who called at least 20 games behind the plate in the 2017 season. The four inconsistency measures I_R1, I_R10, I_CH and I_ACH have been averaged over all the games called by each umpire. Notice that these inconsistency indices are correlated positively with each other and negatively with the two accuracy measures A_C and A_R. The other measures considered, symmetric-difference nonconformity D_S, zone size S, walk rate r_W, and strikeout rate r_K, generally do not show strong associations with the inconsistency and accuracy measures.

Figure 13:

Pairwise correlations for season averages of I_R1, I_R10, I_CH, I_ACH, along with A_C, A_R, D_S, S, r_W, r_K, for MLB umpires with at least 20 games called in 2017.

The association between accuracy and inconsistency is not surprising, but it is not strong. In particular, for these umpires, average I_R10 + I_ACH and A_C have a correlation coefficient of r = −0.58. The scatterplot in Figure 14 illustrates the negative relationship between consensus zone accuracy A_C and season-average inconsistency, measured as the sum I_R10 + I_ACH of the 10-rectangle and α-convex hull indices. Notable in this graph are the outliers. For example, Carlos Torres, Tim Timmons, and Cory Blaser stand out has having above-average consistency but only average accuracy, suggesting that they would be underrated if evaluated on the basis of accuracy alone.

Figure 14:

Scatterplot of consensus accuracy A_C versus average inconsistency I_R10 + I_ACH.

4.2 Principal component analysis

Any single rating system for umpires can produce a ranking of umpires. When combining several different metrics for umpire performance, we can organize the information that accounts for most of the variation between umpires by examining principal components.

Table 3 shows the coefficients for a principal component analysis (Mardia, Kent, and Bibby 1980) of the season-average inconsistency indices I_R1, I_R10, I_CH, I_ACH, along with consensus accuracy A_C, rule book accuracy A_R, zone size S, walk rate r_W, and strikeout rate r_K, each normalized to have unit variance. The first two components, PC1 and PC2, account for 68% of the variation in the data. Component PC1 is dominated by the accuracy measures (positive) and inconsistency measures (negative), so it seems appropriate to designate it as “strike zone quality.” Meanwhile, component PC2 is dominated by walk rate (negative) and strikeout rate and zone size (positive), so this component measures “pitcher friendliness.”

The principal components provide a way to summarize the various umpire evaluations developed above. Using the coefficients in each column of Table 3 to form linear combinations of the normalized metrics, we obtain component scores for each umpire. Figure 15 is a scatter plot of the component scores for the first two principal components, PC1 and PC2, labeled by umpire. The most consistent and accurate umpires are those furthest right, while the most neutral arbiters between pitcher and batter are found along the horizontal axis.

Figure 15:

Plot of the first two principal components. The component on the horizontal axis is dominated by accuracy and consistency, designated as “strike zone quality,” where the average quality score is zero. The vertical axis component is dominated by walk rate (negative) and strikeout rate and zone size (positive), designated as “pitcher friendliness,” with neutrality between pitcher and hitter at zero.