Meta-analytics: tools for understanding the statistical properties of sports metrics

2.1 Discrimination

For a metric measuring player ability to be applicable, it must be a useful tool for discriminating between different players. This implies that most of the variability between players reflects true variation in player ability and not chance variation or noise from small sample sizes. As a useful baseline for discrimination, we compare the average intrinsic variability of a metric to the total between player variation in this metric. Similar approaches which partially inspired our version of this metric have been used in analyzing Major League Baseball data (Tango, Lichtman and Dolphin 2007; Arthur 2015).

To characterize the discriminative power of a metric, we need to quantify the fraction of total between player variance that is due to chance and the fraction that is due to signal. By the law of total variance, the between player variance can be decomposed as

Here, V_sm[X] corresponds to the total variation in metric m between players in season s, whereas E_sm[V_spm[X]] is the average (across players) sampling variability for metric m in season s. With this decomposition in mind, we define the discriminative power of a metric m in season s as

Intuitively, this describes the fraction (between 0 and 1) of between-player variance in metric m (in season s) due to true differences in player ability. Discrimination meta-metrics for different seasons can be combined as 𝒟m=Em[𝒟sm].

It is helpful to understand the discrimination estimand for the linear mixed effects model defined in Equation 1. When this model holds, Esm[Vspm[X]]=τM2, and Vsm[X]=σPM2+σSPM2+τM2, the between-player variance (equal for all seasons s). Thus, the discrimination meta-metric under the linear mixed effects model is simply

2.2 Stability

In addition to discrimination, which is a meta-metric that describes variation within a single season, it is important to understand how much an individual player’s metric varies from season to season. The notion of stability is particularly important in sports management when making decisions about future acquisitions. For a stable metric, we have more confidence that this year’s performance will be predictive of next year’s performance. A metric can be unstable if it is particularly context dependent (e.g. the player’s performance varies significantly depending on who their teammates are) or if a player’s intrinsic skill set tends to change year to year (e.g. through offseason practice or injury).

Consequently, we define stability as a metric, which describes how much we expect a single player metric to vary over time after removing chance variability. This metric specifically targets the sensitivity of a metric to change in context or intrinsic player skill over time. Mathematically, we define stability as:

with 0 ≤ S_m ≤ 1 (see Appendix for proof). Here, V_pm[X] is the between-season variability in metric m for player p; thus, the numerator in (4) averages the between-season variability in metric m, minus sampling variance, over all players. The denominator is the total variation for metric m minus sampling variance. Again, this metric can be easily understood under the assumption of an exchangeable linear model (Equation 1).:

This estimand reflects the fraction of total variance (with sampling variability removed) that is due to within-player changes over time. If the within player variance is as large as the total variance, then S_m = 0 whereas if a metric is constant over time, then S_m = 1.

2.3 Independence

When multiple metrics measure similar aspects of a player’s ability, we should not treat these metrics as independent pieces of information. This is especially important for decision makers in sports management who use these metrics to inform decisions. Accurate assessments of player ability can only be achieved by appropriately synthesizing the available information. As such, we present a method for quantifying the dependencies between metrics that can help decision makers make sense of the growing number of data summaries.

For some advanced metrics we know their exact formula in terms of basic box score statistics, but this is not always the case. For instance, it is much more challenging to assess the relationships between new and complex model based NBA metrics like adjusted plus minus (Sill 2010), EPV-Added (Cervone et al. 2016) and counterpoints (Franks et al. 2015), which are model-based metrics that incorporate both game-log and player tracking data. Most importantly, even basic box score statistics that are not functionally related will be correlated if they measure similar aspects of intrinsic player skill (e.g., blocks and rebounds in basketball are highly correlated due to their association with height).

As such, we present a general approach for expressing dependencies among an arbitrary set of metrics measuring multiple players’ styles and abilities across multiple seasons. Specifically, we propose a Gaussian copula model in which the dependencies between metrics are expressed with a latent multivariate normal distribution. Assuming we have ℳ metrics of interest, let Z_sp be an ℳ-vector of metrics for player p during season s, and

where C is a ℳ × ℳ correlation matrix, and Fm− 1 is the inverse of the CDF for metric m. We define the independence score of a metric m given a condition set of other metrics, ℳ, as

For the latent variables Z, this corresponds to one minus the R-squared for the regression of Z_m on the latent variables Z_q with q in ℳ. Metrics for which Imℳ is small (e.g. for which the R-squared is large) provide little new information relative to the information in the set of metrics ℳ. In contrast, when Imℳ is large, the metric is nearly independent from the information contained in ℳ. Note that Imℳ=1 implies that metric m is independent from all metrics in ℳ.

We also run a principal component analysis (PCA) on C to evaluate the amount of independent information in a set of metrics. If UΛ U^T is the eigendecomposition of C, with Λ = diag(λ₁, … λ_ℳ) the diagonal matrix of eigenvalues, then we can interpret Fk=∑1kλi∑1Mλi as the fraction of total variance explained by the first k principal components (Mardia, Kent and Bibby 1980). When F_k is large for small k then there is significant redundancy in the set of metrics, and thus dimension reduction is possible.

4.1 Analysis of NBA metrics

In Figure 1 we plot the stability and discrimination meta-metrics for many of the NBA metrics available on basketball-reference.com. When computing the discrimination and stability meta-metrics, we exclude data from players with fewer than 250 minutes played in a season. This leads to 5182 player-seasons of data from the year 2000 onwards. Discrimination scores are estimated from gamelog data for the year 2015 only. There were no minutes or sample size restrictions for the independence analyses: 7507 player-seasons of data from the year 2000 onwards were used for these analyses.

Figure 1

Discrimination and stability score estimates for an ensemble of metrics and box score statistics in the NBA. Raw three point percentage is the least discriminative and stable of the metrics we study; empirical Bayes estimates of three point ability (“3PEB”, Section 5) improve both stability and discrimination. Metrics like rebounds, blocks and assists are strong indicators of player position and for this reason are highly discriminative and stable. Per-minute or per-game statistics are generally more stable but less discriminative.

For basic box score statistics, discrimination and stability scores match intuition. Metrics like rebounds, blocks and assists, which are strong indicators of player position, are highly discriminative and stable because of the relatively large between player variance. As another example, free throw percentage is a relatively non-discriminative statistic within-season but very stable over time. This makes sense because free throw shooting requires little athleticism (e.g., does not change with age or health) and is isolated from larger team strategy and personnel (e.g., teammates do not have an effect on a player’s free throw ability).

Our results also highlight the distinction between pure rate statistics (e.g., per-game or per-minute metrics) and those that incorporate total playing time. Metrics based on total minutes played are highly discriminative but less stable, whereas per-minute or per-game metrics are less discriminative but more stable. One reason for this is that injuries affect total minutes or games played in a season, but generally have less effect on per-game or per-minute metrics. This is an important observation when comparing the most reliable metrics since it is more meaningful to compare metrics of a similar type (rate-based vs total).

WS/48, ORtg, DRtg and BPM metrics are rate-based metrics whereas WS and VORP based metrics incorporate total minutes played (Sports Reference LLC 2016a). WS and VORP are more discriminative than the rate based statistics primarily because MP increases discrimination, not because there is stronger signal about player ability. Rate based metrics are more relevant for estimating player skill whereas total metrics are more relevant for identifying overall end of season contributions (e.g. for deciding the MVP). Since these classes of metrics serve different purposes, in general they should not be compared directly. Our results show moderately improved stability and discriminative power of the BPM-based metrics over other rate-based metrics like WS/48, ORTg and DRtg. Similarly, we can see that for the omnibus metrics which incorporate total minutes played, VORP is more reliable in both dimensions than total WS.

Perhaps the most striking result is the unreliability of empirical three point percentage. It is both the least stable and least discriminative of the metrics that we evaluate. Amazingly, over 50% of the variation in three point percentage between players who take a non-negligible number of three point shots (at least 10) in a given season is due to chance. This is likely because differences between three point shooters’ true shooting percentage tend to be relatively small, and as such, chance variation tends to be the dominant source of variation. Moreover, contextual variation like a team’s ability to create open shots for a player affect the stability of three point percentage.

Finally, we use independence meta-metrics to explore the dependencies between available NBA metrics. In Figure 2 we plot the independence curves described in Section 3. Of the metrics that we examine, steals (STL) appear to provide some of the most unique information. This is evidenced by the fact that the IℳSTL≈0.40 , meaning that only 60% of the variation in steals across player-seasons is explainable by the other 69 metrics. Moreover, the independence score estimate increases quickly as we reduce the size of the conditioning set, which highlights the relative lack of metrics that measure skills that correlate with steals. While the independence curves for defensive metrics are concave, the independence curves for the omnibus metrics measuring overall skill are roughly linear. Because the omnibus metrics are typically functions of many of the other metrics, they are partially correlated with many of the metrics in the conditioning set.

Figure 2

Independence score estimates as a function of the size of the conditioning set, for overall skill metrics (left) and defensive metrics (right). The curves look more linear for the overall skill metrics, which suggest that they reflect information contained in nearly all existing metrics. The first principal component from the five-by-five sub-correlation matrix consisting of the overall skill metrics, explains 73% of the variation. Defensive metrics have independence curves that are more concave. This highlights the fact that defensive metrics are correlated with a smaller set of metrics. The first principal component from the five-by-five sub-correlation matrix consisting of these defensive metrics, explains only 51% of the variation and the second explains only 73%.

Not surprisingly, there is a significant amount of redundancy across available metrics. Principal component analysis (PCA) on the full correlation matrix C suggests that we can explain over 75% of the dependencies in the data using only the first 15 out of 65 principal components, i.e., F₁₅ ≈ 0.75. Meanwhile, PCA of the sub-matrix Cℳo,ℳo where ℳo={WS, VORP, PER, BPM, PTS} yields F₁ = 0.75, that is, the first component explains 75% of the variation in these five metrics. This means that much of the information in these 5 metrics can be compressed into a single metric that reflects the same latent attributes of player skill. In contrast, for the defensive metrics presented in Figure 2, ℳd={DBPM, STL, BLK, DWS, DRtg}, PCA indicated that the first component explains only 51% of the variation. Adding a second principal component increases the total variance explained to 73%. In Figure 9 we plot the cumulative variance explained, F_k as a function of the number of components k for all metrics ℳ and the subsets ℳ_o and ℳ_d. Figure 8 illustrates a hierarchical clustering of these metrics based on these dependencies.

4.2 Analysis of NHL metrics

NHL analytics is a much younger field than NBA analytics, and as a consequence there are fewer available metrics to analyze. In Figure 3A we plot the estimated discrimination and stability scores for many of the hockey metrics available on hockey-reference.com. When computing the discrimination and stability scores, we exclude data from players with fewer than 500 minutes played in a season. This leads to 4291 player-seasons of data from the year 2000 onwards. Discrimination scores are estimated from gamelog data for the year 2015 only. There were no minutes or sample size restrictions for the independence analyses: 7270 player-seasons of data from the year 2000 onwards were used for these analyses.

Again, we find that metrics like hits (HIT), blocks (BLK) and shots (S) which are strong indicators for player type are the most discriminative and stable because of the large between-player variance.

Figure 3

(Left) Discrimination and stability scores for many NHL metrics. Corsi-based statistics are slightly more reliable than Fenwick statistics. Plus/minus is non-discriminative in hockey because of the paucity of goals scored in a typical game. (Right) Fraction of variance explained (R-squared) for each metric by a set of other metrics in our sample. Only 27% of the total variance in takeways (TK) is explained by all other NHL metrics.

Our results can be used to inform several debates in the NHL analytics community. For example, our results highlight the low discrimination of plus-minus (“±”) in hockey, which can be explained by the relative paucity of goals scored per game. For this reason, NHL analysts typically focus more on shot attempts (including shots on goal, missed shots and blocked shots). In this context, it is often debated whether it is better to use Corsi- or Fenwick-based statistics (Peterson 2014). Fenwick-based statistics incorporate shots and misses whereas Corsi-based statistics additionally incorporate blocked shots. Our results indicate that with the addition of blocks, Corsi metrics (e.g. “CF% rel” and “CF%”) are both more discriminative and stable than the Fenwick metrics.

In Figure 3B we plot the estimated independence scores as a function of the number of statistics in the conditional set for five different metrics. Like steals in the NBA, we found that takeaways (TK) provide the most unique information relative to the other 39 metrics. Here, IℳTK=0.73, meaning that all other metrics together only explain 27% of the total variance in takeaways, which is consistent with the dearth of defensive metrics in the NHL. dZS% is an example of a metric that is highly correlated with only one other metric in the set of metrics we study, but poorly predicted by the others. This metric is almost perfectly predicted by its counterpart oZS% and hence IℳdZS≈0 when oZS% ∈ ℳ and significantly larger otherwise. This is clear from the large uptick in the independence score of dZS% after removing oZS% from ℳ.

Once again, the analysis of the dependencies among metrics reveals significant redundancy in information across NHL metrics. We can explain over 90% of the variation in the data using only 15 out of 40 principal components, that is F₁₅ = 0.90 (Figure 10). Figure 4 illustrates a hierarchical clustering of these metrics based on these dependencies.

Figure 4

Hierarchical clustering of NHL metrics based on the correlation matrix, C. Clustered metrics have larger absolute correlations but can be positively or negatively associated. The metrics that have large loadings on the two different principal component (Figure 7) are highlighted in red and blue.

5.1 Shrinkage estimators

Model-based adjustments of common box score statistics can reduce sampling variability and thus lead to improvements in discrimination and stability. In Section 4.1, we showed how three point percentage was one of the least discriminative and stable metrics in basketball and thus an improved estimator of three point making ability is warranted. We define three point ability using the notation introduced in Section 2 as E_sp(3P%)[X] , i.e. the expected three point percentage for player p in season s, and propose a model-based estimate of this quantity that is both more stable and discriminative than the observed percentage.

For this model, we assume an independent hierarchical Bernoulli model for the three point ability of each player:

where Xsp3P% is the observed three point percentage of player p in season s, π_sp = E_sp(3P%)[X] is the estimand of interest, n_sp is the number of attempts, πp0=Ep(3P%)[X] is the career average for player p, and πp0(1−πp0)/rp is the variance in π_sp over time. We use the R package gbp for empirical Bayes inference of π_sp and r_p, which controls the amount of shrinkage (Tak et al. 2016). In Figure 5 we plot the original and shrunken estimates for LeBron James’ three point ability over his career.

Figure 5

Three point percentages for LeBron James by season, and shrunken estimates using the empirical Bayes model proposed by Tak et al. (2016). Top: raw three point percentage (top line, ordered from lowest to highest) and the corresponding shrunken estimates (bottom line). The length of the purple lines are proportional to the number of attempts and the length of the green lines is proportional to the posterior standard deviation of the shrunken estimate. Bottom: Original (hollow circle) and shrunken (red circle) three point percentage over time. Shrinking three point percentage to a player’s career average improves stability and discrimination.

We can compute discrimination and stability estimates for the estimated three point ability derived from this model using the same approach outlined in Section 3. Although the empirical Bayes’ procedure yields probability intervals for all estimates, we can still compute the frequentist variability using the bootstrap (e.g. see Efron (2015)). In Figure 1 we highlight the comparison between observed three point percentage and the empirical Bayes estimate in red. Observed three point percentage is an unbiased estimate of three point ability but is highly unreliable. The Bayes estimate is biased for all players, but theory suggests that the estimates have lower mean squared error due to a reduction in variance (Efron and Morris 1975). The improved stability and discrimination of the empirical Bayes estimate is consistent with this fact.

There are, of course, other shrinkage estimators that may be more appropriate depending on the context. For instance, rather than shrink each estimate toward a player’s career average, we could derive estimators based on shrinkage to the league average for each season. This might be especially reasonable for rookie players or players with little career playing time. More sophisticated extensions might involve shrinking estimates to a regression surface based on covariates like how long the player has been in the league. The choice of shrinkage model should reflect the decision-maker’s classification of observed variation into signal and noise and assumptions about the relationships between those sources of variation. The shrinkage scheme used in this section classifies both chance variation and inter-season variation as noise, and treats player-specific variation is independent.

5.2 Principal component metrics

The dependency model proposed in Section 2.3 provides a natural way to derive new metrics that describe orthogonal aspects of player ability. In particular, the eigendecomposition of the latent correlation matrix, C, (Equation 6) can be used to develop a (smaller) set of new metrics, which, by construction, are mutually independent and explain much of the variation in the original set. If the latent normal variables Z defined in Equation 6 were known, then we could compute the principal components of this matrix to derive a new set of orthogonal metrics. The principal components are defined as W = ZU where U is the matrix of eigenvectors of C. Then, by definition, W ∼ MVN(0, I) and thus Wk⊥⊥Wj∀k≠j. For the independence score defined in Section 2.3, this means that Ik,ℳ−kW=1 for all k, where ℳ−kW is the set of all metrics W_j, j ≠ k. We estimate Z by normalizing X, that is Z^spm=Φ−1(F^m(Xspm)) where F^_m is the empirical CDF of X_m. Our estimate of the principal components of the latent matrix Z is then simply W^sp=Z^spU. It should be noted that metrics with high independence scores (Equation 7) will not typically have large loadings on the first few principal components by definition since the first principal components capture variation in the most redundant metrics. In our formulation, a metric with an independence score of one will load on exactly one eigenvector and this vector will have a corresponding eigenvalue of one.

We present results based on these new PCA-based metrics for both NBA and NHL statistics. In Figure 6 we list three PCA-based metrics for the NBA and the corresponding original NBA metrics which load most heavily onto them. We also rank the top ten players across seasons according to W^sp and visualize the scores for each of these three PCA-based metrics for four different players in the 2014–2015 season. Here, the fact that LeBron James ranks highly in each of these three independent metrics in the 2014-15 season is indicative of his versatility.

Figure 6

First three principal components of C. The tables indicate the metrics that predominantly load on the components. Each component generally corresponds to interpretable aspects of player style and ability. The table includes the highest ranking players across all seasons for each component. The top row depicts principal component score for four players in the 2014–2015 season. The radius of the corresponding segment is determined by the quantile of the PC score, with higher ranking players having larger segments. LeBron James ranks highly among all 3 independent components in the 2014–2015 season.

Although the meaning of these metrics can be harder to determine, they can provide a useful aggregation of high-dimensional measurements of player skill that facilitate fairer comparisons of players.

In Figure 7 we provide two PCA-based metrics for NHL statistics. We again list the metrics that have the highest loadings on two principal component along with the top ten players (in any season) by component. The first principal component largely reflects variation in offensive skill and easily picks up many of the offensive greats, including Ovechkin and Crosby. For comparison, we include another component, which corresponds to valuable defensive players who make little offensive contribution. This component loads positively on defensive point shares (DPS) and blocks (BLK), but negatively on shots and goals (S, G).

Figure 7

Player rankings based on two principal components. The first PC is associated with offensive ability. The fact that this is the first component implies that a disproportionate fraction of the currently available hockey metrics measure aspects of offensive ability. The other included component reflects valuable defensive players (large positive loadings for defensive point shares and blocks) but players that make few offensive contributions (negative loadings for goals and shots attempted). The metrics that load onto these components are highlighted in the dendrogram of NHL metrics (Figure 4).