Throughout this paper, we write the 3-dimensional array of players, seasons and metrics as *X*, with *X*_{spm} the value of metric *m* for player *p* from season *s*. Our meta-metrics are all R-squared style statistics and can be understood as functions of the (co)variances along the three dimensions of *X*. As a useful example, consider a model for a metric *m* that varies over time *s* and between players *p* in a linear mixed effects model:

$$\begin{array}{ccccc}{X}_{spm}\hfill & ={\mu}_{m}+{Z}_{sm}+{Z}_{pm}+{Z}_{spm}+{\u03f5}_{spm},\hfill & & & \end{array}$$(1)

where

$$\begin{array}{ccccc}{Z}_{sm}\hfill & \sim [0,{\sigma}_{\text{SM}}^{2}]\hfill & & & \\ {Z}_{pm}\hfill & \sim [0,{\sigma}_{\text{PM}}^{2}]\hfill & & & \\ {Z}_{spm}\hfill & \sim [0,{\sigma}_{\text{SPM}}^{2}]\hfill & & & \\ {\u03f5}_{spm}\hfill & \sim [0,{\tau}_{\text{M}}^{2}],\hfill & & & \end{array}$$

and [*μ*, *σ*^{2}] represents a distribution with mean *μ* and variance *σ*^{2}. The terms *Z*_{*} can be thought of as random effects, while *ϵ*_{spm} represents the variation induced by the sampling effects of a season – for instance, binomial variation in made shot percentage given a finite sample size; for an infinitely long season, we would observe ${\tau}_{\text{M}}^{2}\to 0$ and thus *ϵ*_{spm} = 0. *Z*_{spm}, on the other hand, reflects true value (above average) of the “skill” *m* of player *p* in season *s*. This model encodes four sources of variation, although we only intend to discuss three. The extra parameter, ${\sigma}_{\text{SM}}^{2}$, captures variation in league averages over time. For the time scales we will consider in this paper, variation in league averages is small; in practice we will ignore this source of variation, but we will maintain it for completeness in our theoretical development.

In this representation, we can recognize ${\sigma}_{\text{PM}}^{2}+{\sigma}_{\text{SPM}}^{2}+{\tau}_{\text{M}}^{2}$ as the within-season, between-player variance; ${\sigma}_{\text{SM}}^{2}+{\sigma}_{\text{SPM}}^{2}+{\tau}_{\text{M}}^{2}$ as the within-player, beween-season variance; and of course, ${\sigma}_{\text{SM}}^{2}+{\sigma}_{\text{PM}}^{2}+{\sigma}_{\text{SPM}}^{2}+{\tau}_{\text{M}}^{2}$ as the total (between player-season) variance. Both the discrimination and stability meta-metrics defined in this section can be expressed as ratios involving these quantities, along with the sampling variance ${\tau}_{\text{M}}^{2}$.

The linear mixed effects model (1) may be a reasonable choice for some metrics and, due to its simplicity, provides a convenient example to illustrate our meta-metrics. However, an exchangeable, additive model is not appropriate for many of the metrics we consider. A major practical challenge in our analysis is that all of the metrics have unique distributions with distinct support – for example, percentages are constrained to the unit interval, while many per game or per season statistics are discrete and strictly positive. Other advanced metrics like “plus-minus” or “value over replacement” (VORP) in basketball are continuous real-valued metrics that can be negative or positive.

To define meta-metrics with full generality, consider the random variable *X*, which is a single entry *X*_{spm} chosen randomly from *X*. Randomness in *X* thus occurs both from sampling the indexes *S*, *P*, and ℳ of *X*, as well as intrinsic variability in *X*_{spm} due to finite season lengths. We will then use the notational shorthand

$$\begin{array}{ccccc}{E}_{spm}\left[X\right]\hfill & =E\left[X\right|S=s,P=p,M=m]\hfill & & & \\ {V}_{spm}\left[X\right]\hfill & =Var\left[X\right|S=s,P=p,M=m]\hfill & & & \end{array}$$

and analogously for *E*_{sm}[*X*], *V*_{sm}[*X*], *E*_{m}[*X*], etc. For example, *E*_{sm}[*V*_{spm}[*X*]] is the average over all players of the intrinsic variability in *X*_{spm} for metric *m* during season *s*, or ∑_{p} *Var*[*X*_{spm}] / *N*_{sm}, where *N*_{sm} is the number of entries of *X*_{s ⋅ m}.

## 2.1 Discrimination

For a metric measuring player ability to be applicable, it must be a useful tool for discriminating between different players. This implies that most of the variability between players reflects true variation in player ability and not chance variation or noise from small sample sizes. As a useful baseline for discrimination, we compare the average intrinsic variability of a metric to the total between player variation in this metric. Similar approaches which partially inspired our version of this metric have been used in analyzing Major League Baseball data (Tango, Lichtman and Dolphin 2007; Arthur 2015).

To characterize the discriminative power of a metric, we need to quantify the fraction of total between player variance that is due to chance and the fraction that is due to signal. By the law of total variance, the between player variance can be decomposed as

$${V}_{sm}\left[X\right]={E}_{sm}\left[{V}_{spm}\left[X\right]\right]+{V}_{sm}\left[{E}_{spm}\left[X\right]\right].$$

Here, *V*_{sm}[*X*] corresponds to the total variation in metric *m* between players in season *s*, whereas *E*_{sm}[*V*_{spm}[*X*]] is the average (across players) sampling variability for metric *m* in season *s*. With this decomposition in mind, we define the discriminative power of a metric *m* in season *s* as

$$\text{(Discrimination)}{\mathcal{D}}_{sm}=1-\frac{{E}_{sm}\left[{V}_{spm}\left[X\right]\right]}{{V}_{sm}\left[X\right]}.$$(2)

Intuitively, this describes the fraction (between 0 and 1) of between-player variance in metric *m* (in season *s*) due to true differences in player ability. Discrimination meta-metrics for different seasons can be combined as ${\mathcal{D}}_{m}={E}_{m}\left[{\mathcal{D}}_{sm}\right]$.

It is helpful to understand the discrimination estimand for the linear mixed effects model defined in Equation 1. When this model holds, ${E}_{sm}\left[{V}_{spm}\left[X\right]\right]={\tau}_{\text{M}}^{2}$, and ${V}_{sm}\left[X\right]={\sigma}_{\text{PM}}^{2}+{\sigma}_{\text{SPM}}^{2}+{\tau}_{\text{M}}^{2}$, the between-player variance (equal for all seasons *s*). Thus, the discrimination meta-metric under the linear mixed effects model is simply

$$\begin{array}{ccccc}{\mathcal{D}}_{m}\hfill & =1-\frac{{\tau}_{\text{M}}^{2}}{{\sigma}_{\text{PM}}^{2}+{\sigma}_{\text{SPM}}^{2}+{\tau}_{\text{M}}^{2}}\hfill & & & \\ & =\frac{{\sigma}_{\text{PM}}^{2}+{\sigma}_{\text{SPM}}^{2}}{{\sigma}_{\text{PM}}^{2}+{\sigma}_{\text{SPM}}^{2}+{\tau}_{\text{M}}^{2}}.\hfill & & & \end{array}$$(3)

## 2.2 Stability

In addition to discrimination, which is a meta-metric that describes variation within a single season, it is important to understand how much an individual player’s metric varies from season to season. The notion of stability is particularly important in sports management when making decisions about future acquisitions. For a stable metric, we have more confidence that this year’s performance will be predictive of next year’s performance. A metric can be unstable if it is particularly context dependent (e.g. the player’s performance varies significantly depending on who their teammates are) or if a player’s intrinsic skill set tends to change year to year (e.g. through offseason practice or injury).

Consequently, we define stability as a metric, which describes how much we expect a single player metric to vary over time after removing chance variability. This metric specifically targets the sensitivity of a metric to change in context or intrinsic player skill over time. Mathematically, we define stability as:

$$\text{(Stability)}{S}_{m}=1-\frac{{E}_{m}\left[{V}_{pm}\left[X\right]-{V}_{spm}\left[X\right]\right]}{{V}_{m}\left[X\right]-{E}_{m}\left[{V}_{spm}\left[X\right]\right]},$$(4)

with 0 ≤ *S*_{m} ≤ 1 (see Appendix for proof). Here, *V*_{pm}[*X*] is the between-season variability in metric *m* for player *p*; thus, the numerator in (4) averages the between-season variability in metric *m*, minus sampling variance, over all players. The denominator is the total variation for metric *m* minus sampling variance. Again, this metric can be easily understood under the assumption of an exchangeable linear model (Equation 1).:

$$\begin{array}{ccccc}{S}_{m}\hfill & =1-\frac{{\sigma}_{\text{SM}}^{2}+{\sigma}_{\text{SPM}}^{2}+{\tau}_{\text{M}}^{2}-{\tau}_{\text{M}}^{2}}{{\sigma}_{\text{PM}}^{2}+{\sigma}_{\text{SM}}^{2}+{\sigma}_{\text{SPM}}^{2}+{\tau}_{\text{M}}^{2}-{\tau}_{\text{M}}^{2}}\hfill & & & \\ & =\frac{{\sigma}_{\text{PM}}^{2}}{{\sigma}_{\text{PM}}^{2}+{\sigma}_{\text{SM}}^{2}+{\sigma}_{\text{SPM}}^{2}}.\hfill & & & \end{array}$$(5)

This estimand reflects the fraction of total variance (with sampling variability removed) that is due to within-player changes over time. If the within player variance is as large as the total variance, then *S*_{m} = 0 whereas if a metric is constant over time, then *S*_{m} = 1.

## 2.3 Independence

When multiple metrics measure similar aspects of a player’s ability, we should not treat these metrics as independent pieces of information. This is especially important for decision makers in sports management who use these metrics to inform decisions. Accurate assessments of player ability can only be achieved by appropriately synthesizing the available information. As such, we present a method for quantifying the dependencies between metrics that can help decision makers make sense of the growing number of data summaries.

For some advanced metrics we know their exact formula in terms of basic box score statistics, but this is not always the case. For instance, it is much more challenging to assess the relationships between new and complex model based NBA metrics like adjusted plus minus (Sill 2010), EPV-Added (Cervone et al. 2016) and counterpoints (Franks et al. 2015), which are model-based metrics that incorporate both game-log and player tracking data. Most importantly, even basic box score statistics that are not functionally related will be correlated if they measure similar aspects of intrinsic player skill (e.g., blocks and rebounds in basketball are highly correlated due to their association with height).

As such, we present a general approach for expressing dependencies among an arbitrary set of metrics measuring multiple players’ styles and abilities across multiple seasons. Specifically, we propose a Gaussian copula model in which the dependencies between metrics are expressed with a latent multivariate normal distribution. Assuming we have ℳ metrics of interest, let *Z*_{sp} be an ℳ-vector of metrics for player *p* during season *s*, and

$$\begin{array}{ccccc}{Z}_{sp}\hfill & \stackrel{iid}{\sim}\text{MVN}(0,C)\hfill & & & \\ {X}_{spm}\hfill & ={F}_{m}^{-\mathrm{\hspace{0.25em}1}}\left[\mathrm{\Phi}\left({Z}_{spm}\right)\right],\hfill & & & \end{array}$$(6)

where *C* is a ℳ × ℳ correlation matrix, and ${F}_{m}^{-\mathrm{\hspace{0.25em}1}}$ is the inverse of the CDF for metric *m*. We define the independence score of a metric *m* given a condition set of other metrics, ℳ, as

$$\begin{array}{ccccc}{I}_{m\mathcal{M}}\hfill & =\frac{Var[{Z}_{spm}\mid \{{Z}_{spq}:q\in \mathcal{M}\}]}{Var\left[{Z}_{spm}\right]}\hfill & & & \\ & ={C}_{m,m}-{C}_{m,\mathcal{M}}{C}_{\mathcal{M},\mathcal{M}}^{-\mathrm{\hspace{0.25em}1}}{C}_{\mathcal{M},m}.\hfill & & & \end{array}$$(7)

For the latent variables *Z*, this corresponds to one minus the R-squared for the regression of *Z*_{m} on the latent variables *Z*_{q} with *q* in ℳ. Metrics for which ${I}_{m\mathcal{M}}$ is small (e.g. for which the R-squared is large) provide little new information relative to the information in the set of metrics ℳ. In contrast, when ${I}_{m\mathcal{M}}$ is large, the metric is nearly independent from the information contained in ℳ. Note that ${I}_{m\mathcal{M}}=1$ implies that metric *m* is independent from all metrics in ℳ.

We also run a principal component analysis (PCA) on *C* to evaluate the amount of independent information in a set of metrics. If *UΛ U*^{T} is the eigendecomposition of *C*, with Λ = diag(*λ*_{1}, … *λ*_{ℳ}) the diagonal matrix of eigenvalues, then we can interpret ${F}_{k}=\frac{{\sum}_{1}^{k}{\lambda}_{i}}{{\sum}_{1}^{M}{\lambda}_{i}}$ as the fraction of total variance explained by the first *k* principal components (Mardia, Kent and Bibby 1980). When *F*_{k} is large for small *k* then there is significant redundancy in the set of metrics, and thus dimension reduction is possible.

## Comments (0)