Aggregating for Accuracy builds on previous literature to develop a framework for examining the effect of sampling on accuracy in class action litigation. The article examines a procedure by which 1) a number of claims are sampled from a class of claims for individualized adjudication (and individualized damage awards), and 2) the mean of the awards adjudicated in the sample group is applied as the award for all remaining claims, the claims in the extrapolation group.

The article’s analysis is intended to respond to arguments that such procedures increase efficiency (by allowing putative class members to proceed as a class rather than as individual claimants), but only at the cost of reducing accuracy. It builds on assertions by Professors Michael Saks and Peter Blanck in a 1992 Stanford Law Review article to argue that, under certain conditions, sampling can increase accuracy by reducing error associated with judgment variability – that is, uncertainty in the adjudication of an award resulting, for example, from variability in the composition of a jury, the presentation of evidence, and the selection of a judge. (Bavli 2015; Saks & Blanck 1992).

To illustrate, consider how replication may be used to reduce judgment variability:

Aggregating for Accuracy shows that sampling in a class of homogeneous claims, through its use of replication, can improve accuracy by reducing judgment variability (Bavli 2015; Saks & Blanck 1992). In particular, the article concludes that, given a class of $N$ homogeneous claims, and legal restrictions that can be described by “reductive sampling” – where a court will not replace an individually adjudicated award with an award extrapolated from other claims – accuracy is maximized by randomly selecting a sample of $n^{\ast }=\sqrt{N}$ claims for individualized adjudication, assigning the individualized awards as the outcomes of the claims in the sample group, respectively, and then applying the sample mean of the sample-group awards as the outcome of all remaining ($N-\sqrt{N}$) claims in the class (Bavli 2015).

To be more precise: Assume we have a class of $N$ homogeneous claims from which we sample $n$ claims. Let us define a “correct” outcome associated with a particular claim as the average award that would emerge from repeated adjudication of the claim under different conditions, such as different jury combinations, different lawyers, different judges, and different presentations of evidence (Bavli 2015; Saks & Blanck 1992). The “correct” outcome can be defined using various measures of central tendency, such as the mean or median. Throughout this paper, we adopt the measure used in Aggregating for Accuracy and Saks & Blanck (1992) – the mean. Thus, let $\mathrm{\mu }$ be the correct award in each of the $N$ homogeneous claims – the mean of the awards that would result from repeated adjudications of any (or all, since the claims are homogeneous) of the $N$ claims. And let $X_i$ be a random variable defined by the actual award in the $i^{th}$ claim, for $i=1,2,3,...,N$, where the $X_i$ are independent and identically distributed (i.i.d.) with mean $\mathrm{\mu }$ and variance ${\mathrm{\sigma }}^2$, and the sample mean of the $X_i$, for $i=1,2,3,...,n$, is defined as ${\bar{X}}_n$, which is distributed with mean $\mathrm{\mu }$ and variance $\frac{{\mathrm{\sigma }}^2}{n}$. Notationally: $X_i\sim (\mathrm{\mu },{\mathrm{\sigma }}^2)$ and ${\bar{X}}_n\sim (\mathrm{\mu },\frac{{\mathrm{\sigma }}^2}{n})$. Note, in Aggregating for Accuracy, it is assumed that the $X_i$ are distributed normally; but this distributional assumption is not necessary for the results derived in that paper or in the current paper. We therefore drop the normality assumption.

Thus, using the sum of square residuals $r=\sum _{i=1}^n(X_i-\mathrm{\mu }{)}^2+(N-n)({\bar{X}}_n-\mathrm{\mu }{)}^2$

as the criterion for measuring the error associated with all $N$ claims, Aggregating for Accuracy concludes that total error, or “risk” (R) – defined as the expectation of $r$ above – is minimized, and accuracy is maximized, not by adjudicating each claim individually (a procedure often viewed by courts and scholars as the ideal, with respect to accuracy), but rather by sampling and individually adjudicating $n_{hom}^{\ast }=\sqrt{N}$

claims, and applying the sample mean ${\bar{X}}_{n^{\ast }}$ as the outcome for all remaining $N-\sqrt{N}$ claims (Bavli 2015).

Thus, in the context of a homogeneous class of claims, sampling may improve accuracy as well as efficiency. But, as explained in Aggregating for Accuracy, homogeneity is not necessary for sampling to increase accuracy. First, a court may stratify a heterogeneous class to obtain relatively homogeneous subclasses. For example, the District Court for the Eastern District of Texas, in Cimino v. Raymark, 751 F. Supp. 649 (E.D. Tex. 1990), used such a procedure when it divided a heterogeneous class of asbestos claims into five disease categories. Second, although homogeneity is helpful, it is not necessary – the error-reducing benefits of sampling apply even to a class of heterogeneous claims.

Homogeneity allows a court to maximize accuracy by sampling $\sqrt{N}$ claims. However, as a class becomes more heterogeneous, the utility of sampling is reduced, since the benefits of reducing judgment variability must now be balanced with the error introduced by applying a single point estimate – the sample mean – to a class of heterogeneous claims. Thus, for a class of heterogeneous claims, the optimal sample size will fall between $\sqrt{N}$ and $N$ (Bavli 2015). Simply stated, sampling can increase accuracy as long as the heterogeneity of the claims is not too large.

Aggregating for Accuracy models heterogeneous awards as draws from normal distributions with means ${\mathrm{\mu }}_i$ and variance ${\mathrm{\sigma }}^2$. Contrary to a homogeneous class, where all claims have the same correct award $\mathrm{\mu }$, in a heterogeneous class, each claim $i$ has a correct award ${\mathrm{\mu }}_i$, where the correct awards are distributed with mean ${\mathrm{\mu }}_0$ and variance ${\mathrm{\tau }}^2$. Thus, $X_i\sim ({\mathrm{\mu }}_i,{\mathrm{\sigma }}^2)$, independent, where ${\mathrm{\mu }}_i\sim ({\mathrm{\mu }}_0,{\mathrm{\tau }}^2)$, i.i.d. Note, here again we relax the normality assumption used in Aggregating for Accuracy.

Thus, again using the sum of square residuals, we have $r=\sum _{i=1}^n(X_i-{\mathrm{\mu }}_i{)}^2+\sum _{j=n+1}^N({\bar{X}}_n-{\mathrm{\mu }}_j{)}^2$

as the criterion for measuring the error associated with all $N$ claims. Aggregating for Accuracy minimizes the expectation of this expression and derives the optimal sample size for heterogeneous claims to be $n^{\ast }=\sqrt{N}\sqrt{\frac{{\mathrm{\sigma }}^2+{\mathrm{\tau }}^2}{{\mathrm{\sigma }}^2-{\mathrm{\tau }}^2}}.$

Thus, when ${\mathrm{\tau }}^2$, or “claim variability” (i.e., the heterogeneity of the class), is dominated by ${\mathrm{\sigma }}^2$, or judgment variability, $n^{\ast }$ falls between $\sqrt{N}$ and $N$. If claim variability is greater than judgment variability (${\mathrm{\tau }}^2\ge {\mathrm{\sigma }}^2$), then $n^{\ast }=N$. And when claim variability is zero (${\mathrm{\tau }}^2=0$), then $n^{\ast }=\sqrt{N}$, the result obtained for a homogeneous class (Bavli 2015).

Our objective in this subsection is to show that, for a class of claims, replacing an adjudicated damages award with an award based on shrinkage increases accuracy, in expectation, for each adjudicated claim, and therefore, in the aggregate for all sampled claims.

As above, assume we have a class of $N$ claims and that each claim $i$ is associated with a correct award ${\mathrm{\mu }}_i$, for $i=1,2,3,...,N$. Assume the ${\mathrm{\mu }}_i$’s are equal (i.e., the claims are homogeneous) or that they arise from a common distribution with mean ${\mathrm{\mu }}_0$ and variance ${\mathrm{\tau }}^2$. Denote the awards of the $n$ sampled claims by $X_1,\dots ,X_n$, and assume that they are distributed around their correct awards ${\mathrm{\mu }}_i$, respectively, with variance ${\mathrm{\sigma }}^2$. Thus, $X_i\sim ({\mathrm{\mu }}_i,{\mathrm{\sigma }}^2)$, independent, where ${\mathrm{\mu }}_i\text{m}\sim ({\mathrm{\mu }}_0,{\mathrm{\tau }}^2)$, i.i.d., and ${\mathrm{\sigma }}^2$ and ${\mathrm{\tau }}^2$ are known and represent judgment variability and claim variability, respectively. (See Aggregating for Accuracy for recommendations for estimating relevant parameters). As above, we do not rely on distributional assumptions.

Our objective is to impute all of the missing correct outcomes $\{{\mathrm{\mu }}_i{\}}_{i=1}^N$, which are not directly observable, using estimated values $\{{\hat{\mathrm{\mu }}}_i{\}}_{i=1}^N$, and to do so in a way that minimizes error, represented by the (standard) risk function, $R=\mathbb{E}\sum _{i=1}^N({\hat{\mathrm{\mu }}}_i-{\mathrm{\mu }}_i{)}^2=\mathbb{E}\sum _{i=1}^n({\hat{\mathrm{\mu }}}_i-{\mathrm{\mu }}_i{)}^2+\mathbb{E}\sum _{i=n+1}^N({\hat{\mathrm{\mu }}}_i-{\mathrm{\mu }}_i{)}^2.$

We thus replace the classical estimator – an adjudicated award ($X_i$) – with a shrinkage estimator (${\hat{\mathrm{\mu }}}_i^s$), which combines the adjudicated award with additional information obtained from the other claims in the class. We define the shrinkage estimator as: ${\hat{\mathrm{\mu }}}_i^s=\frac{X_i/{\mathrm{\sigma }}^2+{\mathrm{\mu }}_0/{\mathrm{\tau }}^2}{1/{\mathrm{\sigma }}^2+1/{\mathrm{\tau }}^2}.$This estimator (${\hat{\mathrm{\mu }}}_i^s$) for the correct outcome in the $i^{th}$ claim thus differs from the classical estimator, the adjudicated award ($X_i$), in that ${\hat{\mathrm{\mu }}}_i^s$ is a weighted average of the adjudicated award ($X_i$) and the mean (${\mathrm{\mu }}_0$) of the ${\mathrm{\mu }}_i$, weighted by the inverse of the variability of each, respectively. Thus, $X_i$ and ${\mathrm{\mu }}_0$ are weighted by the inverse of the judgment variability (${\mathrm{\sigma }}^2$) and the inverse of the claim variability (${\mathrm{\tau }}^2$), respectively. Intuitively, smaller variability implies greater information, and therefore heavier weight (Casella 1985).

Note, if we assume that the distributions of $X_i$ and ${\mathrm{\mu }}_i$ are Gaussian (or “Normal”) – i.e., if we regard $X_i\sim \text{calN}({\mathrm{\mu }}_i,{\mathrm{\sigma }}^2)$ as the likelihood and ${\mathrm{\mu }}_i\sim \text{Cal}N({\mathrm{\mu }}_0,{\mathrm{\tau }}^2)$ as the prior – we indeed have an explicit statistical justification for this estimator. It is the Bayes estimator, which is admissible. It is also the maximum likelihood estimator in the hierarchical model.

It follows that the risk of the shrinkage estimator is $R_i^s:=\mathbb{E}{({\widehat{\mu }}_i^s-{\mu }_i)}^2=\mathbb{E}{\left(\frac{X_i/{\sigma }^2+{\mu }_0/{\tau }^2}{1/{\sigma }^2+1/{\tau }^2}-{\mu }_i\right)}^2=\left(\frac{1}{{\sigma }^2}+\frac{1}{{\tau }^2}\right)-1,$[1]

which is smaller than $R_i^c:=\mathbb{E}(X_i-{\mathrm{\mu }}_i{)}^2={\mathrm{\sigma }}^2$, the risk associated with the classical estimator $X_i$ of ${\mathrm{\mu }}_i$.

Therefore, for each individual claim, using the shrinkage estimator to compute damages yields greater accuracy on average – that is, lower risk – as compared to an adjudicated award. Furthermore, this result implies that applying an individualized shrinkage award ${\hat{\mathrm{\mu }}}_i^s$ to each claim in the class reduces total risk, since risk is reduced for each individual claim.

In general, we will not know the value of ${\mathrm{\mu }}_0$. However, we can substitute the unbiased estimator ${\hat{\mathrm{\mu }}}_0={\sum }_{i=1}^n{X}_i/n$ for ${\mathrm{\mu }}_0$ in the shrinkage estimator, yielding the “empirical” shrinkage estimator ${\widehat{\mu }}_i^{se}=\frac{X_i/{\sigma }^2+{\widehat{\mu }}_0/{\tau }^2}{1/{\sigma }^2+1/{\tau }^2},$[2]

where ${\hat{\mathrm{\mu }}}_0\sim ({\mathrm{\mu }}_0,\frac{{\mathrm{\sigma }}^2+{\mathrm{\tau }}^2}{n})$. Note that in the Gaussian case, where the $X_i$ are independent with distribution $\text{calN}({\mathrm{\mu }}_i,{\mathrm{\sigma }}^2)$, and ${\mathrm{\mu }}_i$ has prior distribution $\text{calN}({\mathrm{\mu }}_0,{\mathrm{\tau }}^2)$, this estimator is the “empirical Bayes estimator” (Efron & Morris 1973; Robbins 1955), which replaces the hyperparameter ${\mathrm{\mu }}_0$ with its maximum likelihood estimate. This empirical Bayes estimator, which converges to the Bayes estimator as $n\to \mathrm{\infty }$, is asymptotically admissible. Therefore, statistically it is a justified estimator.

Thus, let us confirm that the risk associated with this estimator is less than the risk associated with the classical estimator, an adjudicated award. Letting $A=\frac{{\mathrm{\sigma }}^{-2}}{{\mathrm{\sigma }}^{-2}+{\mathrm{\tau }}^{-2}}$, we can write this “empirical” shrinkage estimator ${\hat{\mathrm{\mu }}}_i$ (note that, for simplicity, we drop the “se” in the superscript) as ${\hat{\mathrm{\mu }}}_i=A{X}_i+(1-A){\hat{\mathrm{\mu }}}_0$. The risk of this estimator is $R_i^{se}:=\mathbb{E}({\hat{\mathrm{\mu }}}_i-{\mathrm{\mu }}_i{)}^2=\mathbb{E}[A(X_i-{\mathrm{\mu }}_i)+(1-A)({\hat{\mathrm{\mu }}}_0-{\mathrm{\mu }}_i){]}^2$[3] $=A^2{\mathrm{\sigma }}^2+2A(1-A)\mathbb{E}(X_i-{\mathrm{\mu }}_i)({\hat{\mathrm{\mu }}}_0-{\mathrm{\mu }}_i)+(1-A{)}^2\mathbb{E}({\hat{\mathrm{\mu }}}_0-{\mathrm{\mu }}_i{)}^2$[4] $={\left(\frac{1}{{\sigma }^2}+\frac{1}{{\tau }^2}\right)}^{-1}+\frac{1}{n}\frac{{\sigma }^4}{{\sigma }^2+{\tau }^2}\to R_i^s(n\to \infty ).$[5]This risk is somewhat larger than $R_i^s$ with $R_i^{se}-R_i^s=n^{-1}{\mathrm{\sigma }}^4/({\mathrm{\sigma }}^2+{\mathrm{\tau }}^2)>0$. Intuitively, because we have less information – we do not know the true value of ${\mathrm{\mu }}_0$ – we have greater risk. However, as the sample size increases, this risk converges to that in the case in which we know the true ${\mathrm{\mu }}_0$. That is, $R_i^{se}\to R_i^s$ as $n\to \mathrm{\infty }$.

Most significantly, however, $R_i^{se}$, the risk associated with the empirical shrinkage estimator, is substantially smaller than $R_i^c$, the risk associated with the classical estimator (i.e., the adjudicated award). That is, $R_i^{se}-R_i^c=(-1+n^{-1}){\mathrm{\sigma }}^4/({\mathrm{\sigma }}^2+{\mathrm{\tau }}^2)<0$. Thus, under the specified criterion, the empirical shrinkage estimator is also (in addition to the shrinkage estimator) a better estimator than an adjudicated award.

Note that if it is not feasible to estimate claim variability (a possibility discussed in Aggregating for Accuracy), we can nevertheless rely on the James-Stein estimator (Efron & Morris 1975; James & Stein 1961), which does not depend on claim variability: ${\widehat{\mu }}_i^{JS}={\widehat{\mu }}_0+\left(1-\frac{n-3}{S}{\sigma }^2\right)(X_i-{\widehat{\mu }}_0),$where $S={\sum }_{i=1}^n(X_i-{\hat{\mathrm{\mu }}}_0{)}^2$ is an unbiased estimator for $(n-1)({\mathrm{\sigma }}^2+{\mathrm{\tau }}^2)$. If $X_i$ and ${\mathrm{\mu }}_i$ are Gaussian, $(n-3)/S$ is an unbiased estimator for $({\mathrm{\sigma }}^2+{\mathrm{\tau }}^2{)}^{-1}$. The risk associated with the James-Stein estimator can be derived under Gaussian assumptions (Efron & Morris 1975); and it can be shown that this risk is also less than the risk associated with the classical estimator. Indeed, the James-Stein estimator converges to the shrinkage estimator as $n\to \mathrm{\infty }$; and the risk associated with the James-Stein estimator and the risk associated with the shrinkage estimator above are asymptotically equal.

In this subsection, we reexamine the accuracy benefits of sampling, but now using the empirical shrinkage estimator (${\hat{\mathrm{\mu }}}_i$), rather than an adjudicated award ($X_i$), for the claims in the sample group. As before, the awards for the remaining claims (the claims in the extrapolation group) are extrapolated using the estimated global mean ${\hat{\mathrm{\mu }}}_0$.

Let us begin by deriving the risk associated with the empirical shrinkage estimator in the sampling framework discussed above. The total risk for sampled and non-sampled claims is: $R_S=R_{sampled}+R_{non-sampled}$ $=\sum _{i=1}^n\mathbb{E}({\hat{\mathrm{\mu }}}_i-{\mathrm{\mu }}_i{)}^2+\sum _{i=n+1}^N\mathbb{E}({\hat{\mathrm{\mu }}}_0-{\mathrm{\mu }}_i{)}^2$ $=\sum _{i=1}^n{R}_i^{se}+(N-n)\left({\mathrm{\tau }}^2+\frac{{\mathrm{\sigma }}^2+{\mathrm{\tau }}^2}{n}\right)$ $=n{\left(\frac{1}{{\mathrm{\sigma }}^2}+\frac{1}{{\mathrm{\tau }}^2}\right)}^{-1}+\frac{{\mathrm{\sigma }}^4}{{\mathrm{\sigma }}^2+{\mathrm{\tau }}^2}+(N-n)\left[\frac{{\mathrm{\sigma }}^2+{\mathrm{\tau }}^2}{n}+{\mathrm{\tau }}^2\right],$where $R_i^{se}$ (eq. [3]) is the risk associated with a single sampled claim $i$ using the empirical shrinkage estimator (eq. [2]).

$R_S$ is thus a monotone decreasing function of $n$: $\frac{\mathrm{\partial }{R}_S}{\mathrm{\partial }n}=-\frac{{\mathrm{\tau }}^4}{{\mathrm{\tau }}^2+{\mathrm{\sigma }}^2}-\frac{N}{n^2}({\mathrm{\sigma }}^2+{\mathrm{\tau }}^2)<0.$[6]This means that risk will continue to decrease as we increase the sample size $n$.

On the other hand, using the classical estimator, an adjudicated award $X_i$, to estimate ${\mathrm{\mu }}_i$ results in the risk function, $R_C=\mathbb{E}\sum _{i=1}^N(X_i-{\mathrm{\mu }}_i{)}^2=\mathbb{E}\sum _{i=1}^n(X_i-{\mathrm{\mu }}_i{)}^2+\mathbb{E}\sum _{j=n+1}^N({\hat{\mathrm{\mu }}}_0-{\mathrm{\mu }}_j{)}^2$ $=n{\mathrm{\sigma }}^2+(N-n)\left(\frac{{\mathrm{\sigma }}^2}{n}+\frac{n+1}{n}{\mathrm{\tau }}^2\right),$

which, as derived in Aggregating for Accuracy (and reviewed above), is minimized at $n^{\ast }=\sqrt{N\frac{{\mathrm{\sigma }}^2+{\mathrm{\tau }}^2}{{\mathrm{\sigma }}^2-{\mathrm{\tau }}^2}}.$[7]Thus, Figure 1 illustrates the divergence of the risk of the empirical shrinkage estimator from the risk of the classical estimator. Specifically, Figure 1 plots the risk associated with a class of $1,000$ claims against the number of claims, $n$, sampled for individual adjudication. The figure shows the risk $R_C$ associated with the classical estimator (adjudicated awards) and the risk $R_S$ associated with the shrinkage estimator for sample sizes between 10 and 200. As Figure 1 illustrates, $R_C$ is minimized based on eq. [7] – where, here, $n^{\ast }$ is approximately 33 (with $\mathrm{\sigma }/\mathrm{\tau }=5$) – whereas $R_S$ is monotonically decreasing in $n$. Therefore, if we relax the reductive sampling constraint in Aggregating for Accuracy, and instead assume the permissibility of shrinkage estimation, the sample size that minimizes risk is $N$, the total number of claims in the class. (Note, if claim variability is unknown, the court can first sample a small number of claims for adjudication and use $S={\sum }_{i=1}^n(X_i-{\hat{\mathrm{\mu }}}_0{)}^2$ to estimate $(n-1)({\mathrm{\sigma }}^2+{\mathrm{\tau }}^2)$ (Bavli 2015). In this case, $n^{\ast }=\sqrt{N}{\left(\frac{2(n-1){\mathrm{\sigma }}^2}{S}-1\right)}^{-\frac{1}{2}}$. We can then obtain the asymptotic interval for this new estimated number of claims using the central limit theorem and the delta method. We can therefore obtain a range for the optimal sample size and determine our choice based on some extraneous criteria.)

Figure 1:

Illustration of risk, plotted against the number of claims, $n$, sampled for adjudication, comparing the risk, $R_C$, associated with the classical estimator (adjudicated awards), and the risk, $R_S$, associated with the shrinkage estimator, given a class of $1,000$ claims and sample sizes between 10 and 200.

Importantly, the foregoing results should not be interpreted as supporting an argument against sampling; the accuracy benefits of sampling are substantial under the conditions, including the legal constraints, described in Aggregating for Accuracy. Rather, these results demonstrate the benefits of shrinkage. Indeed, shrinkage does not detract from the accuracy benefits of sampling; rather, it sufficiently enhances the accuracy of individual estimates – i.e., resulting from individualized adjudication and the replacement of individually adjudicated awards with shrinkage estimates – that, in a sense, it reduces the need for (i.e., the relative benefits of) sampling.

Furthermore, it is significant that in circumstances in which judgment variability dominates claim variability (${\mathrm{\sigma }}^2>{\mathrm{\tau }}^2$), shrinkage estimation reduces risk at a relatively high rate when $n<n^{\ast }$ and at a relatively low rate when $n>n^{\ast }$ (since, as $n$ approaches $N$, the second term on the right side of eq. [6] approaches $0$ and the first term dominates). Therefore, if the reductive sampling constraint is relaxed – and, in particular, if courts are willing to adjust individual adjudications to incorporate aggregate information – then, to balance concerns regarding accuracy with concerns regarding litigation costs, a court may apply a sampling-based method consistent with the results derived in Aggregating for Accuracy, but add the step of replacing the individually adjudicated awards with shrinkage estimates. That is, the court may sample $n^{\ast }=\sqrt{N\frac{{\mathrm{\sigma }}^2+{\mathrm{\tau }}^2}{{\mathrm{\sigma }}^2-{\mathrm{\tau }}^2}}$ claims for individualized adjudication; assign individualized shrinkage estimates to the $n^{\ast }$ sample-group claims (based on their respective individualized awards as well as the mean of the sampled claims); and apply the mean of the awards in the sample group as the outcome of all non-sampled claims. This procedure is identical to the procedure derived in Aggregating for Accuracy, with one difference: here, the court would apply individualized shrinkage estimates, rather than individualized classical estimates (i.e., individually adjudicated awards), as the outcomes of the claims in the sample group.

In concluding this section, we note that we do not intend to make normative statements regarding the appropriate use of shrinkage in litigation. For example, it is beyond the scope of this paper to address the constitutionality of shrinkage or related policy concerns. Instead, we aim to develop a more complete understanding of aggregation, with respect to accuracy, and to examine a number of key results regarding the accuracy benefits of shrinkage.

In light of the results above, choosing an aggregation approach to maximize accuracy depends on the applicable legal and cost constraints. For example, if there is no concern for the reductive sampling constraint or cost constraints, repeated adjudications of each claim in a class would maximize accuracy. If constrained by litigation costs but not by reductive sampling, then following a method such as the method described above involving both sampling and shrinkage may maximize accuracy. If constrained by reductive sampling (whether or not there is also concern regarding litigation costs), then a sampling method without shrinkage, such as the procedure derived in Aggregating for Accuracy, would maximize accuracy.

In the following section, we extend our analysis to the individual-claim context, which, unlike the class action context, involves no predefined set of claims on which to base aggregation.

The problem addressed in The Logic of CCG is the unpredictability (i.e., judgement variability) of awards for pain and suffering and punitive damages – two types of awards for which the jury receives very little guidance from the court. The Supreme Court and lower courts have repeatedly emphasized the importance of reducing the variability of such awards. See, e.g., Exxon Shipping Co. v. Baker, 554 U.S. 471 (2008). (Bavli 2017)

The Logic of CCG highlights problems associated with existing methods, such as additur and remittitur, tools used by courts to increase or decrease the amount of an award found to be inadequate or excessive. Although these tools can be useful, and can be used to incorporate prior-award information in various ways (Kadane 2009), and in conjunction with other methods, alone they address extreme awards only, rather than variability generally, and (in practice) they ordinarily address only excessive awards and not inadequate ones (Bavli 2017). Additionally, widespread use of such methods arguably replaces the discretion of the trier of fact with that of the court, raising constitutional and policy issues. Other methods, such as caps, arbitrarily draw cutoff points, leading to bias and perverse outcomes (Bavli 2017).

As mentioned above, there is empirical evidence that providing the trier of fact with information regarding awards in prior cases is effective in reducing variability. But such studies do not address the effect of prior-award information on accuracy – that is, bias and variability.

The Logic of CCG develops a framework for examining the benefits and limitations of prior-award information in terms of accuracy; and it addresses a number of major challenges to the use of prior-award information to reduce variability, including the possibility of using award information from an “incorrect” set of prior cases (Bavli 2017). In the current section, we apply shrinkage estimation to analyze the effect of prior-award information on accuracy, and to derive a number of important results regarding this latter challenge in particular.

As a preliminary matter, it is important to realize that there is no “correct” or “incorrect” set of prior cases. As discussed in The Logic of CCG, the effect of prior-award information on accuracy depends on 1) the alignment of the mean of the correct awards in the prior cases with the correct award in the subject case (or, in practice, the alignment of the material facts and issues in the prior cases with those in the subject case); 2) the substantive breadth of the prior cases; and 3) the number of prior cases, or the “sample size.” For example, the alignment of material facts and issues (or, for short, the alignment of the prior cases or prior awards) affects the bias introduced by the prior awards. We would like for the average correct award in the prior cases to align with, or be equal to, the correct award in the subject case. The breadth of the prior awards (or cases) affects, for example, the influence of the prior-award information on the subject award; but a set of prior cases that contains only identical, or almost identical, material facts and issues may result in a sample size of one or two, or even zero, prior awards. Thus, in identifying a set of prior cases, a court must balance its interests in maintaining a reasonable sample size, a reasonable breadth, and cases that involve facts and issues that are relatively aligned with those in the subject case (Bavli 2017).

Consider the example of the Seventh Circuit case, Jutzi-Johnson v. United States, 263 F.3d 753 (7th Cir. 2001), described in The Logic of CCG. That case involved an award for pain and suffering arising from circumstances in which a jail inmate committed suicide by hanging, due to a failure of the jail to supervise him appropriately. A court considering prior awards (as Judge Posner did in Jutzi-Johnson) would decide, for example, whether to use only cases involving inmates who hung themselves, individuals who hung themselves from the general population, individuals who committed suicide from the general population, individuals who suffered from asphyxiation (e.g., drowning) from the general population, etc. See Jutzi-Johnson, 263 F.3d at 760-61. If a court were to restrict its consideration to cases involving inmates who hung themselves, it would potentially obtain a very poor sample size; if the court were to use a wider breadth of cases, the prior awards would have less influence and a higher risk of introducing bias (Bavli 2017).

The important point, for purposes of the current analysis, is that there are tradeoffs among breadth, alignment, and sample size; and combinations of these factors correspond to various levels of bias and variance, and therefore accuracy.

Thus, consider an individual claim that receives award $Y$. $Y$ is centered at the correct award ${\mathrm{\mu }}_y$ with judgement variability ${\mathrm{\sigma }}_y^2$, which is assumed to be known. The correct award ${\mathrm{\mu }}_y$ is centered at ${\mathrm{\lambda }}_0$ with variance ${\mathrm{\eta }}_0^2$, and can be understood as a single award from a distribution representing a population of awards from comparable claims. The correct awards in the comparable claims, as well as the correct award in the subject claim, are distributed around a global mean ${\mathrm{\lambda }}_0$ with variability ${\mathrm{\eta }}_0^2$. Thus, $Y\sim ({\mathrm{\mu }}_y,{\mathrm{\sigma }}_y^2),{\mathrm{\mu }}_y\sim ({\mathrm{\lambda }}_0,{\mathrm{\eta }}_0^2).$

To be clear, in statistical terms, by “comparable” claims or cases, we mean to suggest that their awards somehow arise from the same distribution.

Now, if we know the global mean (${\mathrm{\lambda }}_0$) and variability (${\mathrm{\eta }}_0^2$) associated with this population, the shrinkage estimator is ${\hat{\mathrm{\mu }}}_y^s=\frac{Y/{\mathrm{\sigma }}_y^2+{\mathrm{\lambda }}_0/{\mathrm{\eta }}_0^2}{1/{\mathrm{\sigma }}_y^2+1/{\mathrm{\eta }}_0^2}.$

Similar to eq. [1], we know that the risk of this estimator is $({\mathrm{\sigma }}^{-2}+{\mathrm{\eta }}_0^{-2}{)}^{-1}$, which is smaller than ${\mathrm{\sigma }}^2$, the risk associated with the classical estimator $Y$ to estimate ${\mathrm{\mu }}_y$. Notice, however, that this shrinkage estimator requires knowledge of the values of ${\mathrm{\lambda }}_0$ and ${\mathrm{\eta }}_0$. A more realistic scenario is one in which we do not know these values and instead need to estimate them based on prior awards. Assume that the set of prior cases identified involves awards $X_1,\dots ,X_N$ centered at the correct awards ${\mathrm{\mu }}_1,\dots ,{\mathrm{\mu }}_N$, respectively, with judgement variability ${\mathrm{\sigma }}^2$, which can be estimated, but for simplicity is assumed to be known. That is, $X_i\sim ({\mathrm{\mu }}_i,{\mathrm{\sigma }}^2),{\mathrm{\mu }}_i\sim ({\mathrm{\lambda }}_0,{\mathrm{\eta }}_0^2);i=1,\dots ,N.$

We can then use the following unbiased estimators to estimate ${\mathrm{\lambda }}_0$ and ${\mathrm{\eta }}_0^2$: ${\hat{\mathrm{\lambda }}}_0=\frac{{\sum }_{i=1}^N{X}_i}{N},{\hat{\mathrm{\eta }}}_0^2=\frac{{\sum }_{i=1}^N{(X_i-{\hat{\mathrm{\lambda }}}_0)}^2}{N-1}-{\mathrm{\sigma }}^2.$

And we can use the “empirical” shrinkage estimator, ${\hat{\mathrm{\mu }}}_y^{se}=\frac{Y/{\mathrm{\sigma }}_y^2+{\hat{\mathrm{\lambda }}}_0/{\hat{\mathrm{\eta }}}_0^2}{1/{\mathrm{\sigma }}_y^2+1/{\hat{\mathrm{\eta }}}_0^2},$

which converges to the (non-empirical) shrinkage estimator as $N\to \mathrm{\infty }$. As in Section 3, replacing ${\mathrm{\lambda }}_0$ and ${\mathrm{\eta }}_0$ with unbiased estimators introduces some uncertainty, and therefore some risk. However, as $N\to \mathrm{\infty }$, the risk converges to that of the (non-empirical) shrinkage estimator. Therefore, in terms of risk, we benefit from increasing the number of prior cases. If the set of prior cases is too small – in the sense that the risk resulting from the estimated values of ${\mathrm{\lambda }}_0$ and ${\mathrm{\eta }}_0$ dominates the benefits of the empirical shrinkage estimator – a court can increase sample size by expanding the substantive breadth of the prior cases.

In the following subsection, we discuss the breadth and alignment of prior cases. Although sample size is an important consideration, the accuracy benefits of shrinkage are fairly robust to sample size. Specifically, because the shrinkage estimator is influenced by sample size only through estimation of the hyperparameters (i.e., the mean and variance of the correct awards in the prior cases) and because these quantities can be estimated reasonably well with a small sample size, a sample of $5$ to $15$ often suffices. Furthermore, as discussed in detail below, the accuracy benefits of shrinkage are robust to misalignment; therefore, even if the hyperparameter estimation turns out to be relatively poor, shrinkage can generally be expected to improve accuracy. This is especially true in light of procedures courts use to prevent outlying awards, such as remittitur and appellate review. For these reasons, and for simplicity, although concern for sample size is implicit in our analysis, it does not play a central role in our examples and illustrations below.

We are interested in examining two considerations: breadth and alignment. Consider Judge Posner’s opinion in Jutzi-Johnson. Judge Posner disagreed with the prior cases identified by both the plaintiff and the defendant. He explained:

Jutzi-Johnson, 263 F.3d at 760. Judge Posner ultimately concluded that “[t]he parties should have looked at awards in other cases involving asphyxiation, for example cases of drowning, which are numerous.” Id.

In the language of the current section, Judge Posner disagreed with the alignment of the plaintiff’s cases, implying that awards corresponding to cases involving hours, rather than minutes, of pain and suffering would be inappropriately high. He disagreed with the breadth (and the alignment) of the defendant’s cases, suggesting that a set of cases involving the pain and suffering of inmates, rather than the general population, is too narrow, and that the defendant’s focus on inmates led to alignment issues that resulted in “absurd” conclusions. Additionally, Judge Posner seems to suggest that the sets of cases identified by the parties suffered from small sample sizes as well, indicating that broadening the prior cases to include other cases involving asphyxiation in the general population would have led to “numerous” cases. Id. (Bavli 2017)

Thus, consider again a claim that receives an award $Y$ “drawn from” a distribution centered at the correct award ${\mathrm{\mu }}_y$ with judgement variability ${\mathrm{\sigma }}_y^2$ (known), where ${\mathrm{\mu }}_y$ is centered at ${\mathrm{\lambda }}_0$ with variance ${\mathrm{\eta }}_0^2$. That is, $Y\sim ({\mathrm{\mu }}_y,{\mathrm{\sigma }}_y^2),{\mathrm{\mu }}_y\sim ({\mathrm{\lambda }}_0,{\mathrm{\eta }}_0^2).$

Assume that the court identifies a set of prior cases involving awards $X_1,\dots ,X_N$ with correct outcomes ${\mathrm{\mu }}_1,\dots ,{\mathrm{\mu }}_N$ centered at ${\mathrm{\mu }}_0$ with variance ${\mathrm{\tau }}^2$. Thus, $X_i\sim ({\mathrm{\mu }}_i,{\mathrm{\sigma }}^2),{\mathrm{\mu }}_i\sim ({\mathrm{\mu }}_0,{\mathrm{\tau }}^2);i=1,\dots ,N.$

Essentially, this means $X_i\sim ({\mathrm{\mu }}_0,{\mathrm{\psi }}^2)$ for $1\le i\le N$, where ${\mathrm{\psi }}^2={\mathrm{\sigma }}^2+{\mathrm{\tau }}^2$. Now, consider an estimator of the form ${\hat{\mathrm{\mu }}}_y^s=\frac{\frac{Y}{{\mathrm{\sigma }}_y^2}+\frac{{\mathrm{\mu }}_0}{{\mathrm{\psi }}^2}}{\frac{1}{{\mathrm{\sigma }}_y^2}+\frac{1}{{\mathrm{\psi }}^2}},$[8]

which we can approximate by plugging in unbiased estimators $\bar{X}=\frac{{\sum }_{i=1}^N{X}_i}{N}\mathrm{a}\mathrm{n}\mathrm{d}S=\frac{{\sum }_{i=1}^N{(X_i-\bar{X})}^2}{N-1}$

for the unknown parameters ${\mathrm{\mu }}_0$ and ${\mathrm{\psi }}^2={\mathrm{\sigma }}^2+{\mathrm{\tau }}^2$, respectively, to obtain: ${\hat{\mathrm{\mu }}}_y^a=\frac{\frac{Y}{{\mathrm{\sigma }}_y^2}+\frac{\bar{X}}{S}}{\frac{1}{{\mathrm{\sigma }}_y^2}+\frac{1}{S}}.$[9]

Again, as $N\to \mathrm{\infty }$, ${\hat{\mathrm{\mu }}}_y^a\to {\hat{\mathrm{\mu }}}_y^s$. And the risk of the shrinkage estimator ${\hat{\mathrm{\mu }}}_y^s$ (which is roughly equal to the approximation, assuming a reasonable sample size) is $R_y^s=\mathbb{E}({\hat{\mathrm{\mu }}}_y^s-{\mathrm{\mu }}_y{)}^2=\mathbb{E}{\left(\frac{\frac{Y}{{\mathrm{\sigma }}_y^2}+\frac{{\mathrm{\mu }}_0}{{\mathrm{\psi }}^2}}{\frac{1}{{\mathrm{\sigma }}_y^2}+\frac{1}{{\mathrm{\psi }}^2}}-{\mathrm{\mu }}_y\right)}^2=\mathbb{E}{\left(\frac{\frac{Y-{\mathrm{\mu }}_y}{{\mathrm{\sigma }}_y^2}+\frac{{\mathrm{\mu }}_0-{\mathrm{\mu }}_y}{{\mathrm{\psi }}^2}}{\frac{1}{{\mathrm{\sigma }}_y^2}+\frac{1}{{\mathrm{\psi }}^2}}\right)}^2$ $={\left(\frac{1}{{\sigma }_y^2}+\frac{1}{{\psi }^2}\right)}^{-2}\left[\frac{1}{{\sigma }_y^2}+\frac{{({\mu }_0-{\lambda }_0)}^2+{\eta }_0^2}{{\psi }^4}\right],$

which is smaller than ${\mathrm{\sigma }}_y^2$, the risk of the classical estimator, when $({\mathrm{\mu }}_0-{\mathrm{\lambda }}_0{)}^2+{\mathrm{\eta }}_0^2<2{\mathrm{\psi }}^2+{\mathrm{\sigma }}_y^2.$[10]

Let us consider the meaning of this condition and then examine a number of numerical examples to gain a deeper understanding of the circumstances necessary to improve accuracy. On the right side of eq. [10], ${\mathrm{\psi }}^2={\mathrm{\sigma }}^2+{\mathrm{\tau }}^2$ is the total variance (the sum of claim variability and judgment variability) of the prior awards; and ${\mathrm{\sigma }}_y^2$ is the judgment variability of the subject case. On the left side of eq. [10], $({\mathrm{\mu }}_0-{\mathrm{\lambda }}_0{)}^2$ is the square of the misalignment, the square difference between the expected correct award in the subject case and the mean of the correct awards in the prior cases; and ${\mathrm{\eta }}_0^2$ can be understood as the claim variability of the hypothetical population to which the correct award in the subject case belongs. For simplicity, we can set ${\mathrm{\eta }}_0^2=0$ and view the correct award in the subject case as its own population or as a realization of μ_y. Note that, although we employ this assumption throughout this section, our conclusions and illustrations herein are robust to reasonable alternatives and potentially substantial values of η such as τ. The condition is satisfied, for example, if (1) the judgment variability of the subject award is greater than the hypothetical claim variability (e.g., where ${\mathrm{\eta }}_0^2=0$), and (2) the breadth of the prior awards is greater than the misalignment of the prior awards. Of course, the effects of one factor can be offset by the effects of the other. For example, the effects of extreme misalignment can be offset by the effects of extreme judgment variability.

Furthermore, in general, the greater the dispersion of the prior awards, the more “tolerance” there is for misalignment. On the other hand, higher prior-award concentration requires greater alignment. Thus, it may be beneficial for the breadth of the prior awards to reflect the court’s confidence in their alignment with respect to the subject award.

Let us consider an example based on data obtained from Saks et al. (1997), which tested the effects, with respect to variability, of providing mock jurors with certain information regarding prior awards. In one set of control conditions in which mock jurors were provided with a fact pattern (based on actual personal injury cases) involving a “high-severity injury,” a broken back, the mean and standard deviation of the award amounts determined by participants were approximately $3 million and $4 million, respectively (Saks et al. 1997).

Note that these values are based on amounts determined by mock jurors rather than mock juries. Also, however, “[b]ecause the distribution for the raw dollar awards was highly variable and positively skewed, awards greater than two standard deviations above the mean were recoded to the amount at two standard deviations.” Id. The authors thereby limited the variability of the data.

Based on these data we construct Figure 2, which assumes a correct award (${\mathrm{\mu }}_y$) of $3 million and judgment variability (${\mathrm{\sigma }}_y$) of $4 million. It is intuitive to imagine an approximately “normal” distribution with almost all awards falling between 0 and $11 million (that is, the mean ±2 standard deviations). We assume also that ${\mathrm{\mu }}_y={\mathrm{\lambda }}_0=3$ million and ${\mathrm{\eta }}_0=0$. Thus, the figure illustrates risk as a function of the mean of the prior awards (indicating alignment), and displays different curves corresponding to different levels of prior-award variability and a shaded horizontal line corresponding to the risk of the classical estimator (which is not dependent on the mean or variability of the prior awards). We can see, for example, that if the prior awards are centered at $4 million with standard deviation equal to $2 million, we reduce risk by 92 % by using the shrinkage estimator rather than the classical estimator; and we can expect the award to fall within the interval $74,000 to $5.26 million (that is, the mean ±2 standard deviations), rather than $0 to $11 million. If the prior awards are centered at $5 million with standard deviation equal to $3 million, we reduce risk by 76.8 % relative to the classical estimator; and we can expect the award to fall within the interval $0 to $6.85 million. Although not shown in the figure, it can be shown that, for prior award mean and standard deviation equal to $8 million and $4 million, respectively, we reduce risk by 36 % relative to the classical estimator. Finally, if the distribution of prior awards has a mean and standard deviation equal to $3 million (the correct award) and $2 million, respectively, the variability (in terms of standard deviation) of the estimator is $0.8 million rather than $4 million, and we reduce risk by 96 % relative to the classical estimator.

Note that, although Saks et al. (1997) used mock jurors rather than juries, our choice of judgment variability – an important factor for whether prior-award information causes accuracy to increase or decrease – is likely conservative, since our choice ($4 million) reflects the methodology in that study whereby all award amounts above two standard deviations above the mean were reduced to the amount of two standard deviations above the mean. To be sure, however, let us illustrate an example in which we set judgment variability to half the standard deviation used above. Thus, Figure 3 assumes a correct award (${\mathrm{\mu }}_y$) of $3 million and judgment variability (${\mathrm{\sigma }}_y$) of $2 million. In this example, if prior awards are centered at $4 million with a standard deviation of $2 million, we reduce risk by 68.75 % relative to the classical estimator. If the distribution of prior awards is centered at $1 million with a standard deviation of $400,000, a distribution that is concentrated around a significantly incorrect award, we nevertheless reduce risk by 7.5 % relative to the classical estimator. Finally, if the distribution of prior awards has a mean and standard deviation equal to $3 million (the correct award) and $2 million, respectively, we reduce risk by 75 % relative to the classical estimator. Furthermore, if the variability in this final scenario were $200,000, the risk associated with the shrinkage estimator is even smaller – representing a risk reduction of 99.99 % (corresponding to a reduction in standard deviation from $4 million for the classical estimator to $20,000 for the shrinkage estimator).

Figure 2:

Comparison of the risk corresponding to the shrinkage estimator (black curves) and the risk corresponding to the classical estimator (gray horizontal line) plotted against the mean of the prior awards when the correct award is $\mathrm{\$}3$ million (vertical black line), and assuming ${\mathrm{\eta }}_0=0$ and the judgment variability of the subject case, ${\mathrm{\sigma }}_y$, is equal to $4 million. The black curves correspond to different values of prior-award variability, which ranges from $\mathrm{\$}0.2$ million to $\mathrm{\$}3$ million.

Figure 3:

Comparison of the risk corresponding to the shrinkage estimator (black curves) and the risk corresponding to the classical estimator (gray horizontal line) plotted against the mean of the prior awards when the correct award is $\mathrm{\$}3$ million (vertical black line), and assuming ${\mathrm{\eta }}_0=0$ and the judgment variability of the subject case, ${\mathrm{\sigma }}_y$, is equal to $2 million. The black curves correspond to different values of prior-award variability, which ranges from $\mathrm{\$}0.2$ million to $\mathrm{\$}3$ million.

Lastly, we construct a third example using data from Bovbjerg et al. (1988), which examined real award data by severity of injury to analyze the variability of awards for pain and suffering. The data presented in this example are arguably conservative as well, since 1) the authors excluded the 5 % of award values farthest from the median; 2) the data include reported incidents of additur and remittitur; and 3) the data reflect the value of dollars in 1987. Thus, in Figure 4, we consider the example of severity level 7 (out of 9, representing severe, but not maximum-severity, injuries), with mean and standard deviation values of approximately $2 million and $2 million. The graph again displays different curves corresponding to different levels of prior-award variability and a shaded horizontal line representing the risk associated with the classical estimator at $4 million squared, which is equal to the standard deviation squared. First, we see that if prior awards are centered at $2 million (assumed to be the correct award) with a standard deviation of $500,000, we reduce risk by 99.65 % relative to the classical estimator. If the prior awards have a mean and standard deviation equal to $500,000 and $500,000, we reduce risk by 49.83 % – a milder reduction, due to an introduction of bias (but a reduction nevertheless). If the prior awards have a mean and standard deviation equal to $500,000 and $1.5 million, respectively, we reduce risk by 64 % – an improvement relative to the former scenario, due to the increase in breadth, which reduces the impact of the bias. If the prior awards have a mean and standard deviation equal to $4.2 million and $200,000, respectively, we increase risk by 18.63 %, since we have a tightly bound distribution centered at a significantly incorrect award value. On the other hand, if the prior awards have a mean and standard deviation equal to $4.2 million and $1.5 million, we reduce risk by 37.48 %, since, now, the introduction of bias is reduced due to high prior-award breadth, and the beneficial effect of the prior awards on award variability dominates.

Figure 4:

Comparison of the risk corresponding to the shrinkage estimator (black curves) and the risk corresponding to the classical estimator (gray horizontal line) plotted against the mean of the prior awards when the correct award is $\mathrm{\$}2$ million (vertical black line), and assuming ${\mathrm{\eta }}_0=0$ and the judgment variability of the subject case, ${\mathrm{\sigma }}_y$, is equal to $2 million. The black curves correspond to different values of prior-award variability, which ranges from $\mathrm{\$}0.2$ million to $\mathrm{\$}4$ million.

Thus, using the derivations and illustrations above, we state the following conclusions regarding the alignment and breadth of prior awards:

1. Prior awards that are relatively aligned with the correct award in the subject case can lead to large accuracy benefits. These benefits are robust to changes in the alignment and breadth of the prior awards.

2. Prior awards that are misaligned – even significantly misaligned – but have relatively high breadth can lead to accuracy benefits, but benefits that are small relative to those that result from prior awards that are aligned and have lower breadth.

3. Increasing only the breadth of the prior awards (without affecting alignment) will generally not harm accuracy, but will reduce the influence of the prior awards, and therefore reduce their benefits with respect to accuracy. Considerations for determining an appropriate breadth include the sample size and the court’s confidence in the alignment of the prior awards.

4. Prior awards that are significantly misaligned and have low breadth can lead to harmful effects on accuracy. However, such effects generally require the unusual circumstance of tightly bound prior awards that are significantly misaligned.

In short, under relatively mild conditions, the shrinkage estimator outperforms the classical estimator, an adjudicated award.