Statistical Inference for Data Adaptive Target Parameters

1.1 Motivating example

The general methodology for estimation and inference for data adaptive parameters is presented below, however, we illustrate the method by a particularly challenging causal inference estimation problem. Consider data from the (W)estern (C)collaborative (G)roup (S)tudy [4], a prospective study of risk factors of coronary heart disease (CHD). The study consisted of 3,524 males (3,142 of which had complete data) aged 39–59, working in certain California corporations, who were enrolled at the outset of the study and followed for 8.5 years. Our goal is to estimate the impact on CHD from applying a treatment rule regarding cholesterol that is learned from the data. Define the data of interest to be $O=(W,A,Y),$ where W are a vector of confounders, A the treatment of interest (say cholesterol level) which is dichotomized, so that $A=I(Chol>\gamma ),$ where $\gamma$ is a target intervention level; Y is the indicator of a CHD event within the study period. Let ${\bar{Q}}_0(A,W)\equiv E_0(Y|A,W).$ We start with the ambitious goal of estimating a treatment rule for intervening on cholesterol that targets only those people that would be helped by such an intervention. To do so, we start by defining so-called counterfactuals [5], Y_a, which represents the outcome for a subject if, possibly contrary to fact, the subject had level $A=a.$ This leads to a notion of potentially “full” data that includes counterfactuals, or in this case, $X=(W,A,Y_1,Y_0).$ Our goal is to estimate the impact of an intervention rule that lowers a subjects cholesterol (from $A=1$ to $A=0$) only if such a change improves the CHD outcome (that is, lowers cholesterol if $Y_1=1$ and $Y_0=0$ and $A=1$), or:

We will discuss several specific examples below, but note that the sequence of analyses often used in large scale omic studies (genomics, proteomics, metabolomics, etc.; Zhang and Chen [7], Berger et al. [8]) can be the result of a series of suggested patterns that lead to further analyses not previously considered, e. g., multiple testing, to clustering, to exploration of pathways, to more targeted analyses all with the data from the same experiment. In these cases, inference that ignores that the parameters were derived data adaptively will typically be biased. Others have noted the particular dangers of high dimensional data combined with flexible methodologies to generate excessive false positive findings (Ioannidis [9], Broadhurst and Kell [10]). In many cases, even when the best intentions are to stick to a pre-specified data analysis plan, there can be feedback from the data and models chosen (e. g., covariates dropped, different basis functions tried, unplanned sub-group analyses conducted, etc.; Barraclough and Govindan [11], Marler [12]). Thus, it is important to have methods that allow such exploration and also transparent interpretation of the resulting estimates. Though there are advantages for pre-specifying the algorithm used to generate the parameter(s), the general methodology does not even require one to – that is, one can derive inference for methods of deriving patterns, even when the precise methods used to generate the parameters are not known. Thus, it can be applied in circumstances where there is little constraint on how the data is explored to generate potential parameters of interest for estimation and inference.

2.1 Splitting the sample, but using the whole sample to fit the data adaptively generated target parameter

In the above Theorem 1, one need not assume Donsker class conditions, so that the target-parameter choices ${\Psi }_{B_n,P_{n,B_n}^0}$ could be arbitrarily dependent on the data $P_{n,B_n}^0.$ However, now consider an estimator ${\psi }_n^1\equiv E_{B_n}{\hat{\Psi }}_{B_n,P_{n,B_n}^0}\left(P_n\right)$ of the same “estimand” ${\psi }_{0,n}$ but which uses the entire sample as the estimation sample for each of the V parameter-generating samples. The asymptotics will now rely on stronger assumptions, but if the algorithm generating the target parameter and estimator is different across splits, and the stronger assumptions are satisfied, then the estimator is generally more efficient than the algorithm based on theorem 1.

Theorem 2

As above assume that conditional on$(B_n,P_{n,B_n}^0),{\hat{\Psi }}_{B_n,P_{n,B_n}^0}$is asymptotically linear with influence curve $I{C}_{B_n,P_{n,B_n}^0}\left(P_0\right)$ so that

We also assume that $I{C}_{\upsilon ,P_{n,\upsilon }^0}\left(P_0\right)$ falls in a P₀-Donsker class with probability tending to 1.

Then,

The relative efficiency of the two estimators ${\psi }_n$ and ${\psi }_n^1$ is of course based on the two corresponding asymptotic variances

2.2 Using the whole sample to generate the target parameter and to subsequently estimate it: no sample splitting

Consider a mapping $P_n\to ({\Psi }_{P_n},{\hat{\Psi }}_{P_n})$ from a sample to a target parameter mapping ${\Psi }_{P_n}:M\to R$ and corresponding estimator ${\hat{\Psi }}_{P_n}:M_{NP}\to IR.$ The estimand of interest is now ${\Psi }_{P_n}\left(P_0\right)$ and it is estimated with ${\psi }_n^2={\hat{\Psi }}_{P_n}\left(P_n\right).$ The possible advantage of this approach is that the estimand is a single parameter instead of an average over splits of sample-split-specific estimands, and the latter might be harder to interpret. However, as in the previous subsection, stronger conditions are needed to establish the desired asymptotic consistency and normality. In contrast to the method of the previous subsection, in which we only changed the estimator, we now actually changed the estimand as well.

Theorem 3

Assume${\hat{\Psi }}_P\left(P_n\right)$is an asymptotically linear estimator of${\Psi }_P\left(P_0\right)$at P₀with influence curve IC_P (P₀) uniformly in the choice of parameter P in the following sense:

Again, this estimator ${\psi }_n^2$ is as efficient as the oracle estimator ${\hat{\Psi }}_{P_0}\left(P_n\right)$ as an estimator of ${\Psi }_{P_0}\left(P_0\right),$ discussed above, but one should note again that its efficiency is measured relative to a different target ${\Psi }_{P_n}\left(P_0\right)$ instead of ${\Psi }_{P_0}\left(P_0\right).$ Since the parameter ${\Psi }_{P_0}$ is unknown while ${\Psi }_{P_n}$ is a known target parameter mapping, one might often find the parameter ${\Psi }_{P_n}\left(P_0\right)$ more tangible than ${\Psi }_{P_0}\left(P_0\right),$ and thus perhaps easier to interpret. In essence, theorem 3 provides the conditions necessary for consistent inference of a “data-dredging” algorithm; using the same data for generating the parameter of interest, and deriving its inference.

Note, that others have examined the asymptotic of such a procedure (no sample splitting) using different theoretical approaches. For instance, Dwork et al. [3] presents asymptotic consistency for estimating a number (m) of data-adaptively derived functions of the data-generating distribution as a function of m and sample size, n.

3.1 Inference for the sample-split conditional risk of a data adaptive regression estimator

The set-up is identical to Dell et al. [13], Dudoit and van der Laan [14]. Let $O=(W,Y)\sim P_0,$ where W is a vector of input-variables and Y is an outcome one wants to predict; P₀ is composed of the distribution of $Y|W,Q_0\left(W\right)$ and the distribution of W, Q_0,W. Let $\hat{\overline{Q}}$ be an estimator of the true regression function ${\bar{Q}}_0=E_0\left(Y|W\right);$ let ${\bar{Q}}_{P_{n,B_n}^0}$ be the corresponding estimate of ${\bar{Q}}_0=E_0\left(Y|W\right)$ based on the parameter-generating sample $P_{n,B_n}^0.$ The target parameter generated by $P_{n,B_n}^0$ is defined as the mean squared error ${\Psi }_{P_{n,B_n}^0}\left(P_0\right)=E_0(Y-{\bar{Q}}_{P_{n,B_n}^0}(W){)}^2$ or, in general, as the loss-function specific risk $E_{0}L\left({\bar{Q}}_{P_{n,B_n}^0}\right)(W,Y)$ for some loss function $L\left(\bar{Q}\right)$ satisfying ${\bar{Q}}_0=arg\underset{\bar{Q}}{min}{E}_{0}L\left(\bar{Q}\right).$

The estimator of ${\Psi }_{P_{n,B_n}^0}\left(P_0\right)$ based on the estimation sample $P_{n,B_n}^1$ is defined as its empirical counterpart ${\hat{\Psi }}_{P_{n,B_n}^0}\left(P_{n,B_n}^1\right)=P_{n,B_n}^{1}L\left({\bar{Q}}_{P_{n,B_n}^0}\right).$ Conditional on the sample $P_{n,B_n}^0,$ this estimator ${\hat{\Psi }}_{P_{n,B_n}^0}\left(P_{n,B_n}^1\right)$ is asymptotically linear with influence curve $L\left({\bar{Q}}_{P_{n,B_n}^0}\right)-P_{0}L\left({\bar{Q}}_{P_{n,B_n}^0}\right)$ with no remainder. The average across sample-split data adaptive target parameters is thus defined as ${\psi }_{n,0}=E_{B_n}{P}_{0}L\left({\bar{Q}}_{P_{n,B_n}^0}\right)$ and its corresponding estimators are ${\psi }_n=E_{B_n}{P}_{n,B_n}^{1}L\left({\bar{Q}}_{P_{n,B_n}^0}\right),{\psi }_n^1=E_{B_n}{P}_{n}L\left({\bar{Q}}_{P_{n,B_n}^0}\right),$ and ${\psi }_n^2=P_{n}L\left({\bar{Q}}_{P_n}\right).$ Theorem 1 implies that if the loss function chosen is uniformly bounded and the estimator $\hat{\overline{Q}}\left(P_n\right)$ is consistent for a limit $\bar{Q}$ (not necessarily ${\bar{Q}}_0$), then ${\psi }_n-{\psi }_{n,0}$ is asymptotically linear with influence curve $L\left(\bar{Q}\right)-P_{0}L\left(\bar{Q}\right),$ the same influence curve as the estimator $P_{n}L\left(\bar{Q}\right)$ of $P_{0}L\left(\bar{Q}\right)$ treating $\bar{Q}$ as known. This allows us to construct a confidence interval for the true conditional risk ${\psi }_{n,0},$ under these very weak conditions. In particular, the estimator $\hat{\overline{Q}}$ can be a highly data adaptive super learner van der Laan et al. [15].

Similarly, Theorem 2 implies a formal result for ${\psi }_n^1,$ but now $L\left({\bar{Q}}_{P_n}\right)$ has to be an element of a P₀-Donsker class with probability tending to 1, putting some constraints on how adaptive ${\bar{Q}}_{P_n}$ can be. Under the same conditions, we will have that ${\psi }_n^2=P_{n}L\left({\bar{Q}}_{P_n}\right)$ is an asymptotically linear estimator of $P_{0}L\left(\bar{Q}\right)$ with the same influence curve ${\psi }_n$ Even though these conditions might be satisfied for ${\bar{Q}}_n,$ the estimator ${\psi }_n^2$ is known to be wrong for the sake of using $P_{n}L\left({\bar{Q}}_{P_n}\right)$ to select among a collection of candidate estimators of ${\bar{Q}}_0$ since this estimator of risk will favor over-fitted estimators. Nonetheless, if the goal is to obtain confidence intervals for the asymptotic risk $P_{0}L\left({\bar{Q}}_{P_n}\right)$ of an estimator ${\bar{Q}}_{P_n},$ then this method could be considered.

3.2 Inference for sample-split subgroup-specific causal effect, where the subgroups are data adaptively determined

Consider “discovering” sub-groups within the target population that have unique relationships with explanatory variable of interest (e. g., drug treatment, environmental exposure, etc.). In the case that these sub-groups are not defined apart from the data, post-hoc sub-group analysis is typically treated as purely explanatory and thus the statistical inference inherently awed, typically anti-conservatively. However, the approach we have outlined provides explicit framework for aggressively searching for interesting sub-groups, but still allows for consistent statistical inference for the resulting estimators of association parameters.

Suppose that we observe on each subject $O=(W,A,Y),$ where W are baseline covariates, A is a binary treatment, and Y a final outcome. Thus we observe n i.i.d. copies $O_1,\dots ,O_n,$ and consider an algorithm that maps a data set $O_1,\dots ,O_n$ into a subgroup $W\to C\left(W\right)\in \{0,1\},$ where $C\left(W\right)=1$ indicates membership in the subgroup. Denote this subgroup-estimator with $\hat{C}:M_{NP}\to C,$ where $C$ is the space of functions that map a W into a binary indicator. Given a realized subgroup C, let ${\Psi }_C:M\to IR$ be a desired parameter of interest such as the W-controlled effect of treatment A on Y for subgroup C, defined as

Let ${\hat{\Psi }}_C:M_{NP}\to IR$ be an estimator of ${\Psi }_C\left(P_0\right),$ again where one could choose among several different estimators (including the IPTW; Robins et al. [16]), but we focus on TMLE [17, 18]: note that this is just the targeted maximum likelihood estimator for the W-controlled effect of treatment but applied to the subsample $\{i:C(W_i)=1\}.$ Assume that the regularity conditions hold so that this TMLE ${\Psi }_C\left(P_n\right)$ is asymptotically linear with influence curve IC_C(P₀):

Define ${\Psi }_{P{{{}_n^0}_{,B}}_{{}_n}}:M\to IR$ as ${\Psi }_{P{{{}_n^0}_{,B}}_{{}_n}}={\Psi }_{\hat{C}\left(P{{}_{n,}^0}_{B_n}\right)},$ i. e., the W-controlled effect of treatment on the outcome for the data adaptively determined subgroup $\hat{C}\left(P_n^0{{}_{,}}_{B_n}\right).$ Similarly, we define ${\hat{\Psi }}_{P{{{}_n^0}_{,}}_{B_n}}:M_{NP}\to IR$ as ${\hat{\Psi }}_{P{{}_{n,}^0}_{B_n}}={\hat{\Psi }}_{\hat{C}\left(P{{}_n^0}_{,}{{}_B}_{{}_n}\right)},$ i. e. the TMLE of the W-controlled effect of treatment on the outcome for this data adaptively determined subgroup, treating the latter as given. The estimand of interest is thus defined as ${\psi }_{n,0}=E_{B_n}{\Psi }_{P{{}_n^0}_{,B_n}}\left(P_0\right)$ and its estimator is ${\psi }_n=E_{B_n}{\hat{\Psi }}_{P_n^0{}_{,B_n}}\left(P_{n,}^1{}_{B_n}\right).$ That is, for a given split B_n, we use the parameter-generating sample $P_{n,B_n}^0$ to generate a subgroup $\hat{C}\left(P_{n,B_n}^0\right)$ and corresponding TMLE of ${\hat{\Psi }}_{\hat{C}\left(P_{n,B_n}^0\right)}\left(P_0\right)$ applied to the estimation-sample $P_{n,B_n}^1,$ and these sample-split specific estimators are averaged across the V sample splits. By assumption we have for each split B_n

Under the Donsker class condition on $I{C}_{\hat{C}\left(P_{n,B_n}^0\right)}\left(P_0\right)$ we can also establish the formal results for ${\psi }_n^1=E_{B_n}{\hat{\Psi }}_{\hat{C}\left(P_{n,B_n}^0\right)}\left(P_n\right)$ of ${\psi }_{n,0,}$ and the estimator ${\psi }_n^2={\hat{\Psi }}_{\hat{C}\left(P_n\right)}\left(P_n\right)$ of ${\Psi }_{\hat{C}\left(P_n\right)}\left(P_0\right),$ respectively.

4.1 Risk estimation of a data adaptive prediction algorithm

More background on the conditional risk parameter is in Section 3.1, and in this case the goal is estimation and inference regarding the “fit” of a machine learning algorithm. The data is $O=(Y,W),$ for outcome Y, predictor W, where $W\sim N(0,{\sigma }_W^2=4),{\bar{Q}}_0\left(W\right)\equiv E_0\left(Y|W\right)$ is shown in Figure 1, based on a piecewise constant model, and $Y|W\sim N\left({\bar{Q}}_0\right(W),{\sigma }_Y^2=0.25).$ For the $\upsilon -th$ parameter-generating sample, we fit the regression with an ensemble stacking algorithm, called the SuperLearner (SL; van der Laan et al. [15]), resulting in a convex combination of a variety of algorithms ranging from very smooth to highly data adaptive: linear model, stepwise regression based on AIC (stepAIC; Venables and Ripley [19]), Bayesian glm (linear) model (bayesglm; Gelman et al. [20]), generalized additive model with smooth term for covariate Hastie and Tibshirani [21]; neural nets (nnet; Venables and Ripley [19]); and a simple null model (sample average of outcome). For the $\upsilon -th$ parameter-generating sample the data adaptive parameter of interest was defined as the conditional risk (mean squared error; MSE), conditional on the fitted prediction function: ${\Psi }_{P_{n,B_n}^0}\left(P_0\right)\equiv E_0\left[(Y-{\bar{Q}}_{P_{n,B_n}^0}(W){)}^2\right]$ is the true expected squared error loss of SL fit (based on parameter-generating sample, $P_{n,B_n}^0$) and estimated using corresponding validation sample: ${\hat{\Psi }}_{P_{n,B_n}^0}\left(P_{n,B_n}^1\right)=E_{P_{n,B_n}^1}\left[(Y-{\bar{Q}}_{P_{n,B_n}^0}(W){)}^2\right].$ Thus, the estimand is the risk averaged over the V estimation samples: $\Psi P_n\left(P_0\right)=E_{B_n}{\Psi }_{P_{n,B_n}^0}\left(P_0\right)$ and the corresponding estimator is $\hat{\Psi }\left(P_n\right)=E_{B_n}{\hat{\Psi }}_{P_{n,B_n}^0}\left(P_{n,B_n}^1\right).$ Finally, inference is derived based on (4) above, where the estimated influence curve for the v-th estimation sample is given by

• Download Figure
• Download figure as PowerPoint slide

True model ${\bar{Q}}_0\left(W\right)$ for simulations of conditional risk estimation.

Citation: The International Journal of Biostatistics 12, 1; 10.1515/ijb-2015-0013

4.1.1 Results

We examined the empirical distribution of the standardized differences, $({\psi }_n-{\psi }_{n,0})/se\left({\psi }_n\right)$ for the risk. We observe minimal departure from normality (Figure 2), and nearly perfect coverage probability of the confidence intervals, for all sample sizes for algorithms 1 and 2 (see Table 1).However, one can see that algorithm 3, though resulting in a lower “average” risk, results in biased inference (and non-normal sampling distribution for the relatively modest sample sizes. This implies the Donsker conditions are not met. by a sample size of $n=1000,$ though there is still some standardized bias, the coverage probability is nearly perfect. Thus, even for a highly adaptive algorithm, where overfitting (underestimation of risk) seems particularly troublesome, at modest sample sizes, one begins achieving the conditions of theorem 3.

• Download Figure
• Download figure as PowerPoint slide

Distribution of $({\psi }_n-{\psi }_{n,0})/se\left({\psi }_n\right)$ (for $n=100,500,1000$ with N(0, 1) distribution for three algorithms (1, 2 and 3, corresponding to the 3 rows). Dark line represents the mean of these standardized values, so the difference between it and 0 is the standardized bias.

Citation: The International Journal of Biostatistics 12, 1; 10.1515/ijb-2015-0013

Table 1:

Simulation results for risk estimation for data adaptive prediction on theorems 1–3 (${\psi }_n,{\psi }_n^1,{\psi }_n^2,$ respectively). Coverage probability is for a nominal 95 % CI.

N	Methods	Average true parameter	Average estimated	Cov. Prob.
$n=100$	${\psi }_n$	0.55	0.55	0.93
	${\psi }_n^1$	0.54	0.54	0.92
	${\psi }_n^2$	0.45	0.40	0.78
$n=500$	${\psi }_n$	0.55	0.55	0.94
	${\psi }_n^1$	0.55	0.55	0.94
	${\psi }_n^2$	0.46	0.46	0.92
$n=1000$	${\psi }_n$	0.40	0.40	0.95
	${\psi }_n^2$	0.40	0.41	0.95
	${\psi }_n^2$	0.36	0.36	0.93

We also examined the same procedure for estimating the risk difference using algorithm 2. In this case, we observe slower convergence, but still relatively good coverage for an estimate that is particularly sensitive to over-fitting.

4.2 Average treatment effect for given prediction model

Average Treatment Effect, or ATE, is commonly the parameter of interest in applications of causal inference methods (Rubin [5]). We use the same set-up as we did for the example in the introduction, but for the ATE: $E(Y_1-Y_0),$ where (Y₁, Y₀) are the counterfactual outcomes for an individual unit if they have $A=1$ and $A=0,$ respectively. Consider n i.i.d. observations of $O=(W,A,Y)\sim P_0,$ where Y is an outcome, A is a binary treatment of interest, and W a set of potential confounders. Under the assumption, the ATE equals the following statistical estimand: