The stratum-specific treatment effect function is a random variable giving the average treatment effect (ATE) for a randomly drawn stratum of potential confounders a clinician may use to assign treatment. In addition to the ATE, the variance of the stratum-specific treatment effect function is fundamental in determining the heterogeneity of treatment effect values. We offer a non-parametric plug-in estimator, the targeted maximum likelihood estimator (TMLE) and the cross-validated TMLE (CV-TMLE), to simultaneously estimate both the average and variance of the stratum-specific treatment effect function. The CV-TMLE is preferable because it guarantees asymptotic efficiency under two conditions without needing entropy conditions on the initial fits of the outcome model and treatment mechanism, as required by TMLE. Particularly, in circumstances where data adaptive fitting methods are very important to eliminate bias but hold no guarantee of satisfying the entropy condition, we show that the CV-TMLE sampling distributions maintain normality with a lower mean squared error than TMLE. In addition to verifying the theoretical properties of TMLE and CV-TMLE through simulations, we highlight some of the challenges in estimating the variance of the treatment effect, which lack double robustness and might be biased if the true variance is small and sample size insufficient.
1 Introduction and background
The stratum-specific treatment effect function (also referred to as the CATE function for conditional average treatment effect [ATE]) is a random variable defined as the ATE for a randomly drawn stratum of patient characteristics, which are confounders (potential causal parents of treatment) in that they possibly affect the outcome and occur before treatment assignment. The average of the CATE function is the ATE. However, beyond the mean effect, of interest is also the variation in this effect across the strata of potential confounders. One measure of this variation is simply the variance of the CATE function (as a random function of confounders), or variance of treatment effects (VTEs). For instance, though the ATE might indicate that a treatment is beneficial on average, a comparatively large VTE would indicate there might be a significant portion of the population that has deleterious effects from treatment. We construct the targeted maximum likelihood estimator (TMLE) [1,2] and the cross-validated TMLE (CV-TMLE)  to simultaneously estimate ATE and VTE, corresponding to the sample mean and variance of CATE function estimates. We also provide conditions under which the estimators are asymptotically efficient.
Our implementation of both the TMLE and CV-TMLE fits under the general framework of targeted learning  based on the use of ensemble machine learning. We will see the CV-TMLE has an advantage over the TMLE in that it does not require a Donsker (entropy) condition on the initial predictions for the relevant nuisance parameters, enabling more flexibility in the ensemble learning we employ. The inference method proposed in this study relies on a sample standard deviation of the efficient influence curve approximation, thereby allowing substantial savings in computational time over a re-sampling based (bootstrap) approach. In fact, we show in Section 2.3 that without the targeting step in our estimator, employing the non-parametric bootstrap might not guarantee valid inference .
We also discuss the limitations of our VTE estimator. Despite clear advantages of targeted learning, we will see that estimating VTE lacks the desirable robustness properties present when estimating the ATE. Particularly for a randomized trial, although our CV-TMLE estimate for ATE is consistent, knowing the treatment mechanism does not guarantee consistent VTE estimates. This lack of robustness is due to a second-order remainder term that can make coverage unreliable. We offer future improvements to address this problem. Finally, we also discuss limitations in detecting VTE when the true VTE is small relative to the sample size.
Let us define as a random variable, which is a counterfactual outcome drawn from a population under intervention to set treatment to as per the Neyman–Rubin potential outcomes framework [5,6]. The variance in counterfactual treatment effect from individual to individual can be defined as . However, this parameter is not generally identifiable, a problem first addressed by Neyman (1923) when attempting to compute the standard errors for the counterfactual average difference in yield of two crop varietals. Fisher (1951) also realized the difficulty in directly estimating the joint distribution of and and suggested pairing a treated and untreated varietal in the same pot might closely resemble a set of counterfactual differences, . Strong assumptions must be made to identify in the event it is not 0, such as assuming that for any given strata of patient covariates, , the quantiles of and are the same . Heckman (1997) mentions combining the results of Cambanis (1976) and Frechet (1951) to tail bound as a means to test if the lower bound of the confidence interval includes a variance of 0 [8,9,10]. For randomized trial data, Ding, Feller et al. (2016), construct a Fisher randomization test of the null hypothesis that under the untestable assumption that there exists a universal so that . However, this approach is not adapted to the situation where treatment is assigned based on confounders .
Cox (1958) assumes for predefined homogeneous subgroups, essentially assuming all the variation of is due to known variables that affect the response to treatment, i.e., patient characteristics used to assign treatment . Although we do not make this assumption, this is the variation we aim to capture by estimating , i.e., the variance of the CATE function, where is a vector of confounders. As an illustration, let be an indicator of gender, and be a binary outcome indicating survival if the outcome is 1. Suppose males have a treatment effect value of and the females, . Assuming males and females are of equal proportion for the population at hand, then the VTE is 0.25 and ATE is 0.2, thus giving a sense of the spread of CATE function values for which one could generally use Chebyshev’s inequality to tail bound. VTE does not capture the variance of the random variable, . Hence, a clinician’s perception of highly varying outcomes does not mean the VTE is high. Rather, one would want to estimate VTE to see if the varying outcomes were due to measured confounders.
Estimating VTE with a substitution estimator depends on using an estimate of the outcome model, , from which to estimate the CATE function, , and then its variance over an estimate of the distribution of , in our case the empirical distribution of (see Section 2.3). If one knows the CATE function, for instance, one would know an optimal rule for assigning treatment based on confounders . One could also find subgroup-specific treatment effects. Tian et al. (2014) proposed a way to isolate interactions of treatment with confounders in a randomized trial by transforming the predictors of a parametric model . The main idea is to form a variable, , where is the usual treatment indicator, and then put the interaction of this variable with the predictors in the outcome regression. This enables direct estimation of the CATE function from which one could obtain a point estimate of the VTE. One could also employ recursive partitioning  or random forests  to establish homogeneous (constant treatment effect) subgroups of the data. Such subgroups could be used to compute the VTE but as noted in ref. , establishing too coarse subgroups can miss detecting treatment effect heterogeneity. Instead, we are proposing to use TMLE or CV-TMLE to estimate the CATE function (and hence, the VTE) without limiting to tree-based or parametric modeling.
For simplicity, we consider only binary outcome but all the analyses here-in can be extended to bounded continuous outcomes. For the latter, we can scale the outcome to be in via the transformation , where and are minimum and maximum outcomes, respectively, obtained from the data or known a priori. We denote the mean outcome model as , where , and are the random variables for the data, i.e., confounders, treatment and outcome, respectively, and the data generating distribution, . For binary outcome or continuous outcome scaled as above, the mean loss given by will be minimized at the true outcome model, [18,19]. Thus, for a scaled continuous outcome as described above or binary outcome, the above loss is valid and, therefore, the targeting step (see Sections 2.2 and 2.3) is identical. After the CV-TMLE or TMLE algorithm is complete, one may convert the outcomes back to their original scale , and thus can form confidence intervals for ATE and VTE in the original outcome scale.
2.1 Full data statistical model and the link to the observed data
Our full data, including unobserved measures, are assumed to be generated according to the following structural equations [21,22,23]. We can assume a joint distribution, , an unknown distribution of unmeasured variables. are the measured variables. In the time ordering of occurrence we have where is a vector of confounders, , where is a binary treatment and , where is the outcome. Thus, we define a distribution , via .
Under this framework, is a random outcome under , where we intervene on the structural equations to set treatment to , i.e., . The full model, , consists of all possible . The observed data model, , is linked to in that we observe , where is generated by according to the structural equations above. Our true observed data distribution is an element of the non-parametric statistical model, . In the case of a randomized trial or if we have some knowledge of the treatment mechanism, is considered a semi-parametric model and we will incorporate such knowledge. We will show that this knowledge is more helpful for estimating ATE than for VTE.
We define the stratum-specific treatment effect function or CATE function as . Our parameter of interest is a mapping from to defined by . We will impose the randomization assumption [22,23] on which, states , as well as the positivity assumption on so that for all and . Defining yields and we can identify the parameter of interest as a mapping from the observed data model, , to via the g-computation formula (for ATE)  and taking the variance of . That is, , i.e., ATE and VTE, respectively.
2.2 TMLE conditions and asymptotic efficiency
The efficient influence curve at a distribution, , for the parameter mapping, , is a function of the observed data, , notated as . Its variance gives the generalized Cramer–Rao lower bound for the variance of any regular asymptotically linear estimator . In the present case, we consider a two-dimensional efficient influence curve, , where the two components are the efficient influence curves for ATE and VTE, respectively.
We will employ the notation, , to be the empirical average of function, , and to be . We will only consider the case in which our observed data consists of independent identically distributed draws from the true data distribution, . We also define a loss function, , which is a function of the observed data, , and indexed at an element, , of the observed data model such that is minimized when . The TMLE procedure maps an initial estimate, , of the true data generating distribution to such that and such that . is called the TMLE of the initial estimate . We can then write an expansion with second-order remainder term, , as follows: . The TMLE is notated as , since it is a substitution estimator (or plug-in estimator). This ensures our estimates will be within the parameter bounds for VTE, i.e., greater than or equal 0.
2.2.1 Conditions for asymptotic efficiency
Let be the norm. We assume the following TMLE conditions:
is in a P-Donsker class for all . Note that this condition can be dropped in the case of using CV-TMLE .
Second-order remainder condition: is for all .
for all .
are guaranteed to be as small as possible for any regular asymptotically linear estimator, where , the user chosen significance level, is the standard normal, and is the sample standard deviation of . Furthermore, if the TMLE conditions hold for the initial estimate, , then they will also hold for the updated model, , thereby motivating the importance of using ensemble machine learning in constructing .
2.2.2 Remainder term in VTE estimation
We define the true outcome model and the true treatment mechanism , which we assume to be bounded away from 0 and 1. Let be the initial estimate of , and be the estimate for . For estimating ATE and VTE, we will fluctuate an initial outcome model fit, , to (see Section 2.3) but will not change as in the study of van der Laan and Rubin (2006). We define the true CATE function, , via and as its estimate. The second-order remainder term for VTE, notated as , is given by:
which is computed in Theorem A.2 in the Appendix. Due to equation (3) it is necessary and thus VTE estimation is not doubly robust. ensures the necessary condition and is sufficient to satisfy condition 2 of Section 2.2.1 if we know , as in a randomized trial. If we do not know , then it is sufficient if , . We therefore rely upon ensemble machine learning and its associated oracle inequality [29,30] to mitigate the remainder term bias, especially from equation (3), which can lead to poor coverage.
2.3 One-step CV-TMLE algorithm for ATE and VTE
The one-step TMLE algorithm  constructs a parametric submodel through the initial estimate, , of the true distribution, . The submodel is indexed by a single dimensional parameter, and given by where . The construction is performed recursively in such a way that we arrive in one step at an element of the submodel, , which has minimum empirical average loss of all the elements and also solves the efficient influence curve equation, (in our case, , but this holds for general dimension).
The efficient influence curve for , i.e., ATE and VTE, has two components given by
where , and are confounders, binary treatment indicator and either binary outcome or a continuous outcome scaled between 0 and 1, respectively, and . We refer the reader to Appendix A for details on the derivation. In our ensuing algorithm, if there is no heterogeneity identified in the data, that is, the estimated CATE function is identically 0, then the estimator will yield 0 as an estimate of the VTE with no confidence interval at all. So, for our algorithm to yield a confidence interval there must be some heterogeneity detected.
In the algorithm below, we adjusted the original CV-TMLE procedure  for ease of computation without losing any theoretical properties or finite sample performance. Once we obtain initial estimates, there is no difference between CV-TMLE and TMLE as far as implementation is concerned. The reader may consult the technical report by Levy (2018b), for more details regarding the difference between this procedure and the originally defined CV-TMLE  regarding our parameter of interest and why neither require condition 1 in Section 2.2.1 .
2.3.1 Initial estimates
To perform a one-step CV-TMLE we will first randomly select folds (e.g., ), consisting of disjoint validation sets of equal size, comprising all observations and the corresponding training sets. Each training set is the complement of the corresponding validation set so that for each fold, training set and validation set comprise all subjects. For in we perform the following: using the data-adaptive ensemble machine learning package, SuperLearner , we fit the true outcome model, with on the training set and use the fit to make predictions on the validation set. The V sets of validation set predictions yield one estimate of the outcome prediction for each of the subjects, denoted for the subject. In the case of an observational study, we also might use SuperLearner to estimate the treatment mechanism, , with and predict on the validation set. The sets of validation set predictions yield one prediction of the treatment assignment probability for each of the subjects denoted by . For the one-step TMLE algorithm for ATE and VTE, we would have just fit the outcome model and treatment mechanism on the whole data and computed predictions and on that same data using SuperLearner. Hence, the fundamental difference between CV-TMLE and TMLE is that CV-TMLE only uses predictions on validation sets.
Step 1: Initialization of targeting step
This step begins with computing the negative log-likelihood loss for our outcome predictions:
the starting loss for the outcome where is the true outcome, and we use the empirical distribution as the estimate of the distribution of as in ref. . The initial estimate of does not need to be further modified. The only portion of the empirical mean loss lowered from the initial estimate is that of the outcome model, which is why this is the only portion of the mean loss included here. Continuous outcomes (between 0 and 1) would require minimization of the quasibinomial loss, while binary outcomes would require minimization of the log-likelihood (see Section 2).
The targeting step then requires computing and , where . We note that remains fixed for the entire process for this parameter. Next step is to compute , where is the Euclidean norm and
The initial estimate of the parameter is , i.e., the sample mean and variance of the estimated CATE function.
Step 2 to 4: The targeting steps
If for , then and we can go directly to Step 4. denotes the sample standard deviation as described in Section 2.2.1. This step ensures that we stop the process once the bias is second-order as recursions after this occurs are not fruitful. If , then and we need to proceed to Step 3.
Define the following recursion, using Euclidean inner product notation, , the so-called dot product.
where is set to 0.0001 (see Section 5).
This step then requires computing . If , then Step 2 needs to be reiterated. Otherwise, and we need to proceed to Step 4.
The proposed estimate for ATE and VTE is , where . If the outcome is scaled as (see Section 2, paragraph 1), then the estimate is . Standard errors are given by as in Section 2.2.1, where is the sample standard deviation. If the outcomes are scaled according to , then the standard error estimates for ATE and VTE need to be multiplied by and , respectively.
2.3.2 Simultaneous estimation and confidence intervals
From the steps described above, we have , for each subject indexed by . Consider the two-dimensional random variable , defined by two by two matrix, , the sample correlation matrix of . Let be the quantile of the random variable . Let , where is the correlation matrix of . Let be the quantile of the random variable , i.e., the quantile of the random variable giving the max number of standard deviations over the coordinates of . We used 5 million Monte-Carlo samples from the random variables to find . Five million is a sufficient number to guarantee very little error in finding the true . Applying the continuous mapping theorem  assures under TMLE conditions that covers for all , at and that . is therefore an estimate of the number of standard errors needed to simultaneously cover both true parameter values at . This results in the confidence intervals
which will asymptotically cover all coordinates, , of , simultaneously at the significance level, . The reader may note is the same as the Bonferroni correction  if is the identity matrix.
Two different types of simulations were performed: the first one primarily aimed at verifying the remainder conditions in the theory of TMLE (condition 2, Section 2.2.1); the second one evaluated the performance of our proposed estimators in the more realistic situation where the treatment mechanism and the outcome model are unknown and have to be estimated by modelling the observed data. For all TMLE candidates, inference was based on the sample standard deviation of the efficient influence curve approximation to form confidence intervals as per Section 2.2.1. For logistic regression (LR) substitution estimators of ATE and VTE, confidence intervals were formed by using the delta method and the influence curve for the beta coefficients for intercept, main terms and interactions (see Appendix B for the derivation).
3.1 Simulation study 1: simulations with controlled noise
In this first simulation study, we directly added heteroskedastic noise to in such a way that the conditions of TMLE hold and then use the noisy estimate as the initial estimate in the TMLE procedure. This simulation does not necessarily match what happens in practice because the added noise is not related to the noise in the draw of given and . However, it is a valid way to directly test the conditions of TMLE. In this simulation, we assumed is known because the other second-order terms for VTE, involving bias in estimating , are dependent on double robustness in the same way as for the ATE, for which the properties of TMLE are already well known [1,2].
3.1.1 Simulation set-up
were simulated as follows: , , and . We defined , which is the true density of given We kept our propensity scores between about 0.17 and 0.83 so as to avoid large practical positivity violations . The density of given and for a binary outcome was generated by . Based on our definition of the CATE function, , we had .
Below we illustrate the process for one simulation. For each sample size, , we performed the simulation 1,000 times. We note that rate is some number, which we will set to less than ( in this case) in order to satisfy TMLE conditions.
Define heteroskedasticity: .
Define where is standard normal.
Draw and each from standard normals.
The simulation result, displayed in Figure 1, is in alignment with the theory established for the TMLE estimator of VTE.
3.2 Simulation study 2: realistic simulations
These simulations were more realistic in that the treatment mechanism and the outcome model are unknown and need to be approximated by using machine learning to model the observed data. Unless otherwise noted, sample size was of and the number of simulations was 1,000. were generated as follows: , , and . Two situations were simulated: the first one a “best case” scenario where the models are correctly specified and thus our regressions achieve parametric rates of convergence to the truth; the second one where the models are misspecified and we try to recover its non-linear functional form with ensemble machine learning.
3.2.1 TMLE with well-specified initial estimates
In this first simulation scenario, a LR was used to model the correct functional form for both outcome model, and treatment mechanism, , thus achieving parametric rates of convergence. The goal of these simulations was to show that TMLE preserves well-specified initial estimates and also to show what size sample would result in substantial bias of the sampling distribution for VTE estimates when the truth is near the lower parameter bound of 0. We can say as a rule of thumb, a sample size of 500 or more is needed to get reliable estimates for VTE’s in the neighborhood of 0.025, a rule that was confirmed by Figures 2, 3 and 4. for the outcome regression, varying and to adjust the size of the VTE. was the true treatment mechanism and we avoided positivity violations by keeping our propensity scores mostly between 0.10 and 0.90.
3.2.2 Misspecified TMLE initial estimates
In this second simulation scheme, the models for the treatment mechanism and/or the outcome are misspecified, and we try to estimate the corresponding portion of the likelihood using ensemble machine learning.
Targeted learning  features the use of data adaptive prediction methods optimized by the ensemble learning R packages, such as SuperLearner , H2O  or the most recent sl3 . SuperLearner, which picks the best single algorithm in the library, as decided by the cross-validation of a valid loss function, has mean loss that converges to the oracle selector at rate where is the number of candidate algorithms, under very mild assumptions on the library of estimators . Generally, the best or nearly best learner in the library is the optimal convex combination of algorithms that forms the SuperLearner predictor, which is a form of model stacking .
126.96.36.199 SuperLearner library 1
The algorithms included in this library (SL1) were selected to avoid overfitting:
SL.hal is the highly adaptive lasso (HAL) [26,38]. Assuming the true outcome model and treatment mechanism (in the case it is unknown) are right-hand continuous with left-hand limits and have variation norm smaller than a constant M, then our estimates will be asymptotically efficient if we include SL.hal in the library .
SL.gam3, a gam  using degree 3 smoothing splines, screening main terms, top ten correlated variables with the outcome and top six.
SL.glmnet_1, SL.glmnet_2 and SL.glmnet_3  performed a lasso, equal mix between lasso and ridge penalty and ridge regressions.
nnetMain_screen.Main  is a neural network with decay = 0.1 and size = 5 using main terms.
earthMain  is data adaptive penalized regression spline fitting method. They allow for capturing the subtlety of the true functional form. We allowed degree = 2, which is interaction terms with the default penalty = 3 and a minspan = 10 (minimum observations between knots).
SL.glm  LR and we used main terms, top six correlated variables with outcome and top ten as well as a standard glm with main terms and interactions (glm_mainint_screen.Main)
SL.stepAIC  uses Akaike criterion in forward and backward step regression
SL.mean returns the mean outcome for assurance against overfitting
rpartPrune  is recursive partitioning with cp = 0.001 (must decrease the loss by this factor) minsplit = 5 (min observations to make a split), minbucket = 5 (min elements in a terminal node).
188.8.131.52 SuperLearner library 2
This second library (SL2) was identical to Library 1, except the following learners were added:
SL.ranger : A random forest that picked 3 features at a time formed 2,500 trees and had a minimum leaf size set to 10.
SL.xgboost : One xgboost fit on all main terms and interactions with stumps (depth 1 trees), allowing a minimum of 3 observations per node, a learning rate of 0.001 and summing 10,000 trees. We also included an xgboost using depth of 4 trees on main terms only with same shrinkage and minimum observations per node but only 2,500 trees.
3.2.3 Case 1: Well-specified treatment mechanism, misspecified outcome
In this case, the true models for the outcome and the treatment mechanism were defined as follows:
( + + ). ( + ), which we will specify correctly in all cases with a logistic linear fit. The resulting true ATE was 0.078 and the true CATE variance was equal to 0.085.
The results from case 1, as summarized in Figure 5, highlight the following points:
Benefit of ensemble learning. Using a LR with main terms and interactions substitution estimator and the delta method for inference, yielded a bias of (the truth is 0.079), missing almost entirely the VTE and covering at 0%. The TMLE could not help the initial estimates using the same LR, highlighting the risk of using a parametric model as opposed to ensemble learning.
Difference in robustness between estimating ATE and VTE. The severely misspecified LR with main terms and interactions initial estimate for the outcome model and well-specified treatment mechanism yielded a TMLE for ATE (which is doubly robust) that covered at 95.6%, where as for VTE it never covers the truth.
The advantage of CV-TMLE over TMLE. The same SuperLearner used for initial estimates yields bigger bias and variance for TMLE as opposed to a normally distributed CV-TMLE sampling distribution. Some degree of overfitting introduced by random forest about 20% of the time out of the library of 18 learners generated outliers for TMLE, resulting in lack of normality of the sampling distribution and causing higher bias and variance, where as CV-TMLE appeared unaffected by the overfitting.
Regarding the Donsker condition 1 of Section 2.2.1 or entropy condition, a large Donkser class is the set of functions of uniformly bounded variation , meaning the function class is smooth in some sense, limiting the variation in outcome predictions that overfitting might allow. The efficient influence curve approximation, , is defined partially in terms of the outcome model fit and thus, condition 1 of Section 2.2.1 is dependent on the outcome model being constrained to a Donsker class, otherwise the TMLE might not be asymptotically normal. For instance, overfitting, which might manifest in modeling random noise in the data, might then estimate false heterogeneous responses to treatment and cause large outlier estimates for VTE, right-skewing the sampling distribution as we saw in the simulations for case 1. CV-TMLE does not require the Donsker condition and, therefore, allows highly adaptive machine learning we might need to minimize the second-order remainder bias without losing the normality of our sampling distribution (Table 1).
ATE and VTE estimated simultaneously with one-step TMLE covering both parameters for 95% simultaneous confidence intervals. LR indicates logistic regression with main terms and interactions. SL1 library did not overfit SL2 library overfit 20% of the simulations with 1 out of 18 algorithms, still causing outliers, necessitating CV-TMLE as a precaution.
3.2.4 Case 2 and 3
For both cases 2 and 3, we used the following model for the true treatment mechanism:
Case 2 true outcome model:
Case 3 true outcome model:
Cases 2 and 3 again demonstrate how employing targeted learning with CV-TMLE can recover a misspecified treatment mechanism as well as outcome models when parametric models lead to substantially biased results. Indeed, for both cases, standard parametric models were shown to have almost zero coverage and to be missing almost all of the VTE. However, TMLE coverage was below nominal and at times very poor, depending on the situation. This was not a problem solely for the case of an observational study as the authors have found the main culprit in poor coverage to be the second-order remainder term consisting of the integral of true CATE function minus estimated CATE function squared (see Section 2.2.2). Only in case 2 did CV-TMLE, using a pretty small superlearner library of eight algorithms, including, xgboost, neural networks, glm with main terms and interactions, earth, sample mean and the HAL , achieve a coverage of 83% and reduced bias of the initial estimate from to . In case 3, CV-TMLE with the same SuperLearner library only covered 32%.
3.3 Supplementary results
The reader may visit Jonathan Levy’s github for instructions and software on how to reproduce the results obtained in this article, as well as more detailed results that were not included in this manuscript.
4 Demonstration on real data
The CV-TMLE estimator, as detailed in this article, was applied to a real dataset. GER-INF-05 is a placebo-controlled randomized trial published in 2002 that evaluated the impact on mortality of low-dose steroid administration in patients hospitalized in the intensive care unit (ICU) for septic shock . This study was performed in 19 ICUs in France and enrolled a total of 299 patients. Because steroid supplementation in this context was expected to be beneficial in patients with relative adrenal insufficiency, a corticotropin stimulation test was performed in all patients. A pre-specified subgroup analysis was planned in patients who were non-responders to the corticotropin stimulation test (relative adrenal insufficiency) and in those responders to the corticotropin stimulation test (no relative adrenal insufficiency). In this sample, 7 days of low-dose hydrocortisone associated with fludrocortisone were associated with a reduced risk of death in patients with septic shock. As expected, this reduction was significant in patients with relative adrenal insufficiency as reflected by a lack of response following the corticotropin stimulation test. Because of the apparent heterogeneity in treatment effect, at least partially explained by the presence of a relative adrenal insufficiency, we used the proposed estimator to quantify treatment effect variability across the strata of patients.
We controlled the following confounders in fitting the outcome model: Simplified Acute Physiology Score (SAPS2) severity score to assess mortality, Sequential Organ Failure Assessment (SOFA0) severity score at baseline, lactate level, cortisol level before corticotropin (baseline cortisol), maximal cortisol level after corticotropin stimulation (delta cortisol), an indicator of responding to the corticotropin stimulation test (responder), site of infection, mechanical ventilation at baseline, patient origin (hospital acquired infection or not) (origin), indicator for the reason of ICU admission, i.e., medical, elective surgery or urgent surgery (admission type), indicator of use of etomidate for anesthesia (drug known to alter the adrenal function), blood sugar (glycemia) and the responsible pathogen (germ). In this case, the treatment assignment was random and thus we can identify VTE from the data as a measure of how much of the heterogeneity in treatment effect is due to confounders normally used to assign treatment. We note, in the case of variables missing data, we create an indicator of missingness and use the median or, in the case of categorical variables, the mode as an imputed value. The Table 2 summarizes our data.
|Outcome: renal failure|
|Treatment: Rec. steroid|
|Age in years||60.8 (16.1)|
|Baseline cortisol||23.3 (30.9)|
|Delta cortisol||6.1 (22.3)|
We used a SuperLearner library consisting of 40 algorithms, including main terms and interactions when applying any regression methods, such as LR (glm), Bayes generalized linear models, lasso and ridge regressions (combinations of and penalty) and earth, which uses data adaptive regression splines. For boosting trees (xgboost), we used depth 2 trees to allow interaction as well as depth 1 trees with main terms and interactions as the covariates. Also for boosting, we used different hyperparameters for the number of trees in combination with different learning rates. We used recursive partitioning and random forest, as well as neural networks. We also applied -nearest neighbors for prediction. In addition, we applied screening of the top 25 correlated variables with the outcome in conjunction with then running the machine learning algorithm and did the same for the top 5 variables when running Bayes generalized linear models as well as glm. The convex combination of learners (a.k.a. SuperLearner) had the lowest cross-validated risk (negative log-likelihood) of 0.536 with random forest algorithms and the lasso with all interactions included as variables performing equally well. The discrete SuperLearner , or the one that chooses the lowest risk algorithm to use for each fold’s validation set predictions, had a cross-validated risk of 0.57.
We provide estimates in Table 3 for ATE and VTE simultaneously and used the delta method to also give a confidence interval for the , because such is on the scale of measurement of ATE. The lower bounds of the confidence interval for VTE and are negative and therefore are not possible values for these parameters. Log-scaling the confidence intervals only make them excessively large and therefore not useful due to the unscaled confidence intervals centering close to 0. As mentioned earlier in the Section 3.2.1, if the true VTE is as small as the estimate, then the sampling distribution under the best case scenario of well-specified outcome model is skewed. A true VTE of approximately 0.06 would be needed to have normal sampling distributions for estimating VTE in this scenario. As previously discussed, the second-order remainder term can also generate some bias, which could account for missing a truly larger VTE in our estimates. The second-order remainder term in Section 2.2.2 is the square of the norm bias in estimating the CATE function and so, prioritizing the CATE function in the outcome model estimation is the subject for future work to improve CATE function estimation in finite samples.
The results of the simulation and the real case study highlight the existence of two great challenges in estimating VTE, one being the fact that the parameter is bounded below at 0, skewing and biasing the estimates when the true variance is too small for the sample size. In the future, we might develop an improvement over log-scaling to form adjusted confidence bounds if the lower bound is negative. This problem does not seem as crucial as obtaining reliable inference when the true variance is large enough to be estimated for the sample size. On this front, the second-order remainder term condition has proven difficult to satisfy in finite samples so as to avoid bias and sub-optimal coverage. For larger samples, a SuperLearner library, including the HAL  will likely achieve the necessary rates of convergence for this term to be truly second-order, however, such is not trivial to show conclusively via very time-consuming and computer-intensive simulations, and besides, a sample size of 1,000 for four covariates and binary treatment is certainly of practical importance. To this end, the authors plan to propose in a follow-up paper, a novel way to estimate the CATE function that will also yield estimates of the outcome predictions that are necessarily between 0 and 1, all the while performing asymptotically as good as just fitting the outcome model. We need CATE function estimates that are compatible with the outcome predictions in order to perform the crucial targeting step of the TMLE procedure. That is we need the estimate of the CATE function, , where is the estimate of the outcome model.
Another approach to yielding better coverage would be to account for the second-order remainder term via the use of the new HAL non-parametric bootstrap , to form our confidence intervals. HAL, as mentioned, guarantees the necessary rates of convergence of the second-order remainder terms under very weak conditions and is thus guaranteed to yield asymptotically efficient TMLE estimates , using the empirical variance of the efficient influence curve approximation for the standard error. However, in finite samples such a procedure yields lower than nominal coverage due to the second-order remainder term of Section 2.2.2 that arises when estimating VTE. The HAL CV-TMLE, using a non-parametric bootstrap addresses this issue and can possibly account for the second-order bias, the subject of more future work.
There are two other procedures for mapping to . Levy introduces a new iterative analog to the one-step TMLE procedure, which also uses one-dimensional parametric submodels, called canonical least favorable submodels . The iterative TMLE in ref.  uses parametric submodels of the same dimension as the parameter of interest. It has been conjectured that the one-step TMLE may better preserve the properties of the initial fit, , than the aforementioned iterative versions, thereby leading to better finite-sample behavior of the second-order remainder term . If this conjecture is correct, then we would expect similar gains by using the one-step TMLE in our setting. However, in our simulations we found no appreciable differences.
In Step 3 Section 2.3, we used because this step size was small enough that we lost no appreciable accuracy in constructing the integrals that define the elements of the universal least favorable model in van der Laan and Gruber.
We would like to extend a special acknowledgment to Professor Djillali Annane who granted us real data for use in this paper.
Conflict of interest: Prof. Mark van der Laan is one of the Editors in the Journal of Causal Inference but was not involved in the review process of this article.
Appendix The influence curve and remainder term derivations
Our observed data, , is of the form, , where is a set of confounders, is a binary treatment indicator and is the outcome, continuous or binary. , non-parametric. It is important that we can factorize the density for as .
A.0.1 Tangent space for nonparametric model
We consider the one-dimensional set of submodels that pass through at  . The tangent space is the closure in norm of the set of scores, , or directions for the paths defined above. We write:
For a non-parametric model, forms a Hilbert space with inner product defined as . Our notion of orthogonality now is if and only if and, therefore, the above direct sum is valid. In other words, every score, , can be written as , where due to the fact , it follows that , and . Furthermore, we know that a projection of on is
and thus we have the following identities:
A.0.2 Efficiency theory in brief
Our parameter of interest is a mapping from the model, to the real numbers given by where . We can borrow from van der Vaart, who defines an influence function, otherwise known as a gradient, as a continuous linear map from to the reals given by
We note to the reader, we imply a direction, , when we write , which has density , but generally leave it off the notation as understood.
By the Riesz representation theorem  for Hilbert Spaces, if the functional defined in equation (7) is a bounded and linear functional on it can be written in the form of an inner product where is a unique element of , which we call the canonical gradient or efficient influence curve. Thus, in the case of a nonparametric model, the only gradient is the canonical gradient. It is notable that the efficient influence curve has a variance that is the lower bound for any regular asymptotically linear estimator . Since the TMLE, under conditions as discussed in this article, asymptotically achieves variance equal to that of the efficient influence curve, the estimator is asymptotically efficient.
As a note to the reader: Our parameter mapping does not depend on the treatment mechanism, , and also , which means our efficient influence curve must therefore be in for the nonparametric model. Therefore, our efficient influence curve will have two orthogonal components in and , respectively. We have no component in , which is why we need not perform a TMLE update of the initial prediction, , of . This also teaches us that for the semi-parametric model, where the treatment mechanism is known, the efficient influence function will be the same as for the non-parametric model.
A.1 Derivation of the efficient influence curve
For the below proof, when we take the derivatives we will assume that such is in the direction of a given score, . We will assume a dominating measure and merely use to always denote the dominating measure of all densities involved. We could perhaps just assume continuous densities and use Lebesque measure as well and nothing would change below.
Let . The efficient influence curve for at is given by:
Now we can compute the first term in (8).
is the aforementioned representer, completing the proof.□
A.2 Remainder terms and asymptotic linearity
The reader may recall the three conditions assuring asymptotic efficiency of the TMLE estimator. Here, we will focus on condition 2 in Section 2.2.1, which requires the remainder term,
Let , treatment indicator and outcome, resp., , the true distribution, , be the true treatment mechanism and and be estimated counterparts. Assume s.t. for all . A sufficient condition to guarantee (9) is . It is necessary to estimate the true CATE function with so that in order to satisfy (9). Assume and then, in addition to the necessary condition, it is sufficient that also