Confidence limits for the averted infections ratio estimated via the counterfactual placebo incidence rate

Introduction

In a series of papers we have considered the analysis of active-control non-inferiority trials with a time-to-event outcome in the context of HIV prevention trials (Dunn and Glidden 2019; Dunn et al. 2018; Glidden, Stirrup, and Dunn 2020). Our key conclusion is that the standard metric used in such trials, the rate ratio comparing experimental and control arms, is misleading. We further argued that clinically meaningful inference requires estimation or specification of one of two unobserved parameters: (a) the event rate that would have been observed in trial subjects if they had received no treatment (counterfactual placebo arm) or (b) the effectiveness of the control arm relative to the counterfactual placebo arm. With this information, in combination with the observed incidence rates in the control and experimental arms, we can estimate a measure called the averted infections ratio (AIR). The AIR is interpreted as the proportion of events that would be averted by use of the experimental treatment compared with the control treatment. In the context of non-inferiority trials, it is a natural criterion for assessing the degree to which the experimental treatment preserves the effect of the control treatment relative to no treatment (“preservation-of-effect”) (Ghosh et al. 2011; Pigeot et al. 2003; Snapinn and Jiang 2008). Non-inferiority trials using this approach typically aim to demonstrate at least 50% preservation-of-effect, although this value is context specific and higher values may be warranted (Pigeot et al. 2003; Ghosh et al. 2011). In this paper, we consider the derivation of confidence limits for the AIR when it is estimated via the counterfactual placebo incidence.

Notation and statistical formulation

Denote the hypothetical placebo, control, and experimental arms by the subscripts P, C, and E, respectively. We observe F _C person-years follow-up in control arm and F _E person-years follow-up in experimental arm. Let X _C and X _E be the random variables denoting the number of observed events, where we assume that X C ∼ Poi F C λ C and X E ∼ Poi F E λ E . Let λ _P represent the counterfactual placebo incidence. The averted infections ratio is defined as

Alternatively, Ψ can be expressed in terms of the counterfactual control arm effectiveness ( θ C = 1 − λ C / λ P rather than λ _P:

In this formulation, Ψ is a linear function of the rate ratio and confidence limits for Ψ can be obtained by direct transformation of confidence limits for the rate ratio. As the latter problem has been extensively studied (Graham, Mengersen, and Morton 2003; Li, Tang, and Wong 2014; Price and Bonett 2000; Sahai and Khurshid 1993) we focus on formulation (1).

Inference conditional on counterfactual incidence

This section considers the derivation of confidence limits for the AIR when considering a single, pre-specified value of λ _P. This allows exploration of how the confidence limits (and point estimates) vary over a range of plausible values of λ _P, which can be highly informative (Glidden, Stirrup, and Dunn 2020).

Unconditional inference

In addition to exploring how the AIR varies over a range of values of the counterfactual incidence, we may wish to integrate over this parameter to obtain the unconditional distribution of the AIR. Bayesian inference provides a natural framework for this problem. Here we consider the case where trial investigators are able to specify a simple prior distribution for the counterfactual incidence, although more sophisticated approaches which incorporate external information are also possible (Glidden, Stirrup, and Dunn 2020).

Assume that the prior for λ _P can be specified as a Gamma distribution based on background knowledge. For λ _E and λ _C, we use weakly informative priors ∼Gamma(0.5,0.001) – this approximates to Jeffrey’s prior (Gelman et al. 1995), and also corresponds to adding 0.5 to the observed number of events as discussed in Section 6. As the Gamma distribution is the conjugate prior for the Poisson model, the posterior distributions for λ _E and λ _C are Gamma(X _E + 0.5, F _E + 0.001) and Gamma(X _C + 0.5, F _C + 0.001), respectively (Gelman et al. 1995). We generate samples from the distributions of λ _P, λ _E, and λ _C, to derive the posterior distribution for the AIR using Eq. (1).

The main application of the AIR is in non-inferiority trials, where it is reasonable to assume that λ _C < λ _P since the effectiveness of the control drug will already have been established. Further, the AIR is uninterpretable if λ _C > λ _P as this would imply there was no yardstick against which the experimental drug could be compared (nothing to preserve). In most realistic applications it is also reasonable to assume that λ _E < λ _P as the experimental drug will have been selected as having some biological activity. It is therefore problematic if the sampled values λ P * , λ C * , λ E * satisfy either

There are three possible re-sampling strategies: (a) re-sample λ P * only (b) re-sample λ P * , λ C * if λ C * > λ P * ; re-sample λ P * , λ E * if λ E * > λ P * (c) re-sample all three values λ P * , λ C * , λ E * . The best strategy is not obvious, and all are explored in the example in Section 7.

Three arm trials with a placebo arm

Trials are occasionally designed with a placebo arm in addition to the control and experimental arms, thereby providing a direct estimate of λ _P (Ghosh et al. 2011). The Taylor series approximation (Eq. (2)) requires an additional term to reflect the uncertainty in the estimate of λ _P:

The additional term is generally much smaller than the first two terms and, in expectation, (7) tends towards (2) when λ _E = λ _C. This leads to a paradoxical finding, namely that the sample size of the placebo group appears to be irrelevant when this equality is assumed (as is commonly the case when designing non-inferiority trials). This paradox is explained by the fact that Ψ = 1 when λ _E = λ _C regardless of the value of λ _P. However, the placebo group needs to be large enough in order to ensure that the estimate λ ̂ P is sufficiently stable. An interesting, unresolved question is the optimal relative sample size allocation to the three arms. We further note the profile-likelihood approach (Section 3.2) could, in principle, be extended to three arm trials.

Coverage probabilities

Results

The complete set of coverage probabilities for the lower and upper confidence limits are given in the Appendix. However, the lower confidence limit is of primary interest since this is the comparator for the non-inferiority margin. Also, the upper limit of Ψ may be severely constrained for large values of θ _C. Ψ can be expressed as θ _E/θ _C, so that, for example, Ψ ≤ 1.25 if θ _C = 0.8, Ψ ≤ 1.11 if θ _C = 0.9.

Figure 1 shows coverage probabilities using the delta method for the lower one-tailed α = 0.05 confidence limit (similar patterns were observed for α = 0.025). Coverage is generally too low for Ψ = 0.5–0.8, is reasonably accurate for Ψ = 0.9, and is too high for Ψ = 1.0. This pattern is explained by a negative correlation between the empirical AIR and its estimated standard error, conditional on the true AIR (Ψ). Conditional on Ψ, coverage is higher the larger the value of the control arm effectiveness (θ _C), except for Ψ = 1.0 when differences are minor. As expected, actual and nominal coverage are closer the larger the value of λ _P, although convergence is slow with material discrepancies even for λ _P = 100. Coverage probabilities for the upper confidence limit were consistently and substantially too high (Appendix), particularly for lower values of Ψ.

Figure 1:

Coverage of lower 5% confidence limit computed using the delta method.

(A) 40 expected counterfactual events per arm (B) 60 events (C) 80 events (D) 100 events. Efficacy of control treatment: 0.6, red line; 0.7, orange; 0.8, green; 0.9, blue. Horizontal line represents nominal coverage of 0.95.

Table 1 shows coverage probabilities for the profile-likelihood-based lower confidence limit for λ _P = 40 and α = 0.05. Coverage was close to the nominal value of 0.95 (range 0.9468–0.9615) for all permutations of Ψ and θ _C; as expected, correspondence was even closer at higher values of λ _P (not shown). Coverage for the profile-likelihood-based upper confidence limit were also highly accurate, in contrast to the delta method (Appendix). The results of these analyses support the routine use of profile-likelihood-based confidence limits, although the delta method is valid in a conservative sense (i.e. actual coverage exceeds nominal coverage) if the true AIR is ≥0.9 approximately. This is reflected in larger values for the lower confidence limit using the delta method (Appendix).

Example

The BRIEF TB/A5279 study was a randomised, non-inferiority trial that compared two regimens for the prevention of active tuberculosis in HIV-infected patients who were living in areas of high tuberculosis prevalence or who had evidence of latent tuberculosis infection (Swindells et al. 2019). The reference regimen was 9 months of daily isoniazid alone (9-month arm) and the experimental regimen was 1 month of daily rifapentine plus isoniazid (1-month arm). The incidence of the primary endpoint (diagnosis of tuberculosis, or death from tuberculosis or unknown cause) were similar in the 1-month arm (32 endpoints, 4,926 person-years follow-up (PYFU), incidence rate 0.65 per 100 PYFU) and 9-month arm (33 endpoints, 4,896 PYFU, incidence rate 0.67 per 100 PYFU). The primary metric was the rate difference rather than the rate ratio, which is generally used in HIV prevention research. Non-inferiority was declared by the investigators because the upper 97.5% confidence limit of 0.30 per 100 PYFU was less than the pre-specified margin of 1.25 per 100 PYFU. However, this conclusion is questionable as the authors did not take the counterfactual placebo incidence into account. Notably, the observed incidence in the 9-month arm was markedly lower than the incidence rate assumed for the purposes of sample size calculation (2 per 100 PYFU).

Figure 2 shows the lower 5% and upper 95% confidence limits for the AIR as a function of the counterfactual incidence, computed using the delta and profile-likelihood methods. Consistent with results of Section 6.2, the delta method yields narrower confidence intervals. The figure also reveals the sensitive relationship between the lower confidence limit and the assumed counterfactual incidence, underlining the importance of obtaining as much information as possible about this parameter.

Figure 2:

Lower 5% and upper 95% confidence limits for the AIR in the BRIEF TB/A5279 study.

Delta method, blue line; profile-likelihood method, red line.

Figure 3 show the results of a Bayesian analysis (10,000 simulations) under two different priors for the counterfactual incidence: Gamma(10,0.001) and Gamma(10,0.002), corresponding to mean incidence rates of 1 and 2 per 100 PYFU, respectively. Without expert knowledge, we emphasise that this is an illustrative rather than a definitive analysis. The lower incidence rate is broadly consistent with the overall ∼30% efficacy of tuberculosis prophylaxis in HIV-infected patients (Ross et al. 2021); the higher value is the rate that the investigators postulated for the control regimen (post hoc, a substantial over-estimate).

Figure 3:

Bayesian analysis of BRIEF TB/A5279 study.

(A) Posterior distributions of incidence rate in control regimen (orange line) and experimental regimen (red line); prior distributions of counterfactual incidence (mean = 0.01, green line; mean = 0.02, blue line). (B) Posterior distribution of AIR by prior distribution used for counterfactual incidence (mean = 0.01, green line; mean = 0.02, blue line).

For the low incidence scenario, 22.2% of initial simulations had to be re-sampled because of violation of Eq. (6). The posterior median (90% credibility interval) AIR was 1.038 (0.347, 3.627) under re-sampling strategy (a), 1.033 (0.373, 3.228) under strategy (b), and 1.031 (0.357, 3.281) under strategy (c). For the high incidence scenario, only 0.6% of initial simulations had to be re-sampled. The posterior median (90% credibility interval) AIR under re-sampling strategy (a) was 1.009 (0.760, 1.370). The values under the other re-sampling strategies were almost identical (all within ±0.002). In general, our preference is to re-sample λ P * only (strategy (a)) since, lacking empirical data, λ _P is the most uncertain parameter. Figure 3B shows the posterior distributions for the AIR under this strategy, and highlights that inference is much tighter under the high incidence scenario.

Summary

We have described two approaches for calculating confidence limits for the AIR given a pre-specified value of the counterfactual incidence: a closed-form solution based on a Taylor series expansion (delta method), and an iterative method based on the profile-likelihood, for which R code is provided. The profile-likelihood method is preferred because of better coverage properties, but the delta method is valid when the experimental treatment is no less effective than the control treatment. The difference between the two methods is minimal when the counterfactual incidence is much larger than the observed incidence in both the control and experimental arms. We also describe a simple Bayesian approach when the counterfactual incidence rate can be represented as a simple prior distribution. However, more precise inference can be achieved by harnessing other data which inform the prior distribution (Glidden, Stirrup, and Dunn 2020).