An Approach to Evaluating and Comparing Biomarkers for Patient Treatment Selection

1 Introduction

There is an enormous amount of research effort being devoted to discovering and evaluating markers that can predict a patient’s chance of responding to treatment. A December, 2013 PubMed search identified 8,198 papers evaluating such markers over the last 2 years alone. Treatment selection markers, sometimes called “predictive” [1] or “prescriptive” [2] markers, have the potential to improve patient outcomes and reduce medical costs by allowing treatment provision to be restricted to those subjects most likely to benefit, and avoiding treatment in those only likely to suffer its side effects and other costs.

Methods for evaluating treatment selection markers are much less well developed than for markers used to diagnose disease or predict risk under a single treatment. In the medical literature, the most common approach to marker evaluation is to test for a statistical interaction between the marker and treatment in the context of a randomized and controlled trial (RCT; see Coates et al. [3], Busch et al. [4], and Malmstrom et al. and NCBTSG [5] for some recent examples). However this approach has limitations in that it does not provide a clinically relevant measure of the benefit of using the marker to select treatment and does not facilitate comparing candidate markers [6]. Moreover, the scale and magnitude of the interaction coefficient will depend on the form of the regression model used to test for interaction and on the other covariates included in this model [7].

There is a growing literature on statistical methods for evaluating treatment selection markers. A number of papers have focused on descriptive analysis, specifically on modeling the treatment effect as a function of marker [8–12]. In general, these approaches are not well-suited to the task of comparing candidate markers. Other papers have proposed individual measures for evaluating markers [6, 7, 13–16], some of which we adopt as part of our analytic approach as described below. Still others have focused on the specific problem of optimizing marker combinations for treatment selection [17–22]. A complete framework for marker evaluation, on a par with those developed for evaluating markers for classification [23, 24] or risk prediction [25], is still forthcoming.

In this paper, we lay out a comprehensive approach to evaluate markers for treatment selection. We propose tools for descriptive analysis and summary measures for formal evaluation and comparison of markers. The descriptives are conceptually similar to those of Bonetti and Gelber [8], Royston and Sauerbrei [9], Cai et al. [10], but we scale markers to the percentile scale to facilitate making comparisons. Our preferred global summary measure is the same as or closely related to that advocated by Song and Pepe [13], Brinkley et al. [16], Janes et al. [6], Gunter et al. [19], Qian and Murphy [20], McKeague and Qian [21], and Zhang et al. [22], a component of which was described by Zhao et al. [12] and Baker and Kramer [14]. We also propose several novel measures of treatment selection performance, motivated by existing methodology for evaluating markers for predicting outcome under a single treatment, i.e. for risk prediction. We develop methods for estimation and inference that apply to data from a randomized controlled trial comparing two treatment options where the marker is measured at baseline on all or a stratified case–control sample of trial participants. For illustration, we consider the breast cancer treatment context where candidate markers are evaluated for their utility in identifying a subset of women who do not benefit from adjuvant chemotherapy. Appendices include the results of a small-scale simulation study that evaluates the performance of the methods in finite samples and a description of the R package we have written that implements these methods.

2 Setting and notation

Suppose that the task is to decide between two treatment options, referred to as “treatment” ($T=1$) and “no treatment” ($T=0$). The clinical outcome of interest, D, is a binary indicator of a bad event within a specific time-frame following treatment provision; we refer to this outcome as an “adverse event” or “event”. The outcome D is thought to capture all potential impacts of treatment, so that any decrease in the rate of events justifies treatment; a generalization is discussed in Section 6.1. To achieve this, D may be chosen to represent a composite outcome such as an indicator of treatment-associated toxicity or death. We assume that the marginal treatment effect ${\mathrm{\rho }}_0-{\mathrm{\rho }}_1\equiv P(D=1\mid T=0)-P(D=1\mid T=1)$ is positive, so that the default approach is to treat all subjects. The question is whether a marker, Y, if measured prior to treatment provision, is useful for identifying a subset of subjects who can avoid treatment. Note that the scenario where the marginal treatment effect is negative (or zero) and Y identifies a subset of subjects who benefit from treatment can be handled by simply reversing the treatment labels.

We focus on the ideal setting for evaluating treatment efficacy, an RCT comparing $T=1$ with $T=0$. By necessity, this must be a relatively large trial; it is well-known that large sample sizes are generally needed to detect statistical interactions. We assume to begin that Y is continuous and measured at baseline on all trial participants. We generalize our methods to case–control sampling from within an RCT in Section 6.2.

3 Motivating context

We illustrate our methods in the breast cancer treatment context. Women diagnosed with estrogen-receptor-positive and node-positive breast cancer are typically treated with both hormone therapy (e.g. tamoxifen) and adjuvant chemotherapy following surgery. This is despite the fact that it is generally well-accepted in the clinical community that only a subset of these women actually benefit from the adjuvant chemotherapy, and the remaining women suffer its toxic side effects, not to mention the burden and cost of unnecessary treatment [26]. A high public health priority is to identify biomarkers that can be used to predict which women are and are not likely to benefit from the adjuvant chemotherapy [27]. The Oncotype DX recurrence score is an example of a biomarker that is currently being used in clinical practice for this purpose. This marker is a proprietary combination of 21 genes whose expression levels are measured in the tumor tissue obtained at surgery [28–30]. The marker has been shown to have value for identifying a subset of women who are unlikely to benefit from chemotherapy [28, 29, and 30].

To illustrate our methods, we simulated a marker, $Y_1$, with the same performance as Oncotype DX in the SWOG SS8814 trial which evaluated adjuvant chemotherapy (cyclophosphamide, doxorubicin, and fluorouracil) given before tamoxifen for treating post-menopausal women with estrogen-receptor-positive, node-positive breast cancer [28, 31]. We also simulated another marker, $Y_2$, which we will demonstrate is a much stronger marker. Both markers $Y_1$ and $Y_2$ are measured at baseline for 1,000 participants randomized with equal probability to tamoxifen alone ($T=0$) or tamoxifen plus chemotherapy ($T=1$). The outcome, D, is breast cancer recurrence or death within 5 years of randomization, and the marginal treatment effect is ${\mathrm{\rho }}_0-{\mathrm{\rho }}_1=0.24-0.21=0.03$ as seen in SS8814. Marker $Y_1$ is simulated to mimic the Oncotype DX distribution; ${\sqrt{Y}}_1$ is normally distributed with mean 4.8 and standard deviation 1.8. Marker $Y_2$ is standard normal. Each marker is related to D via a linear logistic model, $\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}P(D=1\mid T,Y)={\mathrm{\beta }}_0+{\mathrm{\beta }}_{1}T+{\mathrm{\beta }}_{2}Y+{\mathrm{\beta }}_{3}YT$, where for $Y_1$ the model coefficients are chosen to mimic the performance of the Oncotype DX recurrence score [28]. Methods for simulating the data are described in the Appendix.

4 Methods for evaluating individual markers

4.2 Descriptives

For descriptive analysis, it is useful to display the distribution of risk of the event as a function of the marker under each treatment. We plot “risk curves” $P(D=1\mid T=1,Y)$ and $P(D=1\mid T=0,Y)$ vs marker percentile $F(Y)$, where F is the cumulative distribution function (CDF) of Y [6]. Figure 1 shows the risk curves for the Oncotype-DX-like marker, $Y_1$, and the much better marker, $Y_2$. From these one can visually assess the variability in response on each treatment as a function of marker value. One can also determine the proportion of subjects with negative treatment effects who can avoid chemotherapy, 46% for $Y_1$ vs 38% for $Y_2$.

• Download Figure
• Download figure as PowerPoint slide

Risk of 5-year breast cancer recurrence or death as a function of treatment assignment and marker percentile, for $Y_1$, the Oncotype-DX-like marker (top), and the strong marker, $Y_2$ (bottom). Horizontal pointwise 95% confidence intervals (CIs) are shown. Forty-six percent of women have negative treatment effects according to $Y_1$ vs 38% with $Y_2$; these women can avoid adjuvant chemotherapy

Citation: The International Journal of Biostatistics 10, 1; 10.1515/ijb-2012-0052

Another informative display is the distribution of treatment effect, as summarized by $\mathrm{\Delta }(Y)$ vs $F_{\mathrm{\Delta }}(\mathrm{\Delta }(Y))$ where $F_{\mathrm{\Delta }}$ is the CDF of $\mathrm{\Delta }(Y)$ [7]. The example shown in Figure 2 reveals that $Y_2$ has much greater variation in marker-specific treatment effect than does $Y_1$. For $Y_2$ a greater proportion of marker-specific treatment effects are extreme, whereas for $Y_1$ the range is smaller and most treatment effects are near the average of ${\mathrm{\rho }}_0-{\mathrm{\rho }}_1=0.03$.

• Download Figure
• Download figure as PowerPoint slide

Distribution of the treatment effect, as measured by the difference in the 5-year breast cancer recurrence or death rate without vs with treatment, $\mathrm{\Delta }(Y)=P(D=1\mid T=0,Y)-P(D=1\mid T=1,Y)$, for the Oncotype-DX-like marker ($Y_1$) and the strong marker ($Y_2$). Horizontal pointwise 95% CIs are shown

Citation: The International Journal of Biostatistics 10, 1; 10.1515/ijb-2012-0052

4.5 Calibration assessment

Assessing model calibration is a fundamental step in marker evaluation. We rely on standard methods for visualizing and testing goodness of fit for the risk model (1) and extend these methods to assess calibration of the treatment effect model.

Since patients are provided risk estimates under both treatment options, first we assess the fit of the risk model separately in the two treatment groups. Specifically, we define a well-calibrated model to be one for which $P(D=1\mid T=0,Ris{k}_0(Y)=r)\approx r$ and $P(D=1\mid T=1,Ris{k}_1(Y)=r)\approx r$. To assess this, we split each treatment group $t=0,1$ into G equally-sized groups where the observations in each group have similar ${\widehat{Risk}}_t(Y)$. Commonly $G=10$ and the groups are based on quantiles of ${\widehat{Risk}}_t(Y)$. In each group, we calculate the average predicted risks, ${\overline{Risk}}_{tg}(Y)$, and the observed risks, $\hat{P}(D=1\mid T=t,G=g)$. Following Huang and Pepe [33], we plot the distribution of ${\widehat{Risk}}_0(Y)$ and ${\widehat{Risk}}_1(Y)$, overlaying the G observed risk values on the plot, as shown in Figure 3.

• Download Figure
• Download figure as PowerPoint slide

Plots assessing calibration of the risk and treatment effect models, for the Oncotype-DX-like marker (left) and the strong marker (right)

Citation: The International Journal of Biostatistics 10, 1; 10.1515/ijb-2012-0052

To formally assess model calibration, a traditional Hosmer–Lemeshow goodness of fit test [44] can be applied separately to the two treatment groups. Specifically, for group $T=t$ the test statistic

$H{L}_t=\sum _{g=1}^G\frac{N_{tg}\left(\hat{P}(D=1\mid T=t,G=g\right)-{\overline{Risk}}_{tg}(Y{))}^2}{{\overline{Risk}}_{tg}(Y)\left(1-{\overline{Risk}}_{tg}(Y)\right)},$

where $N_{tg}$ is the number of participants in the $g$th group for $T=t$, is compared to a ${\mathrm{\chi }}^2$ distribution with $G-2$ degrees of freedom.

Another aspect of calibration is the extent to which the treatment effect model fits well. We want to ensure that $P(D=1\mid T=0,\mathrm{\Delta }(Y)=\mathrm{\delta })-P(D=1\mid T=1,\mathrm{\Delta }(Y)=\mathrm{\delta })\approx \mathrm{\delta }$. Following the approach above, we split the data into G evenly-sized groups based on $\hat{\mathrm{\Delta }}(Y)$ and calculate the average predicted treatment effect, ${\overline{\mathrm{\Delta }}}_g(Y)$, and observed treatment effect, $\hat{P}(D=1\mid T=0,G=g)-\hat{P}(D=1\mid T=1,G=g)$, in each group. We plot the treatment effect curve and overlay the G observed treatment effect values as shown in Figure 3. Based on Figure 3, we see that the risk and treatment effect models for $Y_1$ and $Y_2$ in the breast cancer example are well-calibrated; the Hosmer–Lemeshow test statistics are 4.5 ($p=0.81$) and 8.9 ($p=0.35$) given $T=0$ and 5.0 ($p=0.76$) and 2.9 ($p=0.94$) given $T=1$. The $Ris{k}_0(Y_2)$ and $\mathrm{\Delta }(Y_1)$ curves suggest some evidence of poor calibration, which in our simulated data setting is attributable to sampling variability in the observed risks that are calculated using 50 observations each.

5 Comparing markers

The descriptives and summary measures proposed herein form the basis for comparing candidate markers. We assume that the two markers, $Y_1$ and $Y_2$, are measured on the same subjects, i.e. that the data are paired. With unpaired data, the analyses described above can be applied to each individual dataset and inferences can be drawn easily given that the estimated summary measures are statistically independent.

For drawing inference about the relative performance of two markers given paired data, CIs for the differences in performance measures and hypothesis tests of whether these differ from zero are informative. We fit separate models for $P(D=1\mid T,Y_1)$ and $P(D=1\mid T,Y_2)$, use these to estimate performance measures for $Y_1$ and $Y_2$, respectively, and bootstrap the differences in estimated performance measures. While global measures of marker performance such as $\mathrm{\Theta }$, $V_{\mathrm{\Delta }}$, and TG are appropriate as the basis for formal marker comparisons, differences in the other summary measures inform about the nature of the difference between markers. We advocate $\mathrm{\Theta }$ as the primary measure on which to base marker comparisons, given its clear clinical relevance and interpretation.

The results of the comparative analysis for the breast cancer example are shown in Table 1. We can see clearly that $Y_2$ has uniformly better performance than $Y_1$, with an estimated 10% vs 1% reduction in the 5-year recurrence or death rate. Despite the fact that there are estimated to be fewer marker-negative subjects based on $Y_2$ (38% vs 46%), there is a much greater estimated benefit of no chemotherapy among $Y_2$-marker-negatives (26% vs 2% reduction in the 5-year recurrence or death rate). In general the variation in treatment effect is larger for $Y_2$.

6 Extensions

7 Discussion

This paper proposes a statistical framework for evaluating a candidate treatment selection marker and for comparing two markers. Estimation and inference techniques are described for the setting where the marker or markers are measured on all or a treatment-stratified case–control sample of participants in a randomized, controlled trial. An R software package was developed which implements these methods. Developing a solid framework for evaluating and comparing markers is fundamental for accomplishing more sophisticated tasks such as combining markers, accounting for covariates, and assessing the improvement in performance associated with adding a new marker to a set of established markers.

Our approach to marker evaluation also applies when the marker is discrete. In addition, it can be applied when there are multiple markers and interest lies in evaluating their combination; $\mathrm{\Delta }(Y)=P(D=1\mid T=0,Y)-P(D=1\mid T=1,Y)$ is the combination of interest and the measures described here can be used to summarize the performance of this combination.

This work extends existing approaches for evaluating markers for risk prediction [25, 35, 46]. It also unifies existing methodology for evaluating treatment selection markers. In particular, our preferred marker performance measure has been advocated by Song and Pepe [13], Brinkley et al. [16], Janes et al. [6], Gunter et al. [19], Qian and Murphy [20], McKeague and Qian [21], and Zhang et al. [22].

There are challenges with making inference about the performance measures we propose, similar to problems that have been identified with measures of risk prediction model performance including the area under the ROC curve [36, 38–40], the integrated discrimination index [37], and the net reclassification index [41]. The problems may arise when the sample size is modest and marker performance is weak. In particular for the Oncotype DX example, given that the marker is weak and the primary study evaluating its performance by Albain et al. [28] included just 367 women, our simulation results suggest that the resultant estimate of $\mathrm{\Theta }$ is likely an over-estimate and that the CI may be conservative. For this reason, we propose testing for non-null marker performance prior to estimating the magnitude of performance. This approach performed reasonably well in our simulation studies, but improved approaches to inference, for the treatment selection as well as risk prediction problem, merit investigation.

The methods described here can and should be extended to accommodate other types of outcomes. Extension to continuous or count outcomes is straightforward. Specifically, after replacing $P(D=1)$ with $E(D)$ and using a risk model appropriate to the scale of the outcome, e.g. a linear or log-linear model for a continuous outcome, the analysis proceeds as above. The conceptual framework also applies to time-to-event outcomes, with the task being to predict risk of the outcome by a specified landmark time.

The methods may also be generalized to an observational study setting, or to a setting where data on the two treatments come from two different studies – perhaps historical data are paired with a single-arm trial of $T=1$. However, the usual concerns about measured and unmeasured confounding in estimating the treatment effect apply. In this setting, an analyst would be well-advised to stratify on variables that are potentially associated with treatment provision and outcome. More generally, methods for adjusting for covariates in the evaluation of marker performance warrant further research.