As already in the case of equal covariance matrices [1], the derivation of the exact joint distribution of the test statistics would be a challenging problem. The endpoints have different scales, their covariances must be estimated, and the covariance matrices are unequal. In this article, approximations are used based on multivariate t-distributions. Therefore, a validation was done by simulations. Three and five treatment groups, respectively, have been compared in a simulation study. The first group was regarded as the control. The study had different numbers of endpoints with related expected values: ${\mathit{\mu }}_h=(10,100{)}^{\prime }$ for 2 endpoints, ${\mathit{\mu }}_h=(0.1,1,10,100{)}^{\prime }$ for 4 endpoints, and ${\mathit{\mu }}_h=(0.05,0.1,0.5,1,5,10,50,100{)}^{\prime }$ for 8 endpoints, respectively, for all treatment groups $h=1,\dots ,p$. The most critical situation in the context of heteroscedasticity is if the treatment group with the highest variance has the smallest sample size because approaches for homoscedastic data yield very liberal test decisions in this situation (e.g. see Hasler and Hothorn [10]). For that reason, the first $p-1$ treatment groups had same covariance matrices, $\sum _1=\cdots =\sum _{p-1}$, with standard deviations $\sqrt{diag(\sum _h)}=0.1{\mathit{\mu }}_h$ and sample size $n_h=20$ for each endpoint of each group ($h=1,\dots ,p-1$); the last group had standard deviations $\sqrt{diag(\sum _p)}=0.25{\mathit{\mu }}_p$ and sample size $n_p=10$ for each endpoint. Three equicorrelation structures (compound symmetry) of the endpoints were chosen (${\mathrm{\rho }}^{\mathrm{m}\mathrm{i}\mathrm{n}}=-1/(k-1)$, 0, 0.8) as well as a random correlation structure (rand.), which was different for each simulation run. For the random correlation structure, the first $p-1$ groups always had the same correlations per run. Four one-sided MCT problems were considered which are all related to hypotheses (1): Dunnett (many-to-one), Tukey (all-pair), Williams (trend), and Average (mean averages). Usually, a Tukey MCT is a two-sided testing problem. For reasons of consistency this fact was disregarded. The FWE has been simulated at a nominal level of $\mathrm{\alpha }=0.05$. The simulation results have been obtained from 10,000 simulation runs each, with starting seed 10,000, using a program code in the statistical software R [15], packages

mvtnorm

SimComp

Tables 1 and 2 show the simulated $\mathrm{\alpha }$-level for Dunnett and Tukey contrasts, respectively. The results for the Williams and Average contrasts are not shown here for reasons of brevity, but they can be obtained from the author on request. It is clear a priori that the FWE of CE must be greater than the FWE of MIN since ${\mathrm{\nu }}_{li}\ge {\mathrm{\nu }}_l$ for all $l=1,\dots ,q$ and $i=1,\dots ,k$. Indeed, CE shows a liberal behaviour (ranges from 0.049 to 0.066). In addition to the procedures described, a further version has been simulated. Procedure BON is according to a complete (univariate) Bonferroni adjustment with degrees of freedom each according to eq. (6). It is known to produce conservative test decisions (ranges from 0.023 to 0.056), especially for an increasing number of comparisons. Note that the different MCTs imply different numbers of contrasts and also different correlations among them. In general, the MIN procedure maintains the $\mathrm{\alpha }$-level in an admissible range. The slight variation around the nominal $\mathrm{\alpha }=0.05$ (ranges from 0.044 to 0.062) is always bounded by the two other procedures; $\mathrm{\alpha }$-exceeding is very rare. However, MIN can also be slightly conservative compared to BON in a few cases, see Table 1 for $p=3$, $k=4$, $\mathrm{\rho }={\mathrm{\rho }}^{\mathrm{m}\mathrm{i}\mathrm{n}}$, for example. This is clearly caused by the fact that BON uses contrast- and endpoint-specific degrees of freedom (6), and ${\mathrm{\nu }}_{li}\ge {\mathrm{\nu }}_l$ for all $l=1,\dots ,q$ and $i=1,\dots ,k$. Furthermore, a multivariate approach is known not to have most gain in power compared to a univariate Bonferroni-adjustment if related correlations are high. This is, the gain in power of MIN compared to BON is very low for negative or small correlations. Procedure HOM represents the method for homogeneous covariance matrices. Expectedly, HOM is strongly liberal (ranges from 0.121 to 0.371) and shows how important it is not to ignore heteroscedasticity of the data. It is most liberal for negative correlations of the endpoints ($\mathrm{\rho }={\mathrm{\rho }}^{\mathrm{m}\mathrm{i}\mathrm{n}}$) and less liberal (but still strongly) for positive ($\mathrm{\rho }=0.8$). Generally, the correlations of the endpoints have no deciding influence for MIN and CE. This fact coincides with the results of Hasler and Hothorn [1] in the case of equal covariance matrices.

According to Xu et al. [17] and Liu et al. [18], applying multivariate t-distributions in the context of multiple endpoints and using the method of Genz and Bretz [11] may lead to slightly liberal test decisions. Also for that reason, the degrees of freedom for the MIN procedure according to eq. (8) are defined in a conservative manner. For each contrast, the minimum of the degrees of freedom (6) is taken over the endpoints.

Although the procedures presented allow unequal covariance matrices for the treatment groups, the multivariate normal distribution is still a strong assumption. Therefore, a further simulation study was done to check how sensitive the proposed methods are to violations of the multivariate normal distribution. Similarly to the above study, three and five treatment groups, respectively, have been compared. The first group was regarded as the control. The study had different numbers of correlated log-normally distributed endpoints based on multivariate normally distributed data with related parameters: ${\mathit{\mu }}_h=(1,3{)}^{\prime }$, $\sqrt{diag(\sum _h)}=(0.25,0.6{)}^{\prime }$ for 2 endpoints, ${\mathit{\mu }}_h=(0.25,1,2,3{)}^{\prime }$, $\sqrt{diag(\sum _h)}=(0.15,0.25,0.4,0.6{)}^{\prime }$ for 4 endpoints, and ${\mathit{\mu }}_h=(0.1,0.25,0.5,1,1.5,2,2.5,3{)}^{\prime }$, $\sqrt{diag(\sum _h)}=(0.1,0.15,0.2,0.25,0.3,0.4,0.5,0.6{)}^{\prime }$ for 8 endpoints, respectively for all treatment groups $h=1,\dots ,p-1$, where the sample size was $n_h=20$ for each endpoint of each group. The last group had parameters: $\sqrt{diag(\sum _h)}=(0.5,0.9{)}^{\prime }$ for 2 endpoints, $\sqrt{diag(\sum _h)}=(0.3,0.5,0.65,0.9{)}^{\prime }$ for 4 endpoints, and $\sqrt{diag(\sum _h)}=(0.2,0.3,0.4,0.5,0.55,0.65,0.8,0.9{)}^{\prime }$ for 8 endpoints, respectively, where the sample size was $n_p=10$ for each endpoint. As mean and variance of a log-normally distributed variable depend on each other, the means of the last group were chosen accordingly. The endpoints had a random correlation structure.

Tables 3 and 4 show the simulated $\mathrm{\alpha }$-level for Dunnett and Tukey contrasts, respectively, for the multivariate log-normal data. As expected, simulated and nominal $\mathrm{\alpha }$-level differ. Independent of the number of treatment groups, endpoints, or contrast, the procedures yield conservative test decisions, except for HOM which is liberal (ranges from 0.074 to 0.162). MIN (ranges from 0.021 to 0.034) is more conservative than CE (ranges from 0.022 to 0.035), and BON is most conservative (ranges from 0.015 to 0.028). Of course, HOM is not surprising as it still ignores the heteroscedasticity problem. Hence, even in this situation, where the procedures MIN and CE are not developed for, they show an acceptable behaviour in the sense that they do not exceed the nominal $\mathrm{\alpha }$-level. Of course, they cannot be recommended without caution if the data do not follow a multivariate normal distribution.

For test decisions, as well as for the estimation of the contrasts ${\mathrm{\eta }}_{11},\dots ,{\mathrm{\eta }}_{qk}$, approximate $(1-\mathrm{\alpha })100\mathrm{\%}$ SCIs can be derived. The following intervals are related to the MIN procedure. Corresponding lower limits are given by ${\hat{\mathrm{\eta }}}_{li}^{\mathrm{l}\mathrm{o}\mathrm{w}\mathrm{e}\mathrm{r}}={\hat{\mathrm{\eta }}}_{li}-t_{qk,1-\mathrm{\alpha }}({\mathrm{\nu }}_l,{\hat{\mathit{R}}}^{het})\sqrt{{\sum }_{h=1}^p\frac{c_{lh}^2{S}_{hi}^2}{n_h}}\left(l=1,\dots ,q;i=1,\dots ,k\right),$

where $t_{qk,1-\mathrm{\alpha }}({\mathrm{\nu }}_l,{\hat{\mathit{R}}}^{het})$ is the lower $(1-\mathrm{\alpha })$-quantile of the related qk-variate t-distribution, and ${\hat{\mathit{R}}}^{het}$ is the estimation of the correlation matrix ${\mathit{R}}^{het}$. Hence, statistical problem (1) can also be decided as follows: for a specified level $\mathrm{\alpha }$, reject $H_0^{(li)}$ for ${\mathrm{\eta }}_{li}$, if ${\hat{\mathrm{\eta }}}_{li}^{\mathrm{l}\mathrm{o}\mathrm{w}\mathrm{e}\mathrm{r}}>{\mathrm{\delta }}_i$. Note that these intervals do not have same widths. This is because the intervals depend on different degrees of freedom, which are different for the contrast, and on the sample variances, which are different for the endpoints and the treatment groups. Hence, an interval is wider if variances of the corresponding treatment groups and endpoints are greater, and tighter if smaller.

Homma et al. [19] published summary data for five multiple, continuous endpoints of the randomized, placebo-controlled phase IIb dose-finding study of a novel anti-muscarinic agent. For a one-way layout with a zero-dose placebo group ($h=1$) and imidafenacin dose groups of 0.1 ($h=2$), 0.2 ($h=3$), and 0.5 ($h=4$) mg/day, the percentage changes from the baseline were defined as primary efficacy endpoints: (1) incontinence episodes per week (Iepw), (2) urgency incontinence episodes per week (Uiepw), (3) micturitions per day (Mpd), (4) urgency episodes per day (Uepd), and (5) urine volume voids per micturition (Uvvpm). Although these endpoints are baseline-adjusted ratios, they are treated as approximately normally distributed endpoints. The authors published means, standard deviations, sample sizes, and adjusted p-values of separate Dunnett [2] procedures. Because the raw data were not available, multivariate normally distributed data were generated covering Table 2 of Homma et al. [19] by using the package

SimComp

ermvnorm()

representing plausible highly positive (e.g. 0.8 for Uiepw and Uepd) and lightly negative (e.g. –0.3 for Mpd and Uvvpm) correlations for these multiple urinary endpoints. Note that means and standard deviations of the generated and of the original data set are exactly the same. Actually, some changes to the baseline were negative. The related data were multiplied by minus one, so that all endpoints have the same positive direction and higher values indicate a better treatment effect. The standard deviations per endpoint clearly differ depending on the treatment group. For example, endpoint Uiepw has standard deviations 272.76 (Placebo), 72.88 (Imid0.1), 41.11 (Imid0.2), and 93.45 (Imid0.5).

These data are the same as already used in Hasler and Hothorn [3]. The authors considered SCIs for ratios of means, and contrasts related to the trend test of Williams [4]. Here, SCIs for differences of means, and contrasts related to Dunnett [2] are applied. The Dunnett-type contrast matrix is given by $C={\left(c_{lh}\right)}_{l,h}=\left(\begin{array}{cccc}-1& 1& 0& 0\\ -1& 0& 1& 0\\ -1& 0& 0& 1\end{array}\right),$

where the rows represent the comparisons versus the placebo, and the columns represent the treatment groups. The differences of interest are ${\mathrm{\eta }}_{li}={\mathrm{\mu }}_{l+1,i}-{\mathrm{\mu }}_{1i}(l=1,2,3,i=1,\dots ,5),$

and the hypotheses to be tested are given by $H_0^{(li)}:{\mathrm{\eta }}_{li}\le 0(l=1,2,3,i=1,\dots ,5).$

Table 6 shows the estimated differences to the placebo (Estimate), p-values of Dunnett tests, separately for the endpoints and assuming homogeneous variances for the dose groups according to Homma et al. [19] (p-val. (prev.), adjusted p-values according to the multivariate MIN procedure described (p-val. (adj.), and related lower simultaneous confidence limits (Lower limit) for each dose and each endpoint. By definition, the multivariate procedure is more conservative than elementary Dunnett tests separately for the endpoints, each at level $\mathrm{\alpha }$ (p-val. (prev.)). Consequently, column p-val. (adj.) in Table 6 has substantially higher values than column p-val. (prev.), i.e. there is a price to pay for selecting at least one of $q=p-1$ doses and at least one of k endpoints. This fulfils the conservativeness principle of claims in randomized clinical trials. Furthermore, Homma et al. [19] have assumed homogeneous variances for the dose groups, which does not seem to be fulfilled and causes biased test decisions. This example shows how much the assumptions about the data and the method for incorporating the endpoints can influence test decisions. All dose effects for endpoint Uiepw are insignificant, whereas according to Homma et al. [19] the Imid0.2 effect is significant. Except for Uiepw, Imid0.5 shows a significant effect for all endpoints. For example, Imid0.5 causes at least 18.55 percent more change from baseline compared to placebo for Iepw. The two lower doses, Imid0.2 and Imid0.1, show significances for Iepw and Uepd, respectively.

The package

SimComp

SimCiDiff(data=data object,
grp="name of the grouping variable",
resp=c("name of endpoint 1"," name of endpoint 2","..."),
type="Dunnett",
base=alphanumerical number of the control group,
alternative="greater",
covar.equal=FALSE).

For the related p-values use the command

SimTestDiff()

SimTestRat()

SimCiRat()

For the special cases of contrasts related to Dunnett [2] and Williams [4], Hasler and Hothorn [1, 3] had proposed extensions to the case of multiple endpoints. Their approaches have been generalized in this article to any MCTs and to the situation of heteroscedasticity, i.e. unequal covariance matrices $\sum _1,\dots ,\sum _p$ for the treatment groups. Approximate multivariate t-distributions will be applied, based on the approach of Hasler and Hothorn [10]. Correlations among both the contrasts and the endpoints will be taken into account. Test decisions – adjusted p-values as well as SCIs – are available for all contrasts and all endpoints. The intervals and tests may be one- or two-sided. An adaption to a formulation based on ratios of means is possible.

In the presence of heteroscedasticity, the CE procedure with contrast- and endpoint-specific degrees of freedom tends to liberal test decisions, whereas the MIN procedure with (minimized) contrast-specific degrees of freedom maintains the FWE in the strong sense in a passable range. This was shown by simulations. Possible slight variations around the nominal FWE $\mathrm{\alpha }$ are necessarily bounded by the versions CE and BON. A naive competing approach would obviously be the application of conventional MCTs for heteroscedastic data according to Hasler and Hothorn [10] for all the endpoints separately and to adjust for the multiple endpoints by methods of Holm [20] or Hommel [21]. Expectedly, it would also realize a FWE between the versions CE and BON. However, such an approach would not provide (meaningful) SCIs. Furthermore, similar to the BON procedure, it would not exploit the correlations between the endpoints.

If the assumption of a multivariate normal distribution for the data is not fulfilled, the procedures considered cannot be recommended without caution. The simulation results show that they yield conservative test decisions if the data follow distributions with a positive skew. In this case or for other non-normal distributions, non-parametric procedures – like those of Munzel and Brunner [22]; Bathke and Harrar [23]; Harrar and Bathke [24, 25] – should be used instead. Note that these procedures do not provide multiplicity-adjusted p-values or SCIs for each contrast-endpoint combination as they use ANOVA-type ${\mathrm{\chi }}^2$ or F statistics, respectively.

The main advantage of usual MCTs over other testing procedures is that the correlations of the contrasts are taken into account. These correlations mainly depend on whether and which treatment means are involved in the contrasts. All treatment means, however, which are involved in the same contrast are independent. Consequently, the MIN procedure can also be used for the analysis of repeated measures by simply regarding the time points as endpoints, as long as the condition $(n_h-1)\ge k$ is fulfilled for all $h=1,\dots ,p$. However, the contrasts only apply to the treatment means. Comparisons related to the time points are not possible then. These two testing problems seem similar but they are not. Potential contrasts associated with the time points have the problem that the means which then are involved in the same contrast are no longer independent. Hence, approaches to overcome this problem are not the same as considered in this article. MCTs and SCIs for repeated measures are described by Hasler [26], for example.

A solution to the following problem might be a task for the future: the MIN procedure presented generally assumes unequal covariance matrices which occur if variances or correlations of some endpoints differ depending on the treatment groups. The procedure does not distinguish whether unequal correlations or unequal variances lead to unequal covariances. In practice, however, it can occur that just the variances differ, and the correlation structure stays the same for the treatment groups (or the other way round).