Abstract
The before-and-after study with multiple unaffected control groups is widely applied to study treatment effects. The current methods usually assume that the control groups’ differences between the before and after periods, i.e. the group time effects, follow a normal distribution. However, there is usually no strong a priori evidence for the normality assumption, and there are not enough control groups to check the assumption. We propose to use a flexible skew-t distribution family to model group time effects, and consider a range of plausible skew-t distributions. Based on the skew-t distribution assumption, we propose a robust-t method to guarantee nominal significance level under a wide range of skew-t distributions, and hence make the inference robust to misspecification of the distribution of group time effects. We also propose a two-stage approach, which has lower power compared to the robust-t method, but provides an opportunity to conduct sensitivity analysis. Hence, the overall method of analysis is to use the robust-t method to test for the overall hypothesized range of shapes of group variation; if the test fails to reject, use the two-stage method to conduct a sensitivity analysis to see if there is a subset of group variation parameters for which we can be confident that there is a treatment effect. We apply the proposed methods to two datasets. One dataset is from the Current Population Survey (CPS) to study the impact of the Mariel Boatlift on Miami unemployment rates between 1979 and 1982.The other dataset contains the student enrollment and grade repeating data in West Germany in the 1960s with which we study the impact of the short school year in 1966–1967 on grade repeating rates.
0.2 no matter how large the treatment effect is.
be the average outcome of group k in period t assuming treatment was never applied to that group, and we model
as
indicates the period before and after the treatment,
is the overall baseline level,
is the time effect of group k, and
’s are independent normally distributed random errors with known standard deviation
which signify the random measurement errors in our observations.
and
represent the treatment and control groups, respectively. Letting
be the average observed response for group k in period t, it follows
is the causal effect of applying treatment in the groups k belonging to
in the after period.
has the same distribution as
. This means that we assume that the group(s) to which the treatment was applied was effectively randomly chosen. Presence of control groups allows us to separate the treatment effect
from the time effect
. If there is only one treatment and one control group, then it is common to assume time effects
to be constant across groups, while if there are several treatment and/or control groups, the variation in the time effects can be taken into account by assuming that, for example,
’s follow a normal distribution with unknown variance. However, there is usually no strong a priori evidence for the normal distribution assumption.
, we assume that
are independently and identically distributed as a skew-
distribution [
controlling the skewness and scale, where
is the nuisance scaling parameter.
,
denotes the beta function, and a and b are positive real numbers. When
reduces to the density of the t-distribution with 2a degrees of freedom, and hence when
and they are large, this distribution is similar to the normal distribution. When
or
is skewed to the left or right, respectively. The density functions of skew-t distributions with a range of parameters as well as the normal distribution are drawn in 
was obtained from a complex survey, along with standard error of the estimator
. Considering the size of the dataset, it is appropriate to assume that the estimated unemployment rate
is the true unemployment rate with an error following
distribution. We regard the unemployment rates as observations. Hence, there is one treatment group (Miami) and four control groups (the other four cities), and each group has one observation before and after the treatment period. Consider
to be the unemployment rate in kth city in year 1979 (if
) or 1982 (if
) estimated from the survey, along with
be the standard error of the estimator (see 
be indicator of Miami and
be the indicators of the four unaffected cities. Letting
and the Mariel Boatlift effect be denoted by
the unemployment rates can be modeled by eq. [
, and hence provides the researcher an idea about what kind of assumptions are necessary to reject the null hypothesis (
and
for skew-t distribution, and show that they are not able to yield hypothesis tests with the nominal significance levels when there are few treatment and control groups and when group time effects are not normally distributed.
,
is the indicator of treatment group,
are the indicators of control groups, and
and
for skew-t distribution are given. With
and standard deviation of
and
to be 1 for
the true rejection rates under a collection of the parameters
’s are listed in 
can be found in 
, where
.
.
is the sample standard deviation of the treatment observations
for
, and
is the sample standard deviation of the control observations
for
.
are independently and identically distributed as a skew-
distribution [
. The idea is to use simulation to determine the cut-off values of the t-statistic to guarantee nominal significance level under this set of plausible parameters a, b and
’s. Hence, the hypothesis test with the cut-off values determined by simulation is robust to the distribution of group time effects deviating from normality over the considered range of parameters and hence we call it the robust-t method.
and
to be
the same for all k. According to Proposition 2 (See 
and
simultaneously with the same scale factor would not change the rejection rates under the null hypothesis. We only need to consider a range of
, rather than
and
separately, to determine the cut-off values of the t statistic in the robust-t method. In practice, when the standard deviations 
and
differ, we just use the corresponding standard deviations of
and
in the simulation, and let
’s range be the maximum of
across all possible k and t. We adjust the cut-off values of the t-statistic to guarantee the required significance level under the above settings of
,
and
. Suppose one wants to conduct the hypothesis test at significance level 0.05, the test procedure is summarized as below.
and
to be
Specify a wide range of
. When the standard deviations
and
differ, we just use the corresponding standard deviations of
and
in the simulation, and let
’s range be the maximum of
across all possible k and t.
, simulate datasets according to model 2, obtain the t-statistics, and calculated the 0.025, 0.05, 0.95, 0.975 quantiles. (Note that if one wants to test with significance level other than 0.05, one just needs to modify the quantiles correspondingly.)
and
, we calculate the minimum of the 0.025,0.05 quantiles, and the maximum of 0.95,0.975 quantiles under the above settings. We use the minimum of 0.05 quantiles as the cut-off values of the left-sided test, the maximum of 0.95 quantiles as the cut-off value of the right-sided test, and minimum of 0.025 quantiles and maximum of 0.975 quantiles as the cut-off value of the two-sided test. This guarantees that the robust-t method test has significance level 0.05 under all the above settings of
and
.
. It only requires a vague range of the parameters
and
, and does not require specification of
. Noting that we assume
, rather than
, is following skew-
, the distribution of
implied by the null hypothesis is properly centered at 0. Even if the skew-
is extremely skew, the implied distribution of
is symmetric. For example, it will always accept the null hypothesis if
.
of the skew-t distribution is available, another method overcoming the difficulty that
is unknown is to conduct a two-stage inference.
we may use a simulation-based two-stage method to test hypothesis (3).
with significance level
with an unknown nuisance parameter
the basic two-stage inference is conducted as follows.
where
.
of
with coverage
.
conduct the hypothesis test with significance level
.
we accept the null hypothesis. Then under null hypothesis, the false rejection rate is upper-bounded by
.
is the interval with the left end to be 0 and the right end to be
upper bound of
denoted by
. One can prove that given larger
, the null hypothesis is more likely to be rejected in the second stage. Thus in the second stage, we only need to test the null hypothesis with
. The whole test procedure is summarized as follows.
values.
estimate an upper bound of
with coverage
with Monte Carlo simulation. The procedure to find the upper bound of
is:
for the upper bound, for example, 0.001. Set sufficiently large number to be
and 0 to be
. Set k
0 and
.
and
, and determine the empirical quantile (denoted by
) of the observed standard deviation of the control groups’ time effects.
set
. If
set k to be
and return to step (b). If
exit and use
as the upper bound of
.
set
. If
set k to be
and return to step (b). If
exit and use
as the upper bound of
.
to be
. Let the test statistic to be
Compute the value
of the test statistic T from the data. With
and eq. [
.
depending on the empirical quantile of the observed test statistic value
among the estimated distribution computed on the previous step, with significance level
.
. Usually, the two-stage procedure is conservative, which means the rejection rate under the null hypothesis could be much smaller than
.
and
. We conducted a simulation study, and found that the choice of
and
has only slight influence on the rejection rate under the null hypothesis. In the rest of the paper, we set
.
is available. For given
assuming
is known, one can conduct the hypothesis test
as the second stage in the two-stage method. The test result may be heavily influenced by
, and a sensitivity analysis can be conducted to find a subset of
with which the null hypothesis is rejected.
to range from low to high and investigate how the p-value of the hypothesis test
changes along with
. The procedure is summarized as follows:
values.
specify a range of
’s.
in the range considered, compute the value
of the test statistic T from the data, calculated the p-value of the test, and see how the p-value would change with
.
’s and the corresponding p-values, the method provides a researcher with an idea of what kinds of assumptions about the variation, skewness, and heavy-tailedness of the control groups’ differences would be necessary to infer that the treatment has an effect. See the applications in 
and
. In many of the cases, it is not difficult to make the range large enough to guarantee to cover the range of parameters of interest. For example, in the study of mass immigration effect on Miami unemployment rates, observations are proportions, and hence differences before and after the treatment period should be within
. It implies that
could not be too large. Also,
can be set to arrange from small to large (which represents heavy-tailed to thin-tailed), and from
to
or
(which represents symmetry to heavy asymmetry) to cover the shape of the distribution of group time effects. The robust-t method is robust to a wide range of misspecification of
. We set uniform cut-off values for the test for all the
within the range of interest, and guarantee required significance level.
. We compare the rejection rates and powers of the two methods with simulation as in 
. The advantages of the robust-t method suggest that we use the robust-t method to conduct the hypothesis test for the overall range of
of interest in the before-and-after study when there are few control and/or treatment groups.
and
for which we can be confident that there is a treatment effect.
be the estimated unemployment rate in the kth city in year 1979 (if
) or 1982 (if
). Let
be indicator of Miami and
be the indicators of the four unaffected cities. The standard error of the estimation is estimated with binomial approximation corrected by a sampling strategy factor provided by CPS. Considering the size of the dataset, it is appropriate to assume that the estimated
is the true unemployment rate with an error following
distribution, where
is the estimated standard error of estimated
. Letting
and the Mariel Boatlift effect denoted by
the unemployment rate data can be models by eq. [
and
as follows.
by the ratio of number of unemployed sampled individuals and available labor force in the dataset. We use the variable “ftpt79 (Full-time or part-time labor force status)” as the indicator of employment condition. The value of this variable includes: not in labor force (0), employed full-time (1), part-time for economic reasons (2), unemployed full-time (3), employed part-time (4), and unemployed part-time (5). We estimate the unemployment rate by the ratio of the number of individuals with value (3) and (5) to the number of individuals with value values (1), (2), (3), (4) or (5).
is estimated as the standard error of estimated
. For each city k at time t, let x be the number of unemployed people and y be the labor force, we have
. The CPS calculate the standard error of
by
are the design effect coefficients on standard errors. We use the design effects for total monthly variances after composing on Unemployment and Civilian Labor Force as Table 14–
,
and obtain
as the standard error of unemployment rate. The estimated standard error of city k at time t are treated as the standard deviation of the error terms
, for
and
range from 2 to 160,
and
range from 2–10 to 210. Hence, our test is robust to a wide range of
the scaling factor of the skew-t distribution, and to a wide range of parameters
for the skew-t distribution changing from seriously skewed to symmetric, from heavy-tailed to thin-tailed. Using the simulation procedure specified in 
we use to generate the datasets.
to be various values and look at how the p-value of the hypothesis test
changes along with the
. As an example, we summarize the results with Blacks’ unemployment rate with a small collection of
’s in 
, the treatment effect is not statistically significant because of the variation in the estimated unemployment rates, and hence there is not significant evident to conclude that the migration affects the Blacks’ unemployment rates.
for blacks.
be the grade repeating rate in kth state in year 1965 (if
) or 1966 (if
), which is estimated by the ratio of the number of students repeating a grade and the number of students enrolled in the corresponding grade.
is the standard error of the corresponding grade repeating rate, which is estimated by binomial approximation. Let
be the indicators of the seven treatment states (Schleswig-Holstein, Bremen, Nordrhein-Westphalen, Hessen, Rheinland-Pfalz, Baden-Württemberg and Saarland) and
be the indicators of the three unaffected states (Hamburg, Bayern and Berlin). The short school year effect is represented by
and the hypothesis test of interest is
ranging from 2 to 160,
, and
ranging from 2–10 to 210, using to the simulation procedure specified in 
we use to generate the datasets.
to be various value and look at how the p-value of the hypothesis test
changes along with the
. As an example, we summarize the results with the grade 2 repeating rate with a small collection of
’s in 
is small to conclude that the short school year affects the grade 2 repeating rate. Typically
needs to be smaller than 0.005 for the collection of
’s we tested.
in stage 1. For the collection of
’s we test and the grade 2 repeating rate data, the 0.975 upper bounds of
varies from 0.002 to 0.01, which is not large comparing to the 0.005 threshold. If one assumes
to be smaller than 0.005, it may be concluded that the short school year affects the grade 2 repeating rate.
for grade 2 repeating rate.
, the variation parameter of group time effects. In the study of a real problem, we propose to use the robust-t method to test for the overall range of shapes of group variation. If the test fails to reject, use the two-stage method to conduct a sensitivity analysis to see if there is a subset range of parameters for which we can be confident that there is a treatment effect. In a study with only one control group such as Card and Krueger [
is arbitrary but one should make the range broad enough to cover the situation (skewness, fat-tailedness, scale, etc.) of the dataset under research.
,
is the indicator of treatment group,
are the indicators of control groups, and
and
for skew-t distribution are given. With
and standard deviation of
and
to be 1 for
, besides 
under more settings of
are listed in 
and
to be
the same for all
. We assume that
range from 2 to 160,
and
ranges from 2–10 to 210.
.
range from 2 to 160. Scaling
and
simultaneously with the same scale factor would not change the rejection rates under the null hypothesis, according to the proposition 1 (
, rather than
and
separately, to investigate the rejection rates under null hypothesis in various settings. So we let
and
range from 2–10 to 210. Using the simulation procedure specified in 
we used to generate the datasets.
and
to be
the same for all k.
and
range from 1 to 100, and the ratio of
and
be 1, we summarize the powers of the two methods for some combinations of
in 
and the ratio of
and
, the results are similar. The figures show that, in general the robust-t method has larger power than the two-stage method.
from the given dataset to the statistics T’s from the generated datasets. However, when the skew-t distribution is very skewed, the statistics T’s from the generated datasets can be very spread out, and hence result in insignificant p-value even when the true
is large.
on each of the plot. The rejection rates under null hypothesis (which is the power when
) and the power under gamma and uniform distribution are close to the skew-t distribution, which shows the robustness of the method against misspecification.
of the skew-t distribution, consider two models. The first model follows eq. [
skew-
, and
while the second model follows eq. [
skew-
, and
. Conducting the robust-t method inference for the treatment effect with data following the two models, the rejection rates under the null hypothesis
are identical.
and
simultaneously with the same scale factor would not change the rejection rates under the null hypothesis. Thus we only need to consider a range of
, rather than
and
separately, to determine the cut-off values of the t statistic in the robust-t method.
follows the first model, it has
skew-
,
is the set of the indicators of treatment groups and
is the set of the indicators of control groups.
,
and
simultaneously by factor
the model becomes
and
then
skew-
and 
. Hence, the dataset
follows the second model.
of the
-t distribution, consider two models. The first model follows eq. [
skew-
, and
while the second model follows eq. [
skew-
, and
. Conducting the two-stage inference for the treatment effect with data following the two models, the rejection rates under the null hypothesis
are identical.
and
simultaneously with the same scale factor would not change the rejection rates under the null hypothesis. Thus we only need to consider a range of
, rather than
and
separately, to find the rejection rates under the null hypothesis.
follows the first model, it has
skew-
,
is the set of the indicators of treatment groups and
is the set of the indicators of control groups.
,
and
simultaneously by factor
the model becomes
and
then
skew-
and
. Hence, the dataset
follows the second model.
to be
for the first dataset and
for the second, then obviously
. In the second stage, letting the test statistic to be
for the first dataset and
for the second dataset, then under null hypothesis
. Thus these two test statistics would have identical quantiles among the distributions of test statistic generated by
and 
correspondingly. Thus, if we accept the null hypothesis for the first dataset, we accept the null hypothesis for the second dataset and vice versa. Hence, the rejection rates under null hypothesis are identical for the two models. Proof completed.▪
Comments (0)