Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter December 21, 2020

Identifying Causal Channels of Policy Reforms with Multiple Treatments and Different Types of Selection

Annabelle Doerr and Anthony Strittmatter ORCID logo

Abstract

We study the identification of channels of policy reforms with multiple treatments and different types of selection for each treatment. We disentangle reform effects into policy effects, selection effects, and time effects under the assumption of conditional independence, common trends, and an additional exclusion restriction on the non-treated. Furthermore, we show the identification of direct- and indirect policy effects after imposing additional sequential conditional independence assumptions on mediating variables. We illustrate the approach using the German reform of the allocation system of vocational training for unemployed persons. The reform changed the allocation of training from a mandatory system to a voluntary voucher system. Simultaneously, the selection criteria for participants changed, and the reform altered the composition of course types. We consider the course composition as a mediator of the policy reform. We show that the empirical evidence from previous studies reverses when considering the course composition. This has important implications for policy conclusions.

JEL-Classification: C21; J68; H43

Corresponding author: Anthony Strittmatter, CREST-ENSAE, 5 Avenue Le Chatelier, 91120 Palaiseau, France; University St. Gallen, St. Gallen, Switzerland; and CESifo, Munich, Germany, E-mail:

  1. *This study is part of the project “Regional Allocation Intensities, Effectiveness and Reform Effects of Training Vouchers in Active Labor Market Policies”, IAB project 1155. This is a joint project of the Institute for Employment Research (IAB) and the University of Freiburg. We gratefully acknowledge financial and material support from the IAB. The paper was presented at ESPE in Aarhus, CAFE Workshop in Børkop, SOLE in Washington, EALE in Ljubljana, Joint Research Centre of the European Commission, Centre for European Economic Research, and the University of Bern. We thank participants for helpful comments, in particular Hugo Bodory, Bernd Fitzenberger, Hans Fricke, Michael Lechner, Michael Knaus, Thomas Kruppe, Marie Paul, and Gesine Stephan. We are particularly grateful for detailed comments and remarks from Conny Wunsch. Furthermore, we thank two anonymous referees. The usual disclaimer applies.

References

Abadie, A. 2005. “Semiparametric Difference-in-Differences Estimators.” The Review of Economic Studies 72 (1): 1–19. https://doi.org/10.1111/0034-6527.00321.Search in Google Scholar

Baron, R. M., and D. A. Kenny. 1986. “The Moderator-Mediator Variable Distinction in Social Psychological Research: Conceptual, Strategic, and Statistical Considerations.” Journal of Personality and Social Psychology 51: 1173–82. https://doi.org/10.1037/0022-3514.51.6.1173.Search in Google Scholar

Biewen, M., B. Fitzenberger, A. Osikominu, and M. Paul. 2014. “The Effectiveness of Public Sponsored Training Revisited: The Importance of Data and Methodological Choices.” Journal of Labor Economics 32 (4): 837–97. https://doi.org/10.1086/677233.Search in Google Scholar

Bruttel, O. 2005. “Delivering Active Labour Market Policy Through Vouchers: Experiences with Training Vouchers in Germany.” International Review of Administrative Sciences 71 (3): 391–404. https://doi.org/10.1177/0020852305056809.Search in Google Scholar

Card, D., and D. R. Hyslop. 2005. “Estimating the Effects of a Time-Limited Earnings Subsidy for Welfare-Leavers.” Econometrica 73 (6): 1723–70. https://doi.org/10.1111/j.1468-0262.2005.00637.x.Search in Google Scholar

Cox, D. R. 1958. “The Regression Analysis of Binary Sequences.” Journal of the Royal Statistical Society: Series B (Methodological) 20: 215–42. https://doi.org/10.1111/j.2517-6161.1958.tb00292.x.Search in Google Scholar

Deuchert, E., M. Huber, and M. Schelker. 2019. “Direct and Indirect Effects Based on Difference-in-Differences with an Application to Political Preferences Following the Vietnam Draft Lottery.” Journal of Business and Economic Statistics 37 (4): 710–20. https://doi.org/10.1080/07350015.2017.1419139.Search in Google Scholar

Doerr, A., B. Fitzenberger, T. Kruppe, M. Paul, and A. Strittmatter. 2017. “Employment and Earnings Effects of Awarding Training Vouchers in Germany.” Industrial and Labor Relations Review 70 (3): 767–812. https://doi.org/10.1177/0019793916660091.Search in Google Scholar

Felfe, C., N. Nollenberger, and N. Rodriguez-Planas. 2014. “Can’t Buy Mommy’s Love? Universal Childcare and Children’s Long-Term Cognitive Development.” Journal of Population Economics 283 (2): 393–422. https://doi.org/10.1007/s00148-014-0532-x.Search in Google Scholar

Flores, C., and A. Flores-Lagunes. 2009. “Identification and Estimation of Causal Mechanisms and Net Effects of a Treatment under Unconfoundedness.” IZA Discussion Paper, 4237.Search in Google Scholar

Fricke, H. 2017. “Identifcation Based on Difference-in-Differences Approaches with Multiple Treatments.” Oxford Bulletin of Economics and Statistics 79 (3): 426–33. https://doi.org/10.1111/obes.12178.Search in Google Scholar

Graham, B. S., C. C. De Xavier Pinto, and D. Egel. 2016. “Efficient Estimation of Data Combination Models by the Method of Auxiliary-to-Study Tilting.” Journal of Business & Economic Statistics 34 (2): 288–301. https://doi.org/10.1080/07350015.2015.1038544.Search in Google Scholar

Gundersen, C., B. Kreider, J. Pepper, and V. Tarasuk. 2017. “Food Assistance Programs and Food Insecurity: Implications for Canada in Light of the Mixing Problem.” Empirical Economics 52 (3): 1065–87. https://doi.org/10.1007/s00181-016-1191-4.Search in Google Scholar

Havnes, T., and M. Mogstad. 2011a. “Money for Nothing? Universal Child Care and Maternal Employment.” Journal of Public Economics 95 (11–12): 1455–65. https://doi.org/10.1016/j.jpubeco.2011.05.016.Search in Google Scholar

Havnes, T., and M. Mogstad. 2011b. “No Child Left Behind: Subsidized Child Care and Children’s Long-Run Outcomes.” American Economic Journal: Economic Policy 3 (2): 97–129. https://doi.org/10.1257/pol.3.2.97.Search in Google Scholar

Heckman, J. J., H. Ichimura, and P. E. Todd. 1997. “Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme.” The Review of Economic Studies 64 (4): 605–54. https://doi.org/10.2307/2971733.Search in Google Scholar

Huber, M. 2014. “Identifying Causal Mechanisms (Primarily) Based on Inverse Probability Weighting.” Journal of Applied Econometrics 29 (6): 920–43. https://doi.org/10.1002/jae.2341.Search in Google Scholar

Huber, M., M. Lechner, and G. Mellace. 2017. “Why Do Tougher Caseworkers Increase Employment? The Role of Programme Assignment as a Causal Mechanism.” The Review of Economics and Statistics 99 (1): 180–3. https://doi.org/10.1162/rest_a_00632.Search in Google Scholar

Huber, M., M. Lechner, and A. Strittmatter. 2018. “Direct and Indirect Effects of Training Vouchers for the Unemployed.” Journal of the Royal Statistical Society, Series A 181 (2): 441–63. https://doi.org/10.1111/rssa.12279.Search in Google Scholar

Huber, M., M. Schelker, and A. Strittmatter. 2020. “Direct and Indirect Effects Based on Changes-in-Changes.” Journal of Business and Economic Statistics forthcoming, https://doi.org/10.1080/07350015.2020.1831929.Search in Google Scholar

Imai, K., L. Keele, D. Tingley, and T. Yamamoto. 2011. “Unpacking the Black Box of Causality: Learning about Causal Mechanisms from Experimental and Observational Studies.” American Political Science Review 105 (4): 765–89. https://doi.org/10.1017/s0003055411000414.Search in Google Scholar

Imai, K., L. Keele, and T. Yamamoto. 2010. “Identification, Inference and Sensitivity Analysis for Causal Mediation Effects.” Statistical Science 25: 51–71. https://doi.org/10.1214/10-sts321.Search in Google Scholar

Imbens, G. 2000. “The Role of the Propensity Score in Estimating Dose-Response Functions.” Biometrika 87 (3): 706–10. https://doi.org/10.1093/biomet/87.3.706.Search in Google Scholar

Kikuchi, N. 2017. “Intergenerational Transmission of Education in Japan: Nonparametric Bounds Analysis with Multiple Treatments.” ISER Discussion Paper No. 1011.10.2139/ssrn.3039898Search in Google Scholar

Lechner, M. 1999. “Earnings and Employment Effects of Continuous Off-the-Job Training in East Germany after Unification.” Journal of Business and Economic Statistics 17 (1): 74–90. https://doi.org/10.1080/07350015.1999.10524798.Search in Google Scholar

Lechner, M. 2001. “Identification and Estimation of Causal Effects of Multiple Treatments under the Conditional Independence Assumption,” In Econometric Evaluation of Labour Market Policies, edited by M. Lechner, and F. Pfeiffer, 43–58. ZEW Economic Studies 13. New York: Springer-Verlag.10.1007/978-3-642-57615-7_3Search in Google Scholar

Lechner, M. 2010. “The Estimation of Causal Effects by Difference-in-Difference Methods.” Foundations and Trends in Econometrics 4 (3): 165–224. https://doi.org/10.1561/0800000014.Search in Google Scholar

Lechner, M., R. Miquel, and C. Wunsch. 2011. “Long-run Effects of Public Sector Sponsored Training.” The Journal of the European Economic Association 9 (4): 742–84. https://doi.org/10.1111/j.1542-4774.2011.01029.x.Search in Google Scholar

Lechner, M., and J. Smith. 2007. “What is the Value Added by Caseworkers?” Labour Economics 14 (2): 135–51. https://doi.org/10.1016/j.labeco.2004.12.002.Search in Google Scholar

Lechner, M., and A. Strittmatter. 2019. “Practical Procedures to Deal with Common Support Problems in Matching Estimation.” Econometric Reviews 38 (2): 193–207. https://doi.org/10.1080/07474938.2017.1318509.Search in Google Scholar

Lechner, M., and C. Wunsch. 2013. “Sensitivity of Matching-Based Program Evaluations to the Availability of Control Variables.” Labour Economics 21 (C): 111–21. https://doi.org/10.1016/j.labeco.2013.01.004.Search in Google Scholar

Manski, C. 1997. “The Mixing Problem in Programme Evaluation.” The Review of Economic Studies 64 (4): 537–53. https://doi.org/10.2307/2971730.Search in Google Scholar

McCall, B., J. A. Smith, and C. Wunsch. 2016. “Government-Sponsored Vocational Education for Adults.” Handbook of the Economics of Education 5: 479–652. https://doi.org/10.1016/b978-0-444-63459-7.00009-9.Search in Google Scholar

Pearl, J. 2001. “Direct and Indirect Effects.” In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, 411–20. San Francisco: Morgan Kaufmann.10.1145/3501714.3501736Search in Google Scholar

Perez-Johnson, I., Q. Moore, and R. Santillano. 2011. Improving the Effectiveness of Individual Training Accounts: Long-Term Findings from an Experimental Evaluation of Three Service Delivery Models. Final Report. Princeton, NJ: Mathematica Policy Research.Search in Google Scholar

Petersen, M. L., S. E. Sinisi, and M. J. van der Laan. 2006. “Estimation of Direct Causal Effects.” Epidemiology 17: 276–84. https://doi.org/10.1097/01.ede.0000208475.99429.2d.Search in Google Scholar

Rinne, U., A. Uhlendorff, and Z. Zhao. 2013. “Vouchers and Caseworkers in Training Programs for the Unemployed.” Empirical Economics 45 (3): 1089–127. https://doi.org/10.1007/s00181-012-0662-5.Search in Google Scholar

Robins, J., and S. Greenland. 1992. “Identifiability and Exchangeability for Direct and Indirect Effects.” Epidemiology 3: 143–55. https://doi.org/10.1097/00001648-199203000-00013.Search in Google Scholar

Rosenbaum, P., and D. Rubin. 1983. “The Central Role of Propensity Score in Observational Studies for Causal Effects.” Biometrica 70 (1): 41–55. https://doi.org/10.1093/biomet/70.1.41.Search in Google Scholar

Rubin, D. B. 1974. “Estimating the Causal Effect of Treatments in Randomized and Non-Randomized Studies.” Journal of Educational Psychology 66 (5): 688–701. https://doi.org/10.1037/h0037350.Search in Google Scholar

Strittmatter, A. 2016. “What Effect Do Vocational Training Vouchers Have on the Unemployed?” IZA World of Labor 316, https://doi.org/10.15185/izawol.316.Search in Google Scholar

Tomini, F., W. Groot, and H. Maassen van den Brink. 2016. “The Effectiveness of the Voucher Training Programs: A Systematic Review of the Evidence from Evaluations.” TIER Working Paper Series, 16/08.Search in Google Scholar

Twinam, T. 2017. “Complementarity and Identification.” Econometric Theory 33 (5): 1154–85. https://doi.org/10.1017/s0266466616000359.Search in Google Scholar

Van der Weele, T. J. 2009. “Marginal Structural Models for the Estimation of Direct and Indirect Effects.” Epidemiology 20: 18–26. https://doi.org/10.1097/ede.0b013e31818f69ce.Search in Google Scholar


Article note

This study is part of the project “Regional Allocation Intensities, Effectiveness and Reform Effects of Training Vouchers in Active Labor Market Policies”, IAB project 1155. This is a joint project of the Institute for Employment Research (IAB) and the University of Freiburg. We gratefully acknowledge financial and material support from the IAB. The paper was presented at ESPE in Aarhus, CAFE Workshop in Børkop, SOLE in Washington, EALE in Ljubljana, Joint Research Centre of the European Commission, Centre for European Economic Research, and the University of Bern. We thank participants for helpful comments, in particular Hugo Bodory, Bernd Fitzenberger, Hans Fricke, Michael Lechner, Michael Knaus, Thomas Kruppe, Marie Paul, and Gesine Stephan. We are particularly grateful for detailed comments and remarks from Conny Wunsch. Furthermore, we thank two anonymous referees. The usual disclaimer applies.



Supplementary Material

The online version of this article offers supplementary material (https://doi.org/10.1515/jem-2019-0012).


Received: 2019-04-15
Accepted: 2020-11-05
Published Online: 2020-12-21

© 2020 Walter de Gruyter GmbH, Berlin/Boston

Scroll Up Arrow