Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Journal of Causal Inference

Ed. by Imai, Kosuke / Pearl, Judea / Petersen, Maya Liv / Sekhon, Jasjeet / van der Laan, Mark J.

Online
ISSN
2193-3685
See all formats and pricing
More options …

Synthetic Control Method: Inference, Sensitivity Analysis and Confidence Sets

Sergio Firpo / Vitor Possebom
Published Online: 2018-09-15 | DOI: https://doi.org/10.1515/jci-2016-0026

Abstract

We extend the inference procedure for the synthetic control method in two ways. First, we propose parametric weights for the p-value that includes the equal weights benchmark of Abadie et al. [1]. By changing the value of this parameter, we can analyze the sensitivity of the test’s result to deviations from the equal weights benchmark. Second, we modify the RMSPE statistic to test any sharp null hypothesis, including, as a specific case, the null hypothesis of no effect whatsoever analyzed by Abadie et al. [1]. Based on this last extension, we invert the test statistic to estimate confidence sets that quickly show the point-estimates’ precision, and the test’s significance and robustness. We also extend these two tools to other test statistics and to problems with multiple outcome variables or multiple treated units. Furthermore, in a Monte Carlo experiment, we find that the RMSPE statistic has good properties with respect to size, power and robustness. Finally, we illustrate the usefulness of our proposed tools by reanalyzing the economic impact of ETA’s terrorism in the Basque Country, studied first by Abadie and Gardeazabal [2] and Abadie et al. [3].

This article offers supplementary material which is provided at the end of the article.

Keywords: Synthetic Control Estimator; Hypothesis Testing; Sensitivity Analysis; Confidence Sets

References

  • 1.

    Abadie A, Diamond A, Hainmueller J. Comparative politics and the synthetic control method. Am J Polit Sci. 2015;59(2):495–510.CrossrefGoogle Scholar

  • 2.

    Abadie A, Gardeazabal J. The economic costs of conflict: A case study of the basque country. Am Econ Rev. 2003;93(1):113–32.CrossrefGoogle Scholar

  • 3.

    Abadie A, Diamond A, Hainmueller J. Synth: An R package for synthetic control methods in comparative case studies. J Stat Softw. 2011;42(13):1–17.Google Scholar

  • 4.

    Abadie A, Diamond A, Hainmueller J. Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. J Am Stat Assoc. 2010;105(490):493–505.CrossrefGoogle Scholar

  • 5.

    Bove V, Elia L, Smith RP. The relationship between panel and synthetic control estimators on the effect of civil war. Working paper. 2014. Available at http://www.bbk.ac.uk/ems/research/BirkCAM/working-papers/BCAM1406.pdf.Google Scholar

  • 6.

    Li Q. Economics consequences of civil wars in the post-world war II period. Macrotheme Rev. 2012;1(1):50–60.Google Scholar

  • 7.

    Montalvo JG. Voting after the bombings: A natural experiment on the effect of terrorist attacks on democratic elections. Rev Econ Stat. 2011;93(4):1146–54.CrossrefGoogle Scholar

  • 8.

    Yu J, Wang C. Political risk and economic development: A case study of China. Econ Res [Eknomska Istrazianja]. 2013;26(2):35–50.CrossrefGoogle Scholar

  • 9.

    Barone G, Mocetti S. Natural disasters, growth and institutions: A tale of two earthquakes. J Urban Econ. 2014;52–66.Google Scholar

  • 10.

    Cavallo E, Galiani S, Noy I, Pantano J. Catastrophic natural disasters and economic growth. Rev Econ Stat. 2013;95(5):1549–61.CrossrefGoogle Scholar

  • 11.

    Coffman M, Noy I. Hurricane iniki: Measuring the long-term economic impact of natural disaster using synthetic control. Environ Dev Econ. 2011;17:187–205.Google Scholar

  • 12.

    DuPont W, Noy I. What happened to Kobe? A reassessment of the impact of the 1995 earthquake in Japan. Econ Dev Cult Change. 2015;63(4):777–812. Available at http://www.economics.hawaii.edu/research/workingpapers/WP_12-4.pdf.CrossrefGoogle Scholar

  • 13.

    Mideksa TK. The economic impact of natural resources. J Environ Econ Manag. 2013;65:277–89.CrossrefGoogle Scholar

  • 14.

    Sills EO, Herrera D, Kirkpatrick AJ, Brandao A, Dickson R, Hall S, Pattanayak S, Shoch D, Vedoveto M, Young L, Pfaff A. Estimating the impact of a local policy innovation: The synthetic control method applied to tropica desforestation. PLoS One. 2015;10(7):e0132590.Google Scholar

  • 15.

    Smith B. The resource curse exorcised: Evidence from a panel of countries. J Dev Econ. 2015;116:57–73.CrossrefGoogle Scholar

  • 16.

    Jinjarak Y, Noy I, Zheng H. Capital controls in Brazil — Stemming a tide with a signal? J Bank Finance. 2013;37:2938–52.CrossrefGoogle Scholar

  • 17.

    Sanso-Navarro M. The effects on American foreign direct investment in the United Kingdom from not adopting the euro. J Common Mark Stud. 2011;49(2):463–83.CrossrefGoogle Scholar

  • 18.

    Belot M, Vandenberghe V. Evaluating the threat effects of grade repetition: Exploiting the 2001 reform by the French-speaking community of Belgium. Educ Econ. 2014;22(1):73–89.CrossrefGoogle Scholar

  • 19.

    Chan HF, Frey BS, Gallus J, Torgler B. Academic honors and performance. Labour Econ. 2014;31:188–204.CrossrefGoogle Scholar

  • 20.

    Hinrichs P. The effects of affirmative action bans on college enrollment, educational attainment, and the demographic composition of universities. Rev Econ Stat. 2012;94(3):712–22.CrossrefGoogle Scholar

  • 21.

    Bauhoff S. The effect of school nutrition policies on dietary intake and overweight: A synthetic control approach. Econ Human Biol. 2014;45–55.Google Scholar

  • 22.

    Kreif N, Grieve R, Hangartner D, Turner AJ, Nikolova S, Sutton M. Examination of the synthetic control method for evaluating health policies with multiple treated units. Health Econ. 2016;25(12):1514–28.CrossrefGoogle Scholar

  • 23.

    Billmeier A, Nannicini T. Assessing economic liberalization episodes: A synthetic control approach. Rev Econ Stat. 2013;95(3):983–1001.CrossrefGoogle Scholar

  • 24.

    Gathani S, Santini M, Stoelinga D. Innovative techniques to evaluate the impacts of private sector developments reforms: An application to Rwanda and 11 other countries. Working paper. 2013. Available at https://blogs.worldbank.org/impactevaluations/files/impactevaluations/methods_for_impact_evaluations_feb06-final.pdf.Google Scholar

  • 25.

    Hosny AS. Algeria’s trade with GAFTA countries: A synthetic control approach. Transit Stud Rev. 2012;19:35–42.CrossrefGoogle Scholar

  • 26.

    Billmeier A, Nannicini T. Trade openness and growth: Pursuing empirical glasnost. IMF Staff Pap. 2009;56(3):447–75.CrossrefGoogle Scholar

  • 27.

    Carrasco V, de Mello JMP, Duarte I. A Década Perdida: 2003–2012. Texto para Discussão. 2014. Available at http://www.econ.puc-rio.br/uploads/adm/trabalhos/files/td626.pdf.Google Scholar

  • 28.

    Dhungana S. Identifying and evaluating large scale policy interventions: What questions can we answer? 2011. Available at https://openknowledge.worldbank.org/bitstream/handle/10986/3688/WPS5918.pdf?sequence=1.Google Scholar

  • 29.

    Jales H, Ribeiro F, Stein G, Kang T. Measuring the role of the 1959 revolution on Cuba economic performance. 2013. Available at https://drive.google.com/file/d/0B-Z_Rf2gRVJzRXozekhMNGpUVUU/view.Google Scholar

  • 30.

    Bohn S, Lofstrom M, Raphael S. Did the 2007 legal Arizona workers act reduce the state’s unauthorized immigrant population? Rev Econ Stat. 2014;96(2):258–69.CrossrefGoogle Scholar

  • 31.

    Calderon G. The effects of child care provision in Mexico. Working paper. 2014. Available at http://goo.gl/YSEs9B.Google Scholar

  • 32.

    Kleven HJ, Landais C, Saez E. Taxation and international migration of superstars: Evidence from European football market. Am Econ Rev. 2013;103(5):1892–924.CrossrefGoogle Scholar

  • 33.

    de Souza FFA. Tax evasion and inflation: Evidence from the nota fiscal paulista program. Master’s thesis. Pontifícia Universidade Católica. 2014. Available at http://www.dbd.puc-rio.br/pergamum/tesesabertas/1212327_2014_completo.pdf.Google Scholar

  • 34.

    Pinotti P. The economic costs of organized crime: Evidence from Southern Italy. Econ J. 2015;125:203–32.CrossrefGoogle Scholar

  • 35.

    Pinotti P. Lessons from the economics of crime: What reduces offending? The MIT Press. Chapter: Organized crime, violence and the quality of politicians: Evidence from Southern Italy. 2013. Available at http://dx.doi.org/10.2139/ssrn.2144121.Google Scholar

  • 36.

    Saunders J, Lundberg R, Braga AA, Ridgeway G, Miles J. A synthetic control approach to evaluating place-based crime interventions. J Quant Criminol. 2015;31(3):413–34.CrossrefGoogle Scholar

  • 37.

    Acemoglu D, Johnson S, Kermani A, Kwak J, Mitton T. The value of connections in turbulent times: Evidence from the United States. J Financ Econ. 2013;121(2):368–91.Google Scholar

  • 38.

    Ando M. Dreams of urbanization: Quantitative case studies on the local impacts of nuclear power facilities using the synthetic control method. J Urban Econ. 2015;85:68–85.CrossrefGoogle Scholar

  • 39.

    Gobillon L, Magnac T. Regional policy evaluation: Interative fixed effects and synthetic controls. Rev Econ Stat. 2016;98(3):535–51.CrossrefGoogle Scholar

  • 40.

    Kirkpatrick AJ, Bennear LS. Promoting clean enery investment: An empirical analysis of property assessed clean energy. J Environ Econ Manag. 2014;68:357–75.CrossrefGoogle Scholar

  • 41.

    Liu S. Spillovers from universities: Evidence from the land-grant program. J Urban Econ. 2015;87:25–41.CrossrefGoogle Scholar

  • 42.

    Possebom V. Free trade zone of manaus: An impact evaluation using the sythetic control method. Rev Brasil Econ. 2017;71(2):217–31.Google Scholar

  • 43.

    Severnini ER. The power of hydroelectric dams: Agglomeration spillovers. IZA discussion paper, No 8082. Available at http://ftp.iza.org/dp8082.pdf.Google Scholar

  • 44.

    Athey S, Imbens GW. The state of applied econometrics: Causality and policy evaluation. J Econ Perspect. 2017;31(2):3–32.CrossrefGoogle Scholar

  • 45.

    Fisher RA. The design of experiments. 8th ed. Hafner Publishing Company, United States; 1971.Google Scholar

  • 46.

    Imbens GW, Rubin DB. Causal inference for statistics, social and biomedical sciences: An introduction, 1st edn. United Kingdom: Cambridge University Press; 2015.Google Scholar

  • 47.

    Rosenbaum PR. Observational studies. 2nd ed. New York: Springler Science + Business Media; 2002.Google Scholar

  • 48.

    Cattaneo M, Titiunik R, Vazquez-Bare G. Inference in regression discontinuity designs under local randomization. Stata J. 2016;16(2):331–67.Google Scholar

  • 49.

    Conley TG, Taber CR. Inference with difference-in-differences with a small number of policy changes. Rev Econ Stat. 2011;93(1):113–25.CrossrefGoogle Scholar

  • 50.

    Anderson ML. Multiple inference and gender differences in the effects of early intervention: A reevaluation of the Abecedarian, Perry preschool and early training projects. J Am Stat Assoc. 2008;103(484):1481–95.CrossrefGoogle Scholar

  • 51.

    Kaul A, Klöbner S, Pfeifer G, Schieler M. Synthetic control methods: Never use all pre-intervention outcomes as economic predictors. Working paper. 2015. Available at http://www.oekonometrie.uni-saarland.de/papers/SCM_Predictors.pdf.Google Scholar

  • 52.

    Ando M, Sävje F. Hypothesis testing with the synthetic control method. 2013. Working Paper. Available at http://www.eea-esem.com/files/papers/eea-esem/2013/2549/scm.pdf.Google Scholar

  • 53.

    Wong L. Three essays in causal inference. PhD thesis. Stanford University; 2015.Google Scholar

  • 54.

    Dube A, Zipperer B. Pooling multiple case studies using synthetic controls: An application to minimum wage policies. Working paper. 2015. Available at http://ftp.iza.org/dp8944.pdf.Google Scholar

  • 55.

    Carvalho CV, Mansini R, Medeiros MC. ArCo: An artificial counterfactual approach for aggregate data. Working paper. 2017. Available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2823687.Google Scholar

  • 56.

    Hahn J, Shi R. Synthetic control and inference. Econometrics. 2017;5(4):52.CrossrefGoogle Scholar

  • 57.

    Ferman B, Pinto C. Revisiting the synthetic control estimator. 2017. Available at https://dl.dropboxusercontent.com/u/12654869/Ferman%20and%20Pinto%20-%20revisiting%20the%20SC.pdf.Google Scholar

  • 58.

    Ferman B, Pinto C. Placebo tests for synthetic controls. 2017. Available at https://dl.dropboxusercontent.com/u/12654869/Ferman%20and%20Pinto%20-%20placebo%20tests%20for%20SC.pdf.Google Scholar

  • 59.

    Ferman B, Pinto C, Possebom V. Cherry picking with synthetic controls. 2017. Available at https://dl.dropboxusercontent.com/u/12654869/FPP%20-%20Cherry%20Picking.pdf.Google Scholar

  • 60.

    Rosenbaum PR. Sensitivity analysis for certain permutation inferences in matched observational studies. Biometrika. 1987;74(1):13–26.CrossrefGoogle Scholar

  • 61.

    Rosenbaum PR. Sensitivity analysis for matching with multple controls. Biometrika. 1988;75(3):577–81.CrossrefGoogle Scholar

  • 62.

    Rosenbaum PR, Krieger AM. Sensitivity of two-sample permutation inferences in observational studies. J Am Stat Assoc. 1990;85(410):493–8.CrossrefGoogle Scholar

  • 63.

    Rosenbaum PR. Sensitivity analysis for m-estimates, tests, and confidence intervals in matched observational studies. Biometrics. 2007;63:456–64.CrossrefGoogle Scholar

  • 64.

    Rosenbaum PR, Silber JH. Amplification of sensitivity analysis in matched observational studies. J Am Stat Assoc. 2009;104(488):1398–405.CrossrefGoogle Scholar

  • 65.

    Yates F. Tests of significance for 2 x 2 contingency tables. J R Stat Soc A. 1984;147(3):426–63.CrossrefGoogle Scholar

  • 66.

    Lehmann E. Testing statistical hypotheses. New York: John Wiley & Sons; 1959.Google Scholar

  • 67.

    Ibragimov R, Muller UK. T-statistic based correlation and heterogeneity robust inference. J Bus Econ Stat. 2010;28(4):453–68.CrossrefGoogle Scholar

  • 68.

    Young A. Channeling Fisher: Randomization tests and the statistical insignificance of seemingly significant eperimental results. 2016. Available at http://economics.mit.edu/files/11362.Google Scholar

  • 69.

    Eudey TL, Kerr J, Trumbo B. Using R to simulate permutation distributions for some elementary experimental designs. J Stat Educ. 2010;18(1).Google Scholar

  • 70.

    List J, Shaikh AM, Xu Y. Multiple hypothesis testing in experimental economics. NBER working paper 21875. 2016. Available at http://www.nber.org/papers/w21875.Google Scholar

About the article

Received: 2016-11-15

Revised: 2018-08-06

Accepted: 2018-08-11

Published Online: 2018-09-15

Published in Print: 2018-09-25


Funding Source: Fundação de Amparo à Pesquisa do Estado de São Paulo

Award identifier / Grant number: 2014/23731-3

We are grateful to FAPESP that provided financial aid through grant number 2014/23731-3.


Citation Information: Journal of Causal Inference, Volume 6, Issue 2, 20160026, ISSN (Online) 2193-3685, DOI: https://doi.org/10.1515/jci-2016-0026.

Export Citation

© 2018 Walter de Gruyter GmbH, Berlin/Boston.Get Permission

Supplementary Article Materials

Comments (0)

Please log in or register to comment.
Log in