Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Journal of Official Statistics

The Journal of Statistics Sweden

4 Issues per year

IMPACT FACTOR 2016: 0.411
5-year IMPACT FACTOR: 0.776

CiteScore 2016: 0.63

SCImago Journal Rank (SJR) 2016: 0.710
Source Normalized Impact per Paper (SNIP) 2016: 0.975

Open Access
See all formats and pricing
More options …

Comparing Two Inferential Approaches to Handling Measurement Error in Mixed-Mode Surveys

Bart Buelens / Jan A. Van den Brakel
Published Online: 2017-06-12 | DOI: https://doi.org/10.1515/jos-2017-0024


Nowadays sample survey data collection strategies combine web, telephone, face-to-face, or other modes of interviewing in a sequential fashion. Measurement bias of survey estimates of means and totals are composed of different mode-dependent measurement errors as each data collection mode has its own associated measurement error. This article contains an appraisal of two recently proposed methods of inference in this setting. The first is a calibration adjustment to the survey weights so as to balance the survey response to a prespecified distribution of the respondents over the modes. The second is a prediction method that seeks to correct measurements towards a benchmark mode. The two methods are motivated differently but at the same time coincide in some circumstances and agree in terms of required assumptions. The methods are applied to the Labour Force Survey in the Netherlands and are found to provide almost identical estimates of the number of unemployed. Each method has its own specific merits. Both can be applied easily in practice as they do not require additional data collection beyond the regular sequential mixed-mode survey, an attractive element for national statistical institutes and other survey organisations.

Keywords: Generalized regression; mode effects; selection bias; response mode calibration; counterfactuals

6. References

  • Buelens, B. and J.A. Van den Brakel. 2015. “Measurement Error Calibration in Mixed-Mode Sample Surveys.” Sociological Methods & Research 44(3): 391–426. Doi: http://dx.doi.org/10.1177/0049124114532444.Crossref

  • Cannel, C., P. Miller, and L. Oksenberg. 1981. “Research on Interviewing Techniques.” In Sociological Methodology, edited by S. Leinhardt. 389–437. San Fransisco: Jossey-Bass.Google Scholar

  • Centraal Bureau voor de Statistiek. 2015. Methoden en Definties Enquête Beroepsbevolking 2014. Technical report, Statistics Nederlands, Heerlen. Available at: https://www.cbs.nl/NR/rdonlyres/1BB3C645-47CC-4F58-9031-89F490AEE981/0/methodenendefinitiesebb2014.pdf (accessed March 2017).

  • Cernat, A. 2015. “Impact of Mode Design on Measurement Errors and Estimates of Individual Change.” Survey Research Methods 9(2): 83–99. Doi: http://dx.doi.org/10.18148/srm/2015.v9i2.5851.Crossref

  • De Leeuw, E. 2005. “To Mix or not to Mix data Collection Modes in Surveys.” Journal of Official Statistics 21: 233–255.Google Scholar

  • Dillman, D., G. Phelps, R. Tortora, K. Swift, J. Kohrell, J. Berck, and B. Messer. 2009. “Response Rate and Measurement Differences in Mixed-Mode Surveys Using Mail, Telephone, Interactive Voice Response and the Internet.” Social Science Research 39: 1–18. Doi: http://dx.doi.org/10.1016/j.ssresearch.2008.03.007.Crossref

  • Groves R.M. and L. Lyberg. 2010. “Total Survey Error: Past, Present, and Future.” Public Opinion Quarterly 74(5): 849–879. Doi: http://dx.doi.org/10.1093/poq/nfq065.Crossref

  • Holbrook, A., M. Green, and J. Krosnick. 2003. “Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires.” Public Opinion Quarterly 67: 79–125. Doi: http://dx.doi.org/10.1086/346010.Crossref

  • Jäckle, A., C. Roberts, and P. Lynn. 2010. “Assessing the Effect of Data Collection Mode on Measurement.” International Statistical Review 78: 3–20. Doi: http://dx.doi.org/10.1111/j.1751-5823.2010.00102.x.Crossref

  • Klausch, T., J. Hox, and B. Schouten. 2015. “Selection Error in Single- and Mixed Mode Surveys of the Dutch General Population.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 178(4): 945–961. Doi: http://dx.doi.org/10.1111/rssa.12102.Crossref

  • Krosnick, J. 1991. “Response strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology 5: 213–236. Doi: http://dx.doi.org/10.1002/acp.2350050305.Crossref

  • Krosnick, J. and D. Alwin. 1987. “An Evaluation of a Cognitive Theory of Response-Order Effects in Survey Measurement.” Public Opinion Quarterly 51: 201–219. Doi: http://dx.doi.org/10.1086/269029.Crossref

  • Lynn, P. 2013. “Alternative Sequential Mixed-Mode Designs: Effects on Attrition Rates, Attrition Bias, and Costs.” Journal of Survey Statistics and Methodology 1(2): 183–205. Doi: http://dx.doi.org/10.1093/jssam/smt015.Crossref

  • Särndal, C.-E., B. Swensson, and J. Wretman. 1992. Model Assisted Survey Sampling. New York: Springer-Verlag.Google Scholar

  • Schouten, B., J. van den Brakel, B. Buelens, J. van der Laan, and T. Klausch. 2013. “Disentangling Mode-Specific Selection and Measurement Bias in Social Surveys.” Social Science Research 42(6): 1555–1570. Doi: http://dx.doi.org/10.1016/j.ssresearch.2013.07.005.Crossref

  • Suzer-Gurtekin, Z.T. 2013. Investigating the Bias Properties of Alternative Statistical Inference Methods in Mixed-Mode Surveys. Ph.D. thesis, University of Michigan. Available at: https://deepblue.lib.umich.edu/bitstream/handle/2027.42/102471/tsuzer_1.pdf (accessed March 2017).

  • Suzer-Gurtekin, Z.T., S. Heeringa, and R. Vaillant. 2012. “Investigating the Bias of Alternative Statistical Inference Methods in Sequential Mixed-Mode Surveys.” In Proceedings of the JSM, Section on Survey Research Methods, San Diego, July 28–August 2, 2012. American Statistical Association, 4711–25.Google Scholar

  • Tourangeau, R., L. Rips, and K. Rasinski. 2000. The Psychology of Survey Response. Cambridge: Cambridge University Press.Google Scholar

  • Van den Brakel, J. 2008. “Design-Based Analysis of Embedded Experiments with Applications in the Dutch Labour Force Survey.” Journal of the Royal Statistical Society, Series A 171: 581–613. Doi: http://dx.doi.org/10.1111/j.1467-985X.2008.00532.x.Crossref

  • Van den Brakel, J.A. and S. Krieg. 2015. “Dealing with Small Sample Sizes, Rotation Group Bias and Discontinuities in a Rotating Panel Design.” Survey Methodology 41(2): 267–296. Available at: http://www.statcan.gc.ca/pub/12-001-x/2015002/article/14231-eng.pdf (accessed March 2017).

  • Vannieuwenhuyze, J.T.A. and G. Loosveldt. 2013. “Evaluating Relative Mode Effects in Mixed-Mode Surveys: Three Methods to Disentangle Selection and Measurement Effects.” Sociological Methods & Research 42(1): 82–104. Doi: http://dx.doi.org/10.1177/0049124112464868.Crossref

  • Voogt, R. and W. Saris. 2005. “Mixed Mode Designs: Finding the Balance Between Nonresponse Bias and Mode Effects.” Journal of Official Statistics 21: 367–387.Google Scholar

About the article

Received: 2016-01-01

Revised: 2017-02-01

Accepted: 2017-03-01

Published Online: 2017-06-12

Published in Print: 2017-06-01

Citation Information: Journal of Official Statistics, Volume 33, Issue 2, Pages 513–531, ISSN (Online) 2001-7367, DOI: https://doi.org/10.1515/jos-2017-0024.

Export Citation

© 2017 Bart Buelens et al., published by De Gruyter Open. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License. BY-NC-ND 3.0

Comments (0)

Please log in or register to comment.
Log in