Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Journal of Official Statistics

The Journal of Statistics Sweden

4 Issues per year

IMPACT FACTOR 2016: 0.411
5-year IMPACT FACTOR: 0.776

CiteScore 2016: 0.63

SCImago Journal Rank (SJR) 2016: 0.710
Source Normalized Impact per Paper (SNIP) 2016: 0.975

Open Access
See all formats and pricing
More options …

Face-to-Face or Sequential Mixed-Mode Surveys Among Non-Western Minorities in the Netherlands: The Effect of Different Survey Designs on the Possibility of Nonresponse Bias

Johannes W.S. Kappelhof
Published Online: 2015-03-01 | DOI: https://doi.org/10.1515/jos-2015-0001


This article compares the quality of response samples based on a single mode CAPI survey design with the quality of response samples based on a sequential mixed-mode (CAWI-CATICAPI) survey design among four non-Western minority ethnic groups in the Netherlands. The quality is assessed with respect to the representativity of the response samples and the estimated potential for nonresponse bias in survey estimates based on auxiliary variables and the response rate. This article also investigates if these designs systematically enhance response rates differently among various sociodemographic subgroups based on auxiliary variables. Also, costs and cost-related issues particular to this sequential mixed-mode design are discussed. The results show that sequential mixed mode surveys among non-Western ethnic minorities in the Netherlands lead to less representative response samples and show more potential for nonresponse bias in survey estimates. Furthermore, the designs lead to systematic differences in response rates among various sociodemographic subgroups, such as older age groups. Both designs also cause some of the same sociodemographic subgroups to be systematically underrepresented among all non-Western ethnic minority groups. Finally, the results show that in this instance the cost savings did not outweigh the reduction in quality.

Keywords: Survey design; sequential mixed-mode survey; nonresponse bias; non-western ethnic minorities; representativeness


  • AAPOR 2011. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. (7th edition). The American Association for Public Opinion Research. Available at: http://www.aapor.org/AM/Template.cfm?Section=Standard _Definitions2& Template=/CM/ContentDisplay.cfm& ContentID=3156 (accessed December 2013).Google Scholar

  • Andridge, R.R. and R.J. Little. 2011. “Proxy Pattern-Mixture Analysis for Survey Nonresponse.” Journal of Official Statistics 27: 153-180.Google Scholar

  • Bethlehem, J., F. Cobben, and B. Schouten. 2011. Handbook of Nonresponse in Household Surveys. Hoboken, New Jersey: John Wiley & Sons, Inc.Google Scholar

  • Biemer, P.P. and L.E. Lyberg. 2003. Introduction to Survey Quality. Hoboken, New Jersey: John Wiley & Sons, Inc.Google Scholar

  • Bijl, R. and A. Verweij. 2012. Measuring and Monitoring Immigrant Integration in Europe: Integration Policies and Monitoring Efforts in 17 European Countries, 2012-8.Google Scholar

  • Den Haag: SCP. Available at: http://www.scp.nl/english/Publications/ Publications_by_year/Publications_2012/Measuring_and_monitoring_immigrant_ integration_in_Europe (accessed June 2013).Google Scholar

  • CBS-Statline. Available at: http://statline.cbs.nl/StatWeb/publication/?VW= T& DM=SLNL& PA=37325& D1=a& D2=0& D3=0& D4=0& D5=0-4,137,152,220,237 & D6=0,4,9, (l-1),l& HD=130605-0936& HDR=G2,G1,G3,T& STB=G4,G5 (accessed December 2013).Google Scholar

  • Couper, P. 2005. “Technology Trends in Survey Data Collection.” Social Science Computer Review 23: 486-501. DOI: http://dx.doi.org/10.1177/0894439305278972.CrossrefGoogle Scholar

  • Dagevos, J. and R. Schellingerhout. 2003. “Sociaal-culturele integratie. Contacten, cultuur en orientatie op de eigen groep.” In Rapportage minderheden, edited by J. Dagevos, M. Gijsberts, and v. C. Praag, 317-362. [In Dutch: Socio-Cultural integration. Contacts, culture and focus on the own ethnic group]. Den Haag: SCP. Available at: http://www. scp.nl/Publicaties/Alle_publicaties/Publicaties_2003/Rapportage_minderheden_2003 (accessed December 2012).Google Scholar

  • De Leeuw, E.D. 2005. “To Mix or not to Mix Data Collection Modes in Surveys.” Journal of Official Statistics 21: 233-255.Google Scholar

  • De Leeuw, E.D., D.A. Dillman, and J.J. Hox. 2008. “Mixed-Mode Surveys: When and Why.” In International Handbook Of Survey Methodology, edited by E.D. de Leeuw, J.J. Hox, and D.A. Dillman, 299-316. New York: Taylor and Francis Group.Google Scholar

  • Dempster, A.P., N.M. Laird, and D.B. Rubin. 1977. “Maximum Likelihood from Incomplete Data via the EMAlgorithm.” Journal of the Royal Statistical Society. SeriesB (Methodological) 39: 1-38.Google Scholar

  • Dillman, D.A. 2007. Mail and Internet Surveys: The Tailored Design Method (2nd ed.). New York: John Wiley & Sons Inc.Google Scholar

  • Dillman, D.A. and L.M. Christian. 2005. “Survey Mode as a Source of Instability in Responses Across Surveys.” Field Methods 17: 30-52. DOI: http://dx.doi.org/10.1177/ 1525822X04269550.CrossrefGoogle Scholar

  • Feskens, R.C.W. 2009. Difficult Groups in Survey Research and the Development of Tailor-made Approach Strategies. Den Haag/Utrecht: Statistics Netherlands and Kappelhof: Face-to-Face or Sequential Mixed-Mode 27 Utrecht University. Available at: http://www.cbs.nl/NR/rdonlyres/8F317AA9-1074- 4BF7-84EB-015D013DBB80/0/2009x11feskenspub.pdf (accessed October 2011).Google Scholar

  • Feskens, R.C.W., J. Kappelhof, J. Dagevos, and I.A.L Stoop. 2010. Minderheden in de mixed-mode? Een inventarisatie van voor- en nadelen van het inzetten van verschillende dataverzamelingsmethoden onder niet-westerse migranten. SCP-special 57. [In Dutch: Ethnic minorities in the mixed mode? An inventory of the advantages and disadvantages of employing different data collection methods among non-Western migrant] Den Haag: SCP. Available at: http://www.scp.nl/Publicaties/Alle_publicaties/ Publicaties_2010/Minderheden_in_de_mixed_mode (accessed December 2014).Google Scholar

  • Gijsberts, M. and J. Iedema. 2011. “Opleidingsniveau van niet-schoolgaanden en leerprestaties in het basisonderwijs.” In Jaarrapport Integratie 2011, edited by M. Gijsberts, W. Huijnk, and J. Dagevos, 76-99. [In Dutch: Education level of persons who do not attend school and educational attainment in primary school]. Den Haag: SCP. (2012-3). Available at: http://www.scp.nl/Publicaties/Alle_publicaties/ Publicaties_2012/Jaarrapport_integratie_2011 (accessed December 2014).Google Scholar

  • Graham, J.W., A.E. Olchowski, and T.D. Gilreath. 2007. “How Many Imputations are Really Needed? Some Practical Clarifications of Multiple Imputation Theory.” Prevention Science 8: 206-213. DOI: http://dx.doi.org/10.1007/s11121-007-0070-9.CrossrefGoogle Scholar

  • Groves, R.M. 1989. Survey Costs and Survey Errors. New York: Wiley.Google Scholar

  • Groves, R.M. and M.P. Couper. 1998. Nonresponse in Household Interview Surveys. New York: Wiley. Groves, R.M. and E. Peytcheva. 2008. “The Impact of Nonresponse Rates on Nonresponse Bias.” Public Opinion Quarterly 72: 167-189. DOI: http://dx.doi.org/10.1093/poq/ nfn011.CrossrefGoogle Scholar

  • Kappelhof, J.W.S. 2010. Op maat gemaakt? Een evaluatie van enkele respons verbeterende maatregelen onder Nederlanders van niet-westerse afkomst. [In Dutch: An evaluation of several response enhancing measures among Dutch of non-Western origin]. Den Haag: SCP. (SCP-special 53). Available at: http://www.scp.nl/Publicaties/ Alle_publicaties/Publicaties_2010/Op_maat_gemaakt (accessed December 2014).Google Scholar

  • Kappelhof, J.W.S. 2014. “The Effect of Different Survey Designs on Nonresponse in Surveys Among Non-Western Minorities in The Netherlands.” Survey Research Methods 8: 81-98. Available at: http://www.surveymethods.org (accessed December 2014).Google Scholar

  • Kemper, F. 1998. “Gezocht: Marokkanen. Methodische problemen bij het verwerven en interviewen van allochtone respondenten.” [In Dutch: Wanted: Moroccans. Methodological problems with obtaining response and interviewing respondents of foreign origin]. Migrantenstudies 1: 43-57.Google Scholar

  • Korte, K. and J. Dagevos. 2011. Survey Integratie Minderheden 2011. Verantwoording van de opzet en uitvoering van een survey onder Turkse, Marokkaanse, Surinaamse en Antilliaanse Nederlanders en een autochtone vergelijkingsgroep. [In Dutch: Survey on the Integration of Minorities 2011. Report on the design and implementation of the survey among Turkish, Moroccan, Surinamese, Antillean Dutch and a mainstream Dutch reference group]. Den Haag: SCP.Google Scholar

  • Kreuter, F. 2013. Improving Surveys with Paradata. Analytic Uses of Process Information. Hoboken, New Jersey: John Wiley & Sons, Inc. 28 Journal of Official Statistics Little, R.J. and D.B. Rubin. 2002. Statistical Analysis With Missing Data. New York: John Wiley & Sons.Google Scholar

  • Maitland, A., C. Casas-Cordero, and F. Kreuter. 2009. “An Evaluation of Nonresponse Bias Using Paradata from a Health Survey.” In: Proceedings of the Section on Government Statistics: American Statistical Association, Joint Statistical Meetings, 2009. 370-378. Alexandria, VA: American Statistical Association. Available at: http:// www.amstat.org/sections/SRMS/proceedings/y2009/Files/303004.pdf (accessed December 2014).Google Scholar

  • Moorman, P.G., B. Newman, R.C. Millikan, C.K.J. Tse, and D.P. Sandler. 1999. “Participation rates in a Case-control Study: The Impact of Age, Race, and Race of Interviewer.” Annals of epidemiology 9: 188-195. DOI: http://dx.doi.org/10.1016/ S1047-2797(98)00057-X.CrossrefGoogle Scholar

  • Reep, C. 2003. Moeilijk Waarneembare Groepen. Een inventarisatie. [In Dutch: Hard to survey populations: An inventory]. Voorburg/Heerlen: CBS. (H1568-03-s00).Google Scholar

  • Rubin, D.B. 1987. Multiple Imputation for Nonresponse in Surveys. Hoboken, New Jersey: John Wiley & Sons, Inc.Google Scholar

  • Schmeets, H. 2005. “De leefsituatie van allochtonen.” In Enquêteonderzoek onder allochtonen. Problemen en oplossingen, edited by H. Schmeets en and R. van der Bie, 169-176. [In Dutch: The living conditions of Dutch of foreign origin]. Voorburg/Heerlen: CBS.Google Scholar

  • Schothorst, Y. 2002. “Onderzoek onder allochtonen: wat mag, wat moet en wat kan?” In: Interviewen in de multiculturele samenleving. Problemen en oplossingen, edited by H. Houtkoop en and J. Veenman, 101-116. [In Dutch: Survey research among Dutch of foreign origin: What may be done, what has to be done and what is possible?]. Assen: Koninklijke Van Gorcum.Google Scholar

  • Schouten, B. and F. Cobben. 2007. R-Indexes for the Comparison of Different Fieldwork Strategies and Data Collection Modes. Voorburg/Heerlen: CBS. (Discussion Paper 07002). Available at: http://www.risq-project.eu/papers/schouten-cobben-2007-a.pdf (accessed December 2013).Google Scholar

  • Schouten, B.F. and F. Cobben. 2008. An Empirical Validation of R-Indicators. Voorburg/Heerlen: CBS. (Discussion Paper 08006) Available at: http://www. risq-project.eu/papers/cobben-schouten-2008-a.pdf (accessed December 2013).Google Scholar

  • Schouten,B. F. Cobben, and J. Bethlehem. 2009. “Indicators of Representativeness of Survey Nonresponse.” Survey Methodology 35: 101-113. Available at: http://www.risq-project. eu/papers/schouten-cobben-bethlehem-2009.pdf (accessed December 2013).Google Scholar

  • Schouten, B. J. Bethlehem, K. Beullens, Ø. Kleven, G. Loosveldt, A. Luiten, K. Rutar, N. Shlomo, and C. Skinner. 2012. “Evaluating, Comparing, Monitoring, and Improving Representativity of Survey Response Through R-Indicators and Partial R-Indicators.” International Statistical Review 80: 382-399. Available at: http://onlinelibrary.wiley. com/doi/10.1111/j.1751-5823.2012.00189.x/abstract (accessed December 2013).CrossrefWeb of ScienceGoogle Scholar

  • Schouten, B., N. Shlomo, and C. Skinner. 2011. “Indicators for Monitoring and Improving Representativity of Response.” Journal of Official Statistics 27: 231-253.Google Scholar

  • Shlomo, N., C. Skinner, B. Schouten, T. Carolina, and M. Morren. 2009. Partial Indicators for Representative Response. Rep. No. RISQ deliverable 4. version 2. Available at: http:// www.risq-project.eu/papers/RISQ-Deliverable-4-V2.pdf (accessed December 2013). Kappelhof: Face-to-Face or Sequential Mixed-Mode 29 Singer, E. 2002. “The Use of Incentives to Reduce Nonresponse in Household Surveys.” In Survey Nonresponse, edited by R.M. Groves, D. Dillman, J.L. Eltinge, and R.J.A. Little, 163-177. New York: John Wiley & Sons.Google Scholar

  • Singer, E., J. Van Hoewyk, N. Gebler, T. Raghunathan, and K. McGonagle. 1999. “The Effect of Incentives on Response Rates in Interviewer-Mediated Surveys.” Journal of Official Statistics 15: 217-230.Google Scholar

  • Singer, E., J. Van Hoewyk, and M.P. Maher. 2000. “Experiments With Incentives in Telephone Surveys.” Public Opinion Quarterly 64: 171-188. DOI: http://dx.doi.org/ 10.1086/317761.CrossrefGoogle Scholar

  • Smulders, M. 2011. Onderzoeksverantwoording Survey Integratie Minderheden 2011 [In Dutch: research description on the Survey on the Integration of Minorities 2011]. Dongen: GFK Panel Services Benelux B.V. Stoop, I.A.L. 2005. The Hunt for the Last Respondent: Nonresponse in Sample Surveys. Den Haag: SCP.Google Scholar

  • Särndal, C.-E. 2011. “Three Factors to Signal Non Response Bias With Applications to Categorical Auxiliary Variables.” International Statistical Review 79: 233-254. DOI: http://dx.doi.org/10.1111/j.1751-5823.2011.00142.x.Web of ScienceCrossrefGoogle Scholar

  • Särndal, C.-E., and S. Lundström. 2010. “Design for Estimation: Identifying Auxiliary Vectors to Reduce Nonresponse Bias.” Survey Methodology 36: 131-144.Google Scholar

  • Van Ingen, E., J. De Haan, and M. Duimel. 2007. Achterstand en afstand. Digitale vaardigheden van lager opgeleiden, ouderen, allochtonen en inactieven. [In Dutch: Lagging behind. Digital skills of lower educated, elderly, foreign origin and inactive persons]. Den Haag: SCP. (SCP- 2007/24).Google Scholar

  • Veenman, J. 2002. “Interviewen in multicultureel Nederland.” In Interviewen in de multiculturele samenleving. Problemen en oplossingen, edited by H. Houtkoop en and J.Google Scholar

  • Veenman, 1-19. [In Dutch: Interviewing in the multi-cultural Netherlands]. Assen: Koninklijke Van Gorcum.Google Scholar

  • Voogt, R.J.J. and W.E. Saris. 2005. “Mixed Mode Designs: Finding the Balance Between Nonresponse Bias and Mode Effects.” Journal of Official Statistics 21: 367-387.Google Scholar

  • Wagner, J. 2008. Adaptive Survey Design to Reduce Nonresponse Bias, Michigan: University of Michigan. Available at: http://deepblue.lib.umich.edu/bitstream/2027.42/ 60831/1/jameswag_1.pdf (accessed December 2014).Google Scholar

  • Wagner, J. 2010. “The Fraction of Missing Information as a Tool for Monitoring the Quality of Survey Data.” Public Opinion Quarterly 74: 223-243. DOI: http://dx.doi.org/10.1093/poq/nfq007.CrossrefWeb of ScienceGoogle Scholar

About the article

Received: 2013-03-01

Revised: 2014-12-01

Accepted: 2014-12-01

Published Online: 2015-03-01

Citation Information: Journal of Official Statistics, Volume 31, Issue 1, Pages 1–30, ISSN (Online) 2001-7367, DOI: https://doi.org/10.1515/jos-2015-0001.

Export Citation

© 2015 Johannes W.S. Kappelhof. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License. BY-NC-ND 3.0

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

Johannes W. S. Kappelhof and Edith D. De Leeuw
Sociological Methods & Research, 2017, Page 004912411770147
Min-Jeong Park and Hang J. Kim
Korean Journal of Applied Statistics, 2016, Volume 29, Number 6, Page 1041

Comments (0)

Please log in or register to comment.
Log in