Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Journal of Official Statistics

The Journal of Statistics Sweden

4 Issues per year


IMPACT FACTOR 2016: 0.411
5-year IMPACT FACTOR: 0.776

CiteScore 2016: 0.63

SCImago Journal Rank (SJR) 2016: 0.710
Source Normalized Impact per Paper (SNIP) 2016: 0.975

Open Access
Online
ISSN
2001-7367
See all formats and pricing
More options …

Unit Nonresponse and Weighting Adjustments: A Critical Review

J. Michael Brick
Published Online: 2013-10-03 | DOI: https://doi.org/10.2478/jos-2013-0026

Abstract

This article reviews unit nonresponse in cross-sectional household surveys, the consequences of the nonresponse on the bias of the estimates, and methods of adjusting for it. We describe the development of models for nonresponse bias and their utility, with particular emphasis on the role of response propensity modeling and its assumptions. The article explores the close connection between data collection protocols, estimation strategies, and the resulting nonresponse bias in the estimates. We conclude with some comments on the current state of the art and the need for future developments that expand our understanding of the response phenomenon.

Keywords: Response propensity; bias; data collection; calibration

  • Andridge, R.H. and Little, R.J. (2011). Proxy Pattern-Mixture Analysis for Survey Nonresponse. Journal of Official Statistics, 27, 153-180.Google Scholar

  • Atrostic, B.K., Bates, N., Burt, G., and Silberstein, A. (2001). Nonresponse in U.S. Government Household Surveys: Consistent Measures, Recent Trends, and New Insights. Journal of Official Statistics, 17, 209-226.Google Scholar

  • Bartholomew, D.J. (1961). A Method of Allowing for ‘Not-at-Home’ Bias in Sample Surveys. Applied Statistics, 10, 52-59.CrossrefGoogle Scholar

  • Bates, N., Dahlhamer, J., and Singer, E. (2008). Privacy Concerns, too Busy, or Just not Interested: Using Doorstep Concerns to Predict Survey Nonresponse. Journal of Official Statistics, 24, 591-612.Google Scholar

  • Beaumont, J.F. (2005). On the Use of Data Collection Process Information for the Treatment of Unit Nonresponse Through Weight Adjustment. Survey Methodology, 31, 227-231.Google Scholar

  • Bethlehem, J.G. (1988). Reduction of Nonresponse Bias Through Regression Estimation. Journal of Official Statistics, 4, 251-260.Google Scholar

  • Bethlehem, J.G. (2002). Weighting Nonresponse Adjustments Based on Auxiliary Information. Survey Nonresponse, R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little (eds). New York: Wiley.Google Scholar

  • Bethlehem, J., Cobben, F., and Schouten, B. (2011). Handbook in Nonresponse in Household Surveys. New York: Wiley.Google Scholar

  • Brehm, J. (1993). The Phantom Respondents: Opinion Surveys and Political Representation. Ann Arbor: University of Michigan Press.Google Scholar

  • Brick, J.M. and Jones, M.E. (2008). Propensity to Respond and Nonresponse Bias. Metron-International Journal of Statistics, LXVI, 51-73.Google Scholar

  • Brick, J.M. and Kalton, G. (1996). Handling Missing Data in Survey Research. Statistical Methods in Medical Research, 5, 215-238.CrossrefGoogle Scholar

  • Brick, J.M. and Montaquila, J.M. (2009). Nonresponse and Weighting. Handbook of Statistics. Sample Surveys: Design, Methods, and Applications, D. Pfeffermann and C.R. Rao (eds). Vol. 29A. Amsterdam: Elsevier-North Holland, 163-186.Google Scholar

  • Brick, J.M., Montaquila, J., Han, D., and Williams, D. (2012). Improving Response Rates for Spanish-Speakers in Two-Phase Mail Surveys. Public Opinion Quarterly, 76, 721-732.CrossrefGoogle Scholar

  • Brick, J.M. and Williams, D. (2013). Explaining Rising Nonresponse Rates in Cross- Sectional Surveys. The ANNALS of the American Academy of Political and Social Science, 645, 36-59.Google Scholar

  • Cassel, C., Särndal, C.-E., and Wretman, J. (1983). Some Uses of Statistical Models in Connection With the Nonresponse Problem. Incomplete Data in Sample Surveys, W.G. Madow and I. Olkin (eds). Vol. 3. New York: Academic Press.Google Scholar

  • Chang, T. and Kott, P.S. (2008). Using Calibration Weighting to Adjust for Nonresponse Under a Plausible Model. Biometrika, 95, 557-571.Google Scholar

  • Cochran, W. (1977). Sampling Techniques, (3rd edition). New York: Wiley.Google Scholar

  • Colley, R.H. (1945). Don’t Look Down Your Nose at Mail Questionnaires. Printers’ Ink, March, 16, 21-108.Google Scholar

  • Curtin, R., Presser, S., and Singer, E. (2000). The Effects of Response Rate Changes on the Index of Consumer Sentiment. Public Opinion Quarterly, 64, 413-428.CrossrefPubMedGoogle Scholar

  • Curtin, R., Presser, S., and Singer, E. (2005). Changes in Telephone Survey Nonresponse Error Over the Past Quarter Century. Public Opinion Quarterly, 69, 87-98.CrossrefGoogle Scholar

  • Da Silva, D.N. and Opsomer, J.D. (2004). Properties of the Weighting Cell Estimator Under a Nonparametric Response Mechanism. Survey Methodology, 30, 45-55.Google Scholar

  • Da Silva, D.N. and Opsomer, J.D. (2009). Nonparametric Propensity Weighting for Survey Nonresponse Through Local Polynomial Regression. Survey Methodology, 35, 165-176. Google Scholar

  • Dalenius, T. (1983). Some Reflections on the Problem of Missing Data. Incomplete Data in Sample Surveys, W.G. Madow and I. Olkin (eds). Vol. 3. New York: Academic Press, 411-413.Google Scholar

  • David, M., Little, R., Samuhel, M., and Triest, R. (1983). Nonrandom Nonresponse Models Based on the Propensity to Respond. Proceedings of the Business and Economic Statistics Section of the American Statistical Association, 168-173.Google Scholar

  • David, M., Little, R.J.A., Samuhel, M., and Triest, R. (1986). Alternative Methods for CPS Income Imputation. Journal of the American Statistical Association, 81, 29-41.CrossrefGoogle Scholar

  • De Leeuw, E. and De Heer, W. (2002). Trends in Household Survey Nonresponse: A Longitudinal and International Comparison. Survey Nonresponse, R.M. Groves, D.A.Google Scholar

  • Dillman, J.L. Eltinge, and R.J.A. Little (eds). New York: Wiley, 41-54.Google Scholar

  • Deming, W. (1953). On a Probability Mechanism to Attain an Economic Balance Between Resultant Error of Response and the Bias of Nonresponse. Journal of the American Statistical Association, 48, 743-772.CrossrefGoogle Scholar

  • Deville, J.C. and Särndal, C.-E. (1992). Calibration Estimators in Survey Sampling. Journal of the American Statistical Association, 87, 376-382.CrossrefGoogle Scholar

  • Dillman, D. (1978). Mail and Telephone Surveys: The Total Design Method. New York: Wiley.Google Scholar

  • Dillman, D., Smyth, J., and Christian, L. (2009). Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method, (3rd edition). New York: Wiley.Google Scholar

  • Dunkelburg, W. and Day, G. (1973). Nonresponse Bias and Callbacks in Sample Surveys. Journal of Marketing Research, 10, 160-168.CrossrefGoogle Scholar

  • Ferber, R. (1949). The Problem of Bias in Mail Returns: A Solution. Public Opinion Quarterly, 12, 669-676.Google Scholar

  • Feskins, R., Hoop, J., Lensvelt-Mulders, G., and Schmeets, H. (2011). Collecting Data Among Ethnic Minorities in an International Perspective. Field Methods, 18, 284-304.Google Scholar

  • Fuller, W.A., Loughin, M.M., and Baker, H.D. (1994). Regression Weighting for the 1987-88 National Food Consumption Survey. Survey Methodology, 20, 75-85.Google Scholar

  • Goyder, J. (1987). The Silent Minority: Nonrespondents on Sample Surveys. Boulder, CO: Westview Press.Google Scholar

  • Greenlees, J., Reece, W., and Zieschang, K. (1982). Imputation of Missing Values When the Probability of Response Depends on the Variable Being Imputed. Journal of the American Statistical Association, 77, 251-261.CrossrefGoogle Scholar

  • Groves, R.M. (2006). Nonresponse Rates and Nonresponse Bias in Household Surveys.Google Scholar

  • Groves, R.M. and Couper, M.P. (1998). Nonresponse in Household Interview Surveys. New York: Wiley.Google Scholar

  • Groves, R.M., Couper, M., Presser, S., Singer, E., Tourangeau, R., Acosta, G.P., and Nelson, L. (2006). Experiments in Producing Nonresponse Bias. Public Opinion Quarterly, 70, 720-736.CrossrefGoogle Scholar

  • Groves, R., Dillman, D., Eltinge, J., and Little, R. (2002). Survey Nonresponse. New York: Wiley, 41-54.Google Scholar

  • Groves, R.M. and Heeringa, S.G. (2006). Responsive Design for Household Surveys: Tools for Actively Controlling Survey Errors and Costs. Journal of the Royal Statistical Society, Series A, 169, 439-457. Google Scholar

  • Hansen, M.H. and Hurwitz, W.N. (1946). The Problem of Non-Response in Sample Surveys. Journal of the American Statistical Association, 41, 517-529.CrossrefGoogle Scholar

  • Haring, R., Alte, D., Völzkea, H., Sauer, S., Wallaschofski, H., John, U., and Schmidt, C. (2009). Extended Recruitment Efforts Minimize Attrition but not Necessarily Bias. Journal of Clinical Epidemiology, 62, 252-260.PubMedCrossrefGoogle Scholar

  • Hartley, H.O. (1946). Discussion of “A Review of Recent Statistical Developments in Sampling and Sample surveys.”. Journal of the Royal Statistical Society, 109, 37-38.Google Scholar

  • Heckman, J. (1979). Sample Selection Bias as a Specification Error. Econometrica, 47, 153-162.CrossrefGoogle Scholar

  • Holt, D. and Smith, T.M.F. (1979). Post-Stratification. Journal of the Royal Statistical Society, Series A, 142, 33-46.Google Scholar

  • Ingen, E., Stoop, I., and Breedveld, K. (2009). Nonresponse in the Dutch Time Use Survey: Strategies for Response Enhancement and Bias Reduction. Field Methods, 21, 69-90.Google Scholar

  • Kalton, G. (1983). Compensating for Missing Survey Data. Ann Arbor: University of Michigan Press.Google Scholar

  • Kalton, G. and Flores-Cervantes, I. (2003). Weighting Methods. Journal of Official Statistics, 18, 81-97.Google Scholar

  • Kalton, G. and Kasprzyk, D. (1986). The Treatment of Missing Survey Data. Survey Methodology, 12, 1-16.Google Scholar

  • Keeter, S., Miller, C., Kohut, A., Groves, R.M., and Presser, S. (2000). Consequences of Reducing Nonresponse in a Large National Telephone Survey. Public Opinion Quarterly, 64, 125-148.CrossrefGoogle Scholar

  • Kreuter, F.,Olson,K.,Wagner, J., Yan,T.,Ezzati-Rice, T.M.,Casas-Cordero, C., Lemay,M., Peytchev, A., Groves, R.M., and Raghunathan, T.E. (2010). Using Proxy Measures and Other Correlates of Survey Outcomes to Adjust for Non-Response: Examples from Multiple Surveys. Journal of the Royal Statistical Society, Series A, 173, 389-407.Google Scholar

  • Lin, I.-F. and Schaeffer, N.C. (1995). Using Survey Participants to Estimate the Impact of Nonparticipation. Public Opinion Quarterly, 59, 236-258.CrossrefGoogle Scholar

  • Little, R.J.A. (1986). Survey Nonresponse Adjustments for Estimates of Means. International Statistical Review, 54, 139-157.CrossrefGoogle Scholar

  • Little, R.J.A. (1993). Pattern-Mixture Models for Multivariate Incomplete Data. Journal of the American Statistical Association, 88, 125-134.Google Scholar

  • Little, R.J.A. and Rubin, D.B. (2002). Statistical Analysis With Missing data, (2nd edition). New York: Wiley.Google Scholar

  • Lumley, T., Shaw, P., and Dai, J. (2011). Connections Between Survey Calibration Estimators and Semiparametric Models for Incomplete Data. International Statistical Review, 79, 200-220.CrossrefGoogle Scholar

  • Lundstro¨m, S. and Särndal, C.-E. (1999). Calibration as a Standard Method for Treatment of Nonresponse. Journal of Official Statistics, 15, 305-327.Google Scholar

  • Madow, W.G., Nisselson, H., and Olkin, I. (1983). Incomplete Data in Sample Surveys, Vol. 1. New York: Academic Press.Google Scholar

  • Madow, W.G. and Olkin, I. (1983). Incomplete Data in Sample Surveys, Vol. 3. New York: Academic Press. Madow, W.G., Olkin, I., and Rubin, D.B. (1983). Incomplete Data in Sample Surveys, Vol. 2. New York: Academic Press.Google Scholar

  • Merkle, D., Edelman, M., Dykeman, K., and Brogan, C. (1998). An Experimental Study of Ways to Increase Exit Poll Response Rates and Reduce Survey Error. Paper presented at the Annual Conference of the American Association for Public Opinion Research, St. Louis, MO.Google Scholar

  • Micklewright, J., Schnepf, S., and Skinner, C. (2012). Non-Response Biases in Surveys of Schoolchildren: The Case of the English Programme for International Student Assessment (PISA) samples. Journal of the Royal Statistical Society, Series A, 175, 915-938.Google Scholar

  • Mohadjer, L., Berlin, M., Rieger, S., Waksberg, J., Rock, D., Yamamoto, K., Kirsch, I., and Kolstad, A. (1997). The Role of Incentives in Literacy Survey Research. Adult Basic Skills: Innovations in Measurement and Policy Analysis, A. Tuijnman, I. Kirsch, and D. Wagner (eds). Creskill, NJ: Hampton Press.Google Scholar

  • Molenberghs, G., Beunckens, C., Sotto, C., and Kenward, M.G. (2008). Every Missingness not at Random Model has a Missingness at Random Counterpart With Equal Fit. Journal of the Royal Statistical Society: Series B, 70, 371-388.CrossrefGoogle Scholar

  • Oh, H.L. and Scheuren, F.J. (1983). Weighting Adjustments for Unit Nonresponse. Incomplete Data in Sample Surveys, W.G. Madow, I. Olkin, and D.B. Rubin (eds). Vol. 2. New York: Academic Press, 143-184.Google Scholar

  • Olsen, K. and Groves, R.M. (2012). An Examination of Within-Person Variation in Response Propensity over the Data Collection Field Period. Journal of Official Statistics, 28, 29-51.Google Scholar

  • Peytcheva, E. and Groves, R.M. (2009). Using Variation in Response Rates of Demographic Subgroups as Evidence of Nonresponse Bias in Survey Estimates. Journal of Official Statistics, 25, 193-201.Google Scholar

  • Phipps, P. and Toth, D. (2012). Analyzing Establishment Nonresponse Using an Interpretable Regression Tree Model with Linked Administrative Data. Annals of Applied Statistics, 6, 772-794.CrossrefGoogle Scholar

  • Politz, A. and Simmons, W. (1949). An Attempt to Get “Not at Homes” Into the Sample Without Callbacks. Journal of the American Statistical Association, 44, 9-31.Google Scholar

  • Rosenbaum, P.R. and Rubin, D.B. (1983). The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika, 70, 41-55.CrossrefGoogle Scholar

  • Rubin, D.B. (1976). Inference and Missing Data (with discussion). Biometrika, 63, 581-592.CrossrefGoogle Scholar

  • Särndal, C.-E. (2011a). Morris Hansen Lecture: Dealing With Survey Nonresponse in Data Collection, in Estimation. Journal of Official Statistics, 27, 1-21.Google Scholar

  • Särndal, C.-E. (2011b). Three Factors to Signal Non-Response Bias with Applications to Categorical Auxiliary Variables. International Statistical Review, 79, 233-254.CrossrefGoogle Scholar

  • Särndal, C.-E. and Lundstro¨m, S. (2005). Estimation in Surveys with Nonresponse.Chichester, UK: Wiley.Google Scholar

  • Särndal, C.-E. and Lundstro¨m, S. (2008). Assessing Auxiliary Vectors for Control of Nonresponse Bias in the Calibration Estimator. Journal of Official Statistics, 4, 251-260. Google Scholar

  • Särndal, C.-E. and Lundstro¨m, S. (2010). Design for Estimation: Identifying Auxiliary vectors to reduce nonresponse bias. Survey Methodology, 36, 131-144.Google Scholar

  • Särndal, C.-E., Swensson, B., and Wretman, J. (1992). Model Assisted Survey Sampling. New York: Springer-Verlag.Google Scholar

  • Schmeets, H. (2010). Increasing Response Rates and the Consequences in the Dutch Parliamentary Election Study 2006. Field Methods, 22, 391-412.CrossrefGoogle Scholar

  • Schouten, B. (2007). A Selection Strategy for Weighting Variables Under a Not-Missingat- Random Assumption. Journal of Official Statistics, 23, 51-68.Google Scholar

  • Schouten, B., Calinescu, M., and Luiten, A. (2011a). Optimizing Quality of Response Through Adaptive Survey Designs. The Hague: Statistics Netherlands, Available at: http://www.cbs.nl/NR/rdonlyres/2D62BF4A-6783-4AC4-8E4512EF20C6675C/0/2011x1018.pdf. (Accessed May 24, 2013).Google Scholar

  • Schouten, B., Cobben, F., and Bethlehem, J. (2009). Measures for the Representativeness of Survey Response. Survey Methodology, 35, 101-113.Google Scholar

  • Schouten, B., Schlomo, N., and Skinner, C. (2011b). Indicators for Monitoring and Improving Representativeness of Response. Journal of Official Statistics, 27, 231-253.Google Scholar

  • Singer, E. (2002). Use of Incentives to Reduce Nonresponse in Household Surveys.Google Scholar

  • Survey Nonresponse, R. Groves, D. Dillman, J. Eltinge, and R. Little (eds). New York: Wiley, 163-177.Google Scholar

  • Singer, E. and Ye, C. (2013). The Use and Effects of Incentives in Surveys. The ANNALS of the American Academy of Political and Social Science, 645, 112-141.Google Scholar

  • Skinner, C.J. and D’Arrigo, J. (2011). Inverse Probability Weighting for Clustered Nonresponse. Biometrika, 98, 953-966.CrossrefGoogle Scholar

  • Smith, T.W. (1995). Trends in Non-Response Rates. International Journal of Public Opinion Research, 7, 157-171.CrossrefGoogle Scholar

  • Steeh, C., Kirgis, N., Cannon, B., and DeWitt, J. (2001). Are They Really as Bad as They Seem? Nonresponse Rates at the End of the Twentieth Century. Journal of Official Statistics, 17, 227-247.Google Scholar

  • Steele, F. and Durrant, G.B. (2011). Alternative Approaches to Multilevel Modelling of Survey Non-Contact and Refusal. International Statistical Review, 79, 70-91.CrossrefGoogle Scholar

  • Stoop, I.A.L. (2005). The Hunt for the Last Respondent: Nonresponse in Sample Surveys. The Hague: Social and Cultural Planning Office.Google Scholar

  • Stoop, I., Billiet, J., Koch, A., and Fitzgerald, R. (2010). Improving Survey Response: Lessons Learned from the European Social Survey. Chichester: Wiley.Google Scholar

  • Synodinos, N.E. and Yamada, S. (2000). Response Rate Trends in Japanese Surveys. International Journal of Public Opinion Research, 12, 48-72.CrossrefGoogle Scholar

  • Tanur, J. (1999). Looking Backwards and Forwards at the CASM Movement. Cognition and Survey Research, M. Sirken, D. Hermann, S. Schechter, N. Schwarz, J. Tanur, and R. Tourangeau (eds). New York: Wiley, 13-20.Google Scholar

  • Thomsen, I. (1973). A Note on the Efficiency of Weighting Subclass Means to Reduce the Effects of Nonresponse When Analyzing Survey Data. Statistisk Tidskrift, 11, 278-285.Google Scholar

  • Tourangeau, R., Rips, L.J., and Rasinski, K. (2000). The Psychology of Survey Response. New York: Cambridge University Press. Wagner, J. (2010). The Fraction of Missing Information as a Tool for Monitoring the Quality of Survey Data. Public Opinion Quarterly, 74, 223-243.Google Scholar

  • Wetzels, W., Schmeets, H., Van den Brakel, J., and Feskens, R. (2008). Impact of Prepaid Incentives in Face-to-Face Surveys: A Large-Scale Experiment With Postage Stamps. International Journal of Public Opinion Research, 20, 507-516.CrossrefGoogle Scholar

  • Yates, F. (1946). A Review of Recent Statistical Developments in Sampling and Sample Surveys. Journal of the Royal Statistical Society, 109, 12-43. CrossrefGoogle Scholar

About the article

Published Online: 2013-10-03

Published in Print: 2013-06-01


Citation Information: Journal of Official Statistics, ISSN (Online) 2001-7367, DOI: https://doi.org/10.2478/jos-2013-0026.

Export Citation

This content is open access.

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

[1]
Marko Sarstedt, Paul Bengart, Abdel Monim Shaltoni, and Sebastian Lehmann
International Journal of Advertising, 2017, Page 1
[2]
Boyd N. Barrett, Brett van Poorten, Andrew B. Cooper, and Wolfgang Haider
North American Journal of Fisheries Management, 2017, Volume 37, Number 4, Page 756
[3]
[4]
Caroline Vandenplas, Michèle Ernst Stähli, Dominique Joye, and Alexandre Pollien
Mathematical Population Studies, 2017, Volume 24, Number 2, Page 103
[5]
Sarah J. Howcutt, Anna L. Barnett, Sofia Barbosa-Boucas, and Lesley A. Smith
Women & Health, 2017, Page 1
[6]
Michael P. Battaglia, Don A. Dillman, Martin R. Frankel, Rachel Harter, Trent D. Buskirk, Cameron B. McPhee, Jill M. DeMatteis, and Tracey Yancey
Journal of Survey Statistics and Methodology, 2016, Volume 4, Number 4, Page 476
[7]
Per Gösta Andersson and Carl-Erik Särndal
Statistical Journal of the IAOS, 2016, Volume 32, Number 3, Page 375
[8]
Pascal Sciarini and Andreas C. Goldberg
Journal of Survey Statistics and Methodology, 2016, Volume 4, Number 1, Page 110
[9]
Eleonora Dal Grande, Catherine R. Chittleborough, Stefano Campostrini, Graeme Tucker, and Anne W. Taylor
American Journal of Epidemiology, 2015, Volume 182, Number 6, Page 544
[10]
Qixuan Chen, Andrew Gelman, Melissa Tracy, Fran H. Norris, and Sandro Galea
Statistics in Medicine, 2015, Volume 34, Number 28, Page 3637
[11]
Hans Walter Steinhauer, Christian Aßmann, Sabine Zinn, Solange Goßmann, and Susanne Rässler
AStA Wirtschafts- und Sozialstatistisches Archiv, 2015, Volume 9, Number 2, Page 131

Comments (0)

Please log in or register to comment.
Log in