Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Journal of Official Statistics

The Journal of Statistics Sweden

4 Issues per year


IMPACT FACTOR 2016: 0.411
5-year IMPACT FACTOR: 0.776

CiteScore 2016: 0.63

SCImago Journal Rank (SJR) 2016: 0.710
Source Normalized Impact per Paper (SNIP) 2016: 0.975

Open Access
Online
ISSN
2001-7367
See all formats and pricing
More options …

Selective Editing: A Quest for Efficiency and Data Quality

Ton de Waal
Published Online: 2013-11-09 | DOI: https://doi.org/10.2478/jos-2013-0036

Abstract

National statistical institutes are responsible for publishing high quality statistical information on many different aspects of society. This task is complicated considerably by the fact that data collected by statistical offices often contain errors. The process of correcting errors is referred to as statistical data editing. For many years this has been a purely manual process, with people checking the collected data record by record and correcting them if necessary. For this reason the data editing process has been both expensive and time-consuming. This article sketches some of the important methodological developments aiming to improve the efficiency of the data editing process that have occurred during the past few decades. The article focuses on selective editing, which is based on an idea rather shocking for people working in the production of high-quality data: that it is not necessary to find and correct all errors. Instead of trying to correct all errors, it generally suffices to correct only those errors where data editing has substantial influence on publication figures. This overview article sketches the background of selective editing, describes the most usual form of selective editing up to now, and discusses the contributions to this special issue of the Journal of Official Statistics on selective editing. The article concludes with describing some possible directions for future research on selective editing and statistical data editing in general.

Keywords: Errors; score function selective editing; statistical data editing

  • Bakker, B. (2011). Micro-integration. Statistical Methods 201108, Statistics Netherlands.Google Scholar

  • Banff Support Team (2008). Functional Description of the Banff System for Edit and Imputation. Technical Report, Statistics Canada.Google Scholar

  • Barcaroli, G., Ceccarelli, C., Luzi, O., Manzari, A., Riccini, E., and Silvestri, F. (1995). The Methodology of Editing and Imputation of Qualitative Variables Implemented in SCIA. Internal Report, Istituto Nazionale di Statistica, Rome.Google Scholar

  • De Waal, T. (1996). CherryPi: A Computer Program for Automatic Edit and Imputation. UN/ECE Work Session on Statistical Data Editing, 4-7 November, Voorburg.Google Scholar

  • De Waal, T. (2001). SLICE: Generalised Software for Statistical Data Editing. Proceedings in Computational Statistics, J.G. Bethlehem and P.G.M. Van der Heijden (eds). New York: Physica-Verlag, 277-282.Google Scholar

  • De Waal, T. and Coutinho, W. (2005). Automatic Editing for Business Surveys: an Assessment for Selected Algorithms. International Statistical Review, 73, 73-102.Google Scholar

  • De Waal, T., Pannekoek, J., and Scholtus, S. (2011). Handbook of Statistical Data Editing and Imputation. New York: John Wiley and Sons.Google Scholar

  • Farwell, K. (2005). Significance Editing for a Variety of Survey Situations. Paper presented at the 55th session of the International Statistical Institute, 5-12 April, Sydney.Google Scholar

  • Fellegi, I.P. and Holt, D. (1976). A Systematic Approach to Automatic Edit and Imputation. Journal of the American Statistical Association, 71, 17-35.Google Scholar

  • Freund, R.J. and Hartley, H.O. (1967). A Procedure for Automatic Data Editing. Journal of the American Statistical Association, 62, 341-352.Google Scholar

  • Granquist, L. (1984). Data Editing and its Impact on the Further Processing of Statistical Data. In Workshop on Statistical Computing, 12-17, Budapest.Google Scholar

  • Granquist, L. (1990). A Review of Some Macro-Editing Methods for Rationalizing the Editing Process. Proceedings of the Statistics Canada Symposium, 225-234.Google Scholar

  • Granquist, L. (1995). Improving the Traditional Editing Process. In Business Survey Methods, B.G. Cox, D.A. Binder, B.N. Chinnappa, A. Christianson, M.J. Colledge, and P.S. Kott (eds). New York: John Wiley & Sons, 385-401.Google Scholar

  • Granquist, L. (1997). The New View on Editing. International Statistical Review, 65, 381-387.Web of ScienceGoogle Scholar

  • Granquist, L. and Kovar, J.G. (1997). Editing of Survey Data: How Much is Enough? In Survey Measurement and Process Quality, L.E. Lyberg, P. Biemer, M. Collins, E.D. De Leeuw, C. Dippo, N. Schwartz and D. Trewin (eds). Hoboken, NJ: Wiley Series in Probability and Statistics, Wiley, 416-435. DOI: http://www.dx.doi.org/10.1002/9781118490013.ch18 CrossrefGoogle Scholar

  • Groves, R.M. (2011). Three Eras of Survey Research. Public Opinion Quarterly, 75, 861-871. DOI: http://www.dx.doi.org/10.1093/poq/nfr057Web of ScienceCrossrefGoogle Scholar

  • Hedlin, D. (2003). Score Functions to Reduce Business Survey Editing at the U.K. Office for National Statistics. Journal of Official Statistics, 19, 177-199.Google Scholar

  • Hedlin, D. (2008). Local and Global Score Functions in Selective Editing. UN/ECE Work Session on Statistical Data Editing, 21-23 April, Vienna.Google Scholar

  • Hidiroglou, M.A. and Berthelot, J.-M. (1986). Statistical Editing and Imputation for Periodic Business Surveys. Survey Methodology, 12, 73-83.Google Scholar

  • Kovar, J. and Whitridge, P. (1990). Generalized Edit and Imputation System; Overview and Applications. Revista Brasileira de Estadistica, 51, 85-100.Google Scholar

  • Latouche, M. and Berthelot, J.-M. (1992). Use of a Score Function to Prioritize and Limit Recontacts in Editing Business Surveys. Journal of Official Statistics, 8, 389-400.Google Scholar

  • Lawrence, D. and McDavitt, C. (1994). Significance Editing in the Australian Survey of Average Weekly Earnings. Journal of Official Statistics, 10, 437-447.Google Scholar

  • Lawrence, D. and McKenzie, R. (2000). The General Application of Significance Editing. Journal of Official Statistics, 16, 243-253.Google Scholar

  • Nordbotten, S. (1955). Measuring the Error of Editing the Questionnaires in a Census. Journal of the American Statistical Association, 50, 364-369.Google Scholar

  • Nordbotten, S. (1963). Automatic Editing of Individual Statistical Observations. Statistical Standards and Studies No. 2. UN Statistical Commission and Economic Commission of Europe, New York.Google Scholar

  • Pritzker, L., Ogus, J., and Hansen, M.H. (1965). Computer Editing Methods-Some Applications and Results. Bulletin of the International Statistical Institute, Proceedings of the 35th Session, Belgrade, 395-417.Google Scholar

  • Pullum, T.W., Harpham, T., and Ozsever, N. (1986). The Machine Editing of Large- Sample Surveys: The Experience of the World Fertility Survey. International Statistical Review, 54, 311-326.Google Scholar

  • Pursey, S. (1994). Current and Future Approaches to Editing Canadian Trade Import Data. In proceedings of the Survey Research Methods Section, American Statistical Association, 105-109.Google Scholar

  • Struijs, P., Camstra, A., Renssen, R., and Braaksma, B. (2013). Redesign of Statistics Production within an Architectural Framework: The Dutch Experience. Journal of Official Statistics, 29, 49-71.Web of ScienceGoogle Scholar

  • Todaro, T.A. (1999). Overview and Evaluation of the AGGIES Automated Edit and Imputation System. UN/ECE Work Session on Statistical Data Editing, 2-4 June, Rome.Google Scholar

  • Van de Pol, F. and Bethlehem, J. (1997). Data Editing Perspectives. Statistical Journal of the United Nations ECE, 14, 153-171.Google Scholar

  • Winkler, W.E. and Draper, L.A. (1997). The SPEER Edit System. In Statistical Data Editing (Volume 2); Methods and Techniques, 51-55, Geneva: United Nations.Google Scholar

  • Winkler, W.E. and Petkunas, T.F. (1997). The DISCRETE Edit System. In Statistical Data Editing (Volume 2); Methods and Techniques, 56-62, Geneva: United Nations. Google Scholar

About the article

Published Online: 2013-11-09

Published in Print: 2013-12-01


Citation Information: Journal of Official Statistics, ISSN (Online) 2001-7367, DOI: https://doi.org/10.2478/jos-2013-0036.

Export Citation

This content is open access.

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

[1]
Hang J. Kim, Lawrence H. Cox, Alan F. Karr, Jerome P. Reiter, and Quanli Wang
Journal of the American Statistical Association, 2015, Volume 110, Number 511, Page 987

Comments (0)

Please log in or register to comment.
Log in