Jump to ContentJump to Main Navigation
Show Summary Details

Journal of Official Statistics

The Journal of Statistics Sweden

4 Issues per year


IMPACT FACTOR 2015: 0.467
5-year IMPACT FACTOR: 0.740


SCImago Journal Rank (SJR) 2015: 0.410
Source Normalized Impact per Paper (SNIP) 2015: 0.810
Impact per Publication (IPP) 2015: 0.540

Open Access
Online
ISSN
2001-7367
See all formats and pricing

Selective Editing: A Quest for Efficiency and Data Quality

Ton de Waal
  • Corresponding author
  • Statistics Netherlands, PO Box 24500, 2490 HA The Hague, The Netherlands
  • Email:
Published Online: 2013-11-09 | DOI: https://doi.org/10.2478/jos-2013-0036

Abstract

National statistical institutes are responsible for publishing high quality statistical information on many different aspects of society. This task is complicated considerably by the fact that data collected by statistical offices often contain errors. The process of correcting errors is referred to as statistical data editing. For many years this has been a purely manual process, with people checking the collected data record by record and correcting them if necessary. For this reason the data editing process has been both expensive and time-consuming. This article sketches some of the important methodological developments aiming to improve the efficiency of the data editing process that have occurred during the past few decades. The article focuses on selective editing, which is based on an idea rather shocking for people working in the production of high-quality data: that it is not necessary to find and correct all errors. Instead of trying to correct all errors, it generally suffices to correct only those errors where data editing has substantial influence on publication figures. This overview article sketches the background of selective editing, describes the most usual form of selective editing up to now, and discusses the contributions to this special issue of the Journal of Official Statistics on selective editing. The article concludes with describing some possible directions for future research on selective editing and statistical data editing in general.

Keywords: Errors; score function selective editing; statistical data editing

  • Bakker, B. (2011). Micro-integration. Statistical Methods 201108, Statistics Netherlands.

  • Banff Support Team (2008). Functional Description of the Banff System for Edit and Imputation. Technical Report, Statistics Canada.

  • Barcaroli, G., Ceccarelli, C., Luzi, O., Manzari, A., Riccini, E., and Silvestri, F. (1995). The Methodology of Editing and Imputation of Qualitative Variables Implemented in SCIA. Internal Report, Istituto Nazionale di Statistica, Rome.

  • De Waal, T. (1996). CherryPi: A Computer Program for Automatic Edit and Imputation. UN/ECE Work Session on Statistical Data Editing, 4-7 November, Voorburg.

  • De Waal, T. (2001). SLICE: Generalised Software for Statistical Data Editing. Proceedings in Computational Statistics, J.G. Bethlehem and P.G.M. Van der Heijden (eds). New York: Physica-Verlag, 277-282.

  • De Waal, T. and Coutinho, W. (2005). Automatic Editing for Business Surveys: an Assessment for Selected Algorithms. International Statistical Review, 73, 73-102.

  • De Waal, T., Pannekoek, J., and Scholtus, S. (2011). Handbook of Statistical Data Editing and Imputation. New York: John Wiley and Sons.

  • Farwell, K. (2005). Significance Editing for a Variety of Survey Situations. Paper presented at the 55th session of the International Statistical Institute, 5-12 April, Sydney.

  • Fellegi, I.P. and Holt, D. (1976). A Systematic Approach to Automatic Edit and Imputation. Journal of the American Statistical Association, 71, 17-35.

  • Freund, R.J. and Hartley, H.O. (1967). A Procedure for Automatic Data Editing. Journal of the American Statistical Association, 62, 341-352.

  • Granquist, L. (1984). Data Editing and its Impact on the Further Processing of Statistical Data. In Workshop on Statistical Computing, 12-17, Budapest.

  • Granquist, L. (1990). A Review of Some Macro-Editing Methods for Rationalizing the Editing Process. Proceedings of the Statistics Canada Symposium, 225-234.

  • Granquist, L. (1995). Improving the Traditional Editing Process. In Business Survey Methods, B.G. Cox, D.A. Binder, B.N. Chinnappa, A. Christianson, M.J. Colledge, and P.S. Kott (eds). New York: John Wiley & Sons, 385-401.

  • Granquist, L. (1997). The New View on Editing. International Statistical Review, 65, 381-387. [Web of Science]

  • Granquist, L. and Kovar, J.G. (1997). Editing of Survey Data: How Much is Enough? In Survey Measurement and Process Quality, L.E. Lyberg, P. Biemer, M. Collins, E.D. De Leeuw, C. Dippo, N. Schwartz and D. Trewin (eds). Hoboken, NJ: Wiley Series in Probability and Statistics, Wiley, 416-435. DOI: http://www.dx.doi.org/10.1002/9781118490013.ch18 [Crossref]

  • Groves, R.M. (2011). Three Eras of Survey Research. Public Opinion Quarterly, 75, 861-871. DOI: http://www.dx.doi.org/10.1093/poq/nfr057 [Web of Science] [Crossref]

  • Hedlin, D. (2003). Score Functions to Reduce Business Survey Editing at the U.K. Office for National Statistics. Journal of Official Statistics, 19, 177-199.

  • Hedlin, D. (2008). Local and Global Score Functions in Selective Editing. UN/ECE Work Session on Statistical Data Editing, 21-23 April, Vienna.

  • Hidiroglou, M.A. and Berthelot, J.-M. (1986). Statistical Editing and Imputation for Periodic Business Surveys. Survey Methodology, 12, 73-83.

  • Kovar, J. and Whitridge, P. (1990). Generalized Edit and Imputation System; Overview and Applications. Revista Brasileira de Estadistica, 51, 85-100.

  • Latouche, M. and Berthelot, J.-M. (1992). Use of a Score Function to Prioritize and Limit Recontacts in Editing Business Surveys. Journal of Official Statistics, 8, 389-400.

  • Lawrence, D. and McDavitt, C. (1994). Significance Editing in the Australian Survey of Average Weekly Earnings. Journal of Official Statistics, 10, 437-447.

  • Lawrence, D. and McKenzie, R. (2000). The General Application of Significance Editing. Journal of Official Statistics, 16, 243-253.

  • Nordbotten, S. (1955). Measuring the Error of Editing the Questionnaires in a Census. Journal of the American Statistical Association, 50, 364-369.

  • Nordbotten, S. (1963). Automatic Editing of Individual Statistical Observations. Statistical Standards and Studies No. 2. UN Statistical Commission and Economic Commission of Europe, New York.

  • Pritzker, L., Ogus, J., and Hansen, M.H. (1965). Computer Editing Methods-Some Applications and Results. Bulletin of the International Statistical Institute, Proceedings of the 35th Session, Belgrade, 395-417.

  • Pullum, T.W., Harpham, T., and Ozsever, N. (1986). The Machine Editing of Large- Sample Surveys: The Experience of the World Fertility Survey. International Statistical Review, 54, 311-326.

  • Pursey, S. (1994). Current and Future Approaches to Editing Canadian Trade Import Data. In proceedings of the Survey Research Methods Section, American Statistical Association, 105-109.

  • Struijs, P., Camstra, A., Renssen, R., and Braaksma, B. (2013). Redesign of Statistics Production within an Architectural Framework: The Dutch Experience. Journal of Official Statistics, 29, 49-71. [Web of Science]

  • Todaro, T.A. (1999). Overview and Evaluation of the AGGIES Automated Edit and Imputation System. UN/ECE Work Session on Statistical Data Editing, 2-4 June, Rome.

  • Van de Pol, F. and Bethlehem, J. (1997). Data Editing Perspectives. Statistical Journal of the United Nations ECE, 14, 153-171.

  • Winkler, W.E. and Draper, L.A. (1997). The SPEER Edit System. In Statistical Data Editing (Volume 2); Methods and Techniques, 51-55, Geneva: United Nations.

  • Winkler, W.E. and Petkunas, T.F. (1997). The DISCRETE Edit System. In Statistical Data Editing (Volume 2); Methods and Techniques, 56-62, Geneva: United Nations.

About the article

Published Online: 2013-11-09

Published in Print: 2013-12-01


Citation Information: Journal of Official Statistics, ISSN (Online) 2001-7367, DOI: https://doi.org/10.2478/jos-2013-0036. Export Citation

This content is open access.

Comments (0)

Please log in or register to comment.
Log in