Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Journal of Official Statistics

The Journal of Statistics Sweden

4 Issues per year

IMPACT FACTOR 2016: 0.411
5-year IMPACT FACTOR: 0.776

CiteScore 2016: 0.63

SCImago Journal Rank (SJR) 2016: 0.710
Source Normalized Impact per Paper (SNIP) 2016: 0.975

Open Access
See all formats and pricing
More options …

A System for Managing the Quality of Official Statistics

Paul Biemer / Dennis Trewin / Heather Bergdahl / Lilli Japec
Published Online: 2014-09-02 | DOI: https://doi.org/10.2478/jos-2014-0022


This article describes a general framework for improving the quality of statistical programs in organizations that provide a continual flow of statistical products to users and stakeholders. The work stems from a 2011 mandate to Statistics Sweden issued by the Swedish Ministry of Finance to develop a system of quality indicators for tracking developments and changes in product quality and for achieving continual improvements in survey quality across a diverse set of key statistical products. We describe this system, apply it to a number of products at Statistics Sweden, and summarize key results and lessons learned. The implications of this work for monitoring and evaluating product quality in other statistical organizations are also discussed.

Keywords: Total survey error; process control; GDP; quality indicators; statistical standards


  • Andersen, R., J. Kaspar, and M. Frankel. 1979. Total Survey Error. San Francisco: Jossey-Bass Publishers.Google Scholar

  • Baldrige Performance Excellence Program 2013. The 2013-2014 Criteria for Performance Excellence. Available at: http://www.nist.gov/baldrige/ (accessed August 3, 2013).Google Scholar

  • Barkley, B.T. 2004. Project Risk Management. New York: McGraw Hill Professional.Google Scholar

  • Biemer, P. 2011. Latent Class Analysis of Survey Error. Hoboken, NJ: John Wiley & Sons.Google Scholar

  • Biemer, P. 2014. “Comment on ‘On Information Quality’ by Kenett and Shmueli.” Journal of the Royal Statistical Society, Series A. Vol. 177, Part 1: 27-29.Google Scholar

  • Biemer, P. and L. Lyberg. 2003. Introduction to Survey Quality. New York: John Wiley & Sons.Google Scholar

  • Biemer, P. and D. Trewin. 2012. Development of Quality Indicators at Statistics Sweden. Report to Statistics Sweden, January 2012.Google Scholar

  • Biemer, P. and D. Trewin. 2013. A Second Application of the ASPIRE Quality Evaluation System for Statistics Sweden. Report to Statistics Sweden, January 2013.Google Scholar

  • Biemer, P. and D. Trewin. 2014. A Third Application of ASPIRE for Statistics Sweden. Report to Statistics Sweden, January 2014.Google Scholar

  • Brackstone, G. 1999. “Managing Data Quality in a Statistical Agency.” Survey Methodology 25: 139-149.Google Scholar

  • Breyfogle, F. 2003. Implementing Six Sigma, 2nd edition. New York: John Wiley & Sons.Google Scholar

  • Conley-Tyler, M. 2005. “A Fundamental Choice: Internal or External Evaluation?” Evaluation Journal of Australasia 4: 3-11.Google Scholar

  • COSO, 2004. Enterprise Risk Management - Integrated Framework. Available at: http:// www.coso.org/documents/coso_erm_executivesummary.pdf (accessed August 3, 2013).Google Scholar

  • COSO, 2013. Internal Control - Integrated Framework, 2013. Available at: http://www.coso.org/documents/coso%202013%20icfr%20executive_summary.pdf (accessed August 3, 2013).Google Scholar

  • Couper, M. and L. Lyberg. 2005. “The Use of Paradata in Survey Research.” In Proceedings of the 55th Session of the International Statistical Institute, Sydney, Australia, April 7, 2005. Available at: http://isi.cbs.nl/iamamember/CD6-Sydney2005/ISI_Final_Proceedings.htm (accessed June 26, 2014).Google Scholar

  • Curtin, R., S. Presser, and E. Singer. 2000. “The Effects of Response Rate Changes on the Index of Consumer Sentiment.” Public Opinion Quarterly 64: 413-428.PubMedCrossrefGoogle Scholar

  • Dalenius, T. 1967. Nonsampling Errors in Census and Sample Surveys. Report no. 5 in the research project Errors in Surveys, Stockholm University.Google Scholar

  • Deming, E. 1944. “On Errors in Surveys.” American Sociological Review 9: 359-369.CrossrefGoogle Scholar

  • Deming, E. 1986. Out of the Crisis. Cambridge, MA: MIT Press.Google Scholar

  • EFQM, 2013. “An Overview of the Excellence Model.” Available at: https://www.google.com/url?q=http://www2.efqm.org/en/PdfResources/EFQM%2520Excellence%2520Model%25202013%520EN%2520extract.pdf&sa=U&ei=9BasU4nkHsqTqAbUhIGQCg&ved=0CAUQFjAA&client=internal-uds-cse&usg=AFQjCNHthJnhRPIS1t6cfa4Ka9ePXOLRxg (accessed June 26, 2014).Google Scholar

  • Eltinge, J., P. Biemer, and A. Holmberg. 2013. “A Potential Framework for Integration of Architecture and Methodology to Improve Statistical Production Systems.” Journal of Official Statistics 29: 125-145. DOI: http://dx.doi.org/10.2478/jos-2013-0007.CrossrefGoogle Scholar

  • European Statistical System (ESS) 2011. “Quality Assurance Framework of the European Statistical System, Version 1.1.” Available at: http://epp.eurostat.ec.europa.eu/cache/ITY_PUBLIC/QAF_2012/EN/QAF_2012-EN.PDF (accessed August 9, 2013). Google Scholar

  • Eurostat 2005. “European Statistics Code of Practice, Revised Edition.” Available at: http://epp.eurostat.ec.europa.eu/cache/ITY_OFFPUB/KS-32-11-955/EN/KS-32-11-955-EN.PDF (accessed June 26, 2014).Google Scholar

  • Eurostat 2009. Regulation (EC) No 223/2009 of the European Parliament and of the Council of 11 March 2009, Eurostat General/Standard report, Luxembourg, April 4-5. Available at: http://eur-lex.europa.eu/legal-content/EN/ALL/?uri=CELEX:R0223 (accessed June 18, 2014).Google Scholar

  • Gonzales, M.E., J.L. Ogus, G. Shapiro, and B.J. Tepping. 1975. “Standards for Discussion and Presentation of Errors in Surveys and Census Data.” Journal of American Statistical Association 70: 5-23.Google Scholar

  • Groves, R.M. and L.E. Lyberg. 2010. “Total Survey Error: Past, Present, and Future.” Public Opinion Quarterly 74: 849-879. DOI:http://dx.doi.org/10.1093/poq/nfq065.CrossrefGoogle Scholar

  • Hansen, M., W. Hurwitz, and W. Madow. 1953. Sample Survey Methods and Theory, Volumes I and II. New York: John Wiley & Sons.Google Scholar

  • Hansen, M., W. Hurwitz, and L. Pritzker. 1967. Standardization of Procedures for the Evaluation of Data: Measurement Errors and Statistical Standards in the Bureau of the Census. Paper presented at the 36th session of the International Statistical Institute.Google Scholar

  • Imai, M. 1986. Kaisen: the Key to Japan’s Competitive Success. New York: McGraw-Hill Education.Google Scholar

  • International Monetary Fund (IMF) 2003. Data Quality Assessment Framework and Data Quality Program. Available at: http://www.imf.org/external/np/sta/dsbb/2003/eng/dqaf.htm (accessed June 21, 2013).Google Scholar

  • International Standards Organization 2006. Market, Opinion and Social Research ISO Standard No. 20252. Available at: www.standards.org/standards/listing/iso_20252 (accessed August 8, 2014).Google Scholar

  • International Standards Organization 2009. Risk Mangement: Principles and Guidelines for Implementation, ISO/DIS 31000 Standard No. 31000. Available at: www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=43170 (accessed August 8, 2014).Google Scholar

  • Journal of Official Statistics 2013. Special Issue on Systems and Architectures for High-Quality Statistics Production, edited by B. Lorenc, I. Jansson, P. Biemer, J. Eltinge, and A. Holmberg, Vol. 1, March, 2013.Google Scholar

  • Juran, J. and B. Godfrey. 1999. Juran’s Quality Handbook. New York: McGraw-Hill.Google Scholar

  • Karsak, E.E. 2004. “Fuzzy Multiple Objective Decision Making Approach to Prioritize Design Requirements in Quality Function Deployment.” International Journal of Production Research 42: 3957-3974.CrossrefGoogle Scholar

  • Keeter, S., C. Miller, A. Kohut, R. Groves, and S. Presser. 2000. “Consequences of Reducing Nonresponse in a Large National Telephone Survey.” Public Opinion Quarterly 64: 125-148. DOI: http://dx.doi.org/10.1086/317759.CrossrefGoogle Scholar

  • Kenett, R.S. and G. Shmueli. 2014. “On Information Quality.” Journal of the Royal Statistical Society, Series A 177: 3-38. DOI:http://dx.doi.org/10.1111/rssa.12007.CrossrefGoogle Scholar

  • Kish, L. 1962. “Studies of Interviewer Variance for Attitudinal Variables.” Journal of the American Statistical Association 57: 92-115. Lequiller, F and D. Blades. 2006. Understanding National Accounts. Paris: OECD 2006. Available at: http://www.eastafritac.org/images/uploads/documents_storage/Understanding_National_Accounts_-_OECD.pdf (accessed June 21, 2013).CrossrefGoogle Scholar

  • Lyberg, L. and P. Biemer. 2008. “Quality Assurance and Quality Control in Surveys.” In International Handbook on Survey Methodology, edited by J. Hox, E. de Leeuw, and D. Dillman, 421-441. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar

  • Lyberg, L., L. Japec, and P. Biemer. 1998. “Quality Improvement in Surveys - A Process Perspective.” In Proceedings of the Survey Research Methods Section of the American Statistical Association, 23-31.Google Scholar

  • Lyberg, L. 2012. “Survey Quality.” Survey Methodology 38: 107-130.Google Scholar

  • McDavid, J., I. Huse, and L. Hawthorn. 2013. Program Evaluation and Performance Measurement: An Introduction to Practice, Second Edition. New York: Sage Publications.Google Scholar

  • Michalek, J.J., O. Ceryan, P.Y. Papalambros, and Y. Koren. 2006. “Balancing Marketing and Manufacturing Objectives in Product Line Design.” ASME Journal of Mechanical Design 128: 1196-1204. DOI: http://dx.doi.org/10.1115/1.2336252.CrossrefGoogle Scholar

  • Merkle, D. and M. Edelman. 2002. “Nonresponse in Exit Polls: A Comprehensive Analysis.” In Survey Nonresponse, edited by R. Groves, D. Dillman, J. Eltinge, and R. Little, 243-257. New York: John Wiley and Sons.Google Scholar

  • Morganstein, D. and D. Marker. 1997. “Continuous Quality Improvement in Statistical Agencies.” In Survey Measurement and Process Quality, edited by L. Lyberg, P. Biemer, M. Collins, E. de Leeuw, C. Dippo, N. Schwarz, and D. Trewin, 475-500. New York: Wiley and Sons.Google Scholar

  • Nealon, J. and E. Gleaton. 2013. “Consolidation and Standardization of Survey Operations at a Decentralized Federal Statistical Agency.” Journal of Official Statistic 29: 5-28. DOI: http://dx.doi.org/10.2478/jos-2013-0002.CrossrefGoogle Scholar

  • Neyman, J. 1934. “On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection.” Journal of the Royal Statistical Society 97: 558-606.CrossrefGoogle Scholar

  • Neyman, J. 1938. Lectures and Conferences on Mathematical Statistics and Probability. Washington, DC: U.S. Department of Agriculture.Google Scholar

  • Organisation for Economic Cooperation and Development (OECD) 2011. Quality Framework and Guidelines for OECD Statistical Activities. Available at: http://search.oecd.org/officialdocuments/displaydocumentpdf/?cote=std/qfs%282011%291&doclanguage= en (accessed June 21, 2013).Google Scholar

  • Office of National Statistics (ONS) 2007. Guidelines for Measuring Statistical Quality, Version 3.1. Available at: http://www.ons.gov.uk/ons/guide-method/method-quality/quality/guidelines-for-measuring-statistical-quality/index.html (accessed June 21, 2013).Google Scholar

  • Rossi, P.H., W.M. Lipsey, and H.E. Freeman. 2004. Evaluation: A Systematic Approach. 7th ed. Thousand Oaks, CA: Sage Publishers.Google Scholar

  • Seyb, A., R. McKenzie, and A. Skerrett. 2013. “Innovative Production Systems at New Zealand: Overcoming the Design and Build Bottleneck.” Journal of Official Statistics 29: 73-97. DOI: http://dx.doi.org/10.2478/jos-2013-0005. CrossrefGoogle Scholar

  • Statistics Canada 2009. Statistics Canada Quality Guidelines, Fifth Edition. Available at: http://www5.statcan.gc.ca/bsolc/olc-cel/olc-cel?catno=12-539-X&CHROPG=1&lang=eng (accessed March 10, 2014).Google Scholar

  • Statistiska centralbyra°n 2001. Quality Definition and Recommendations for Quality Declarations of Official Statistics. Available at: http://www.scb.se/Grupp/Hitta_statistik/Forsta_Statistik/Metod/_Dokument/MIS2001_1.pdf (accessed June 18, 2014).Google Scholar

  • Stephan, F.F. 1948. “History of the Uses of Modern Sampling Procedures.” Journal of the American Statistical Association 43: 12-39.CrossrefGoogle Scholar

  • Struijs, P., A. Camstra, R. Renssen, and B. Braaksma. 2013. “Redesign of Statistics Production within an Architectural Framework: The Dutch Experience.” Journal of Official Statistics 29: 49-71. DOI: http://dx.doi.org/10.2478/jos-2013-0004.CrossrefGoogle Scholar

  • U.S. Bureau of the Census 1974. “Technical Paper 32: Standards for Discussion and Presentation of Errors in Data. U.S. Department of Commerce.” U.S. Government Printing Office, Technical Paper 32, Department of Commerce.Google Scholar

  • U.S. Office of Management and Budget 2002. “Guidelines for Ensuring, and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies.” Federal Register, 67, 36, February 22. Google Scholar

About the article

Received: 2013-08-01

Revised: 2014-04-01

Accepted: 2014-06-01

Published Online: 2014-09-02

Published in Print: 2014-09-01

Citation Information: Journal of Official Statistics, Volume 30, Issue 3, Pages 381–415, ISSN (Online) 2001-7367, DOI: https://doi.org/10.2478/jos-2014-0022.

Export Citation

© by Paul Biemer. This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License. BY-NC-ND 3.0

Citing Articles

Here you can find all Crossref-listed publications in which this article is cited. If you would like to receive automatic email messages as soon as this article is cited in other publications, simply activate the “Citation Alert” on the top of this page.

Yingfu Xie, Andreas Lennmalm, Daniel Lennartsson, and Annica de Groote
Statistical Journal of the IAOS, 2017, Page 1
David A. Marker
Statistical Journal of the IAOS, 2017, Page 1
Paul Allin and David J. Hand
International Statistical Review, 2017, Volume 85, Number 2, Page 355

Comments (0)

Please log in or register to comment.
Log in