Skip to content
BY 4.0 license Open Access Published by De Gruyter Open Access December 31, 2022

Content removal bias in web scraped data: A solution applied to real estate ads

  • Gabriele Marconi EMAIL logo
From the journal Open Economics


I propose a solution to content removal bias in statistics from web scraped data. Content removal bias occurs when data is removed from the web before a scraper is able to collect it. The solution I propose is based on inverse probability weights, derived from the parameters of a survival function with complex forms of data censoring. I apply this solution to the calculation of the proportion of newly built dwellings with web scraped data on Luxembourg, and I run a counterfactual experiment and a Montecarlo simulation to confirm the findings. The results show that the extent of content removal bias is relatively small if the scraping occurs frequently compared with the online permanence of the data; and that it grows larger with less frequent scraping.


Ascheri A., Kiss Nagy A., Marconi G., Meszaros M., Paulino R., Reis F. (2021). Competition in urban hiring markets: evidence from online job advertisements. Eurostat Statistical Working Papers, in Google Scholar

Beblavý, M., Fabo, B. and Lenaerts, K. (2016). Demand for Digital Skills in the US Labour Market: The IT Skills Pyramid. CEPS Special Report No. 154, Available at SSRN: in Google Scholar

Brüning, N. and Mangeol, P. (2020). What skills do employers seek in graduates? Using online job posting data to support policy and practice in higher education. OECD Education Working Papers, No. 231, in Google Scholar

Bricongne, J., Meunier, B. and Sylvain, P. (2021). Web Scraping Housing Prices in Real-time: the Covid-19 Crisis in the UK. Banque de France Working Paper No. 827, in Google Scholar

Cole S., Hernán M. (2008). Constructing inverse probability weights for marginal structural models. American Journal of Epidemiology, 168, 656-64 doi: 10.1093/aje/kwn164.10.1093/aje/kwn164Search in Google Scholar

Cole, S. and Hernán, M. (2004). Adjusted survival curves with inverse probability weights. Computer Methods and Programs in Biomedicine, 75, 45-49, in Google Scholar

Cox, D., Kartsonaki, C., & Keoghc, R. (2008). Big data: Some statistical issues. Statistics & Probability Letters, 136, 111-115, in Google Scholar

Emmert-Streib, F., and Dehmer, M. (2019). Introduction to Survival Analysis in Practice. Machine Learning and Knowledge Extraction, 1, 1013-1038, in Google Scholar

Fabo, B. and Kurekova-Mytna, L. (2022). Forthcoming.Search in Google Scholar

Eiglsperger, M. (2019). New features in the Harmonised Index of Consumer Prices: analytical groups, scanner data and web-scraping. Economic Bulletin Boxes, 2, in Google Scholar

Han, S. and Anderson, C. (2021). Web scraping for hospitality research: Overview, opportunities, and implications. Cornell Hospitality Quarterly, 62, 89-104, in Google Scholar

Head, A., Lloyd-Ellis, H. and Sun, H. (2014). Search, Liquidity, and the Dynamics of House Prices and Construction. American Economic Review, 104, 1172-1210, in Google Scholar

Lee, E. and Wang, J. (2003). Statistical methods for survival data analysis. John Wiley & Sons.10.1002/0471458546Search in Google Scholar

Observatoire de l’habitat du ministère du Logement (2021). Simulateur des prix. Accessed on 05 February 2022 at in Google Scholar

OECD (2021), OECD Skills Outlook 2021: Learning for Life. OECD Publishing, in Google Scholar

R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, in Google Scholar

Seaman S. and White I. (2013). Review of inverse probability weighting for dealing with missing data. Statistical Methods in Medical Research, 22, 278-295, in Google Scholar

Statec – Institut National de la Statistique et des Etudes Economiques du Grand-Duché du Luxembourg (2013). Note méthodologique sur la statistique des prix de vente des appartements. Observatoire de l’habitat du ministère du Logement, in Google Scholar

Whitaker, S. (2018). Big Data versus a survey. The Quarterly Review of Economics and Finance, 67, 285-296, https://doi:10.1016/j.qref.2017.07.01110.1016/j.qref.2017.07.011Search in Google Scholar

Zhang, Z., and Sun, J. (2010). Interval censoring. Statistical methods in medical research, 19, 53–70. in Google Scholar

Zhelyazkova, N., and Ritschard, G. (2018). Parental leave take-up of fathers in Luxembourg. Population Research and Policy Review, 37, 769-793, in Google Scholar

Received: 2022-02-25
Accepted: 2022-11-10
Published Online: 2022-12-31
Published in Print: 2022-01-01

© 2022 Gabriele Marconi, published by De Gruyter

This work is licensed under the Creative Commons Attribution 4.0 International License.

Downloaded on 26.2.2024 from
Scroll to top button