Skip to content
BY 4.0 license Open Access Published by De Gruyter Open Access December 31, 2022

Content removal bias in web scraped data: A solution applied to real estate ads

  • Gabriele Marconi EMAIL logo
From the journal Open Economics

Abstract

I propose a solution to content removal bias in statistics from web scraped data. Content removal bias occurs when data is removed from the web before a scraper is able to collect it. The solution I propose is based on inverse probability weights, derived from the parameters of a survival function with complex forms of data censoring. I apply this solution to the calculation of the proportion of newly built dwellings with web scraped data on Luxembourg, and I run a counterfactual experiment and a Montecarlo simulation to confirm the findings. The results show that the extent of content removal bias is relatively small if the scraping occurs frequently compared with the online permanence of the data; and that it grows larger with less frequent scraping.

References

Ascheri A., Kiss Nagy A., Marconi G., Meszaros M., Paulino R., Reis F. (2021). Competition in urban hiring markets: evidence from online job advertisements. Eurostat Statistical Working Papers, http://dx.doi.org/10.2785/667004Search in Google Scholar

Beblavý, M., Fabo, B. and Lenaerts, K. (2016). Demand for Digital Skills in the US Labour Market: The IT Skills Pyramid. CEPS Special Report No. 154, Available at SSRN: https://ssrn.com/abstract=3047102Search in Google Scholar

Brüning, N. and Mangeol, P. (2020). What skills do employers seek in graduates? Using online job posting data to support policy and practice in higher education. OECD Education Working Papers, No. 231, https://doi.org/10.1787/bf533d35-en10.1787/bf533d35-enSearch in Google Scholar

Bricongne, J., Meunier, B. and Sylvain, P. (2021). Web Scraping Housing Prices in Real-time: the Covid-19 Crisis in the UK. Banque de France Working Paper No. 827, http://dx.doi.org/10.2139/ssrn.391619610.2139/ssrn.3916196Search in Google Scholar

Cole S., Hernán M. (2008). Constructing inverse probability weights for marginal structural models. American Journal of Epidemiology, 168, 656-64 doi: 10.1093/aje/kwn164.10.1093/aje/kwn164Search in Google Scholar

Cole, S. and Hernán, M. (2004). Adjusted survival curves with inverse probability weights. Computer Methods and Programs in Biomedicine, 75, 45-49, https://doi.org/10.1016/j.cmpb.2003.10.00410.1016/j.cmpb.2003.10.004Search in Google Scholar

Cox, D., Kartsonaki, C., & Keoghc, R. (2008). Big data: Some statistical issues. Statistics & Probability Letters, 136, 111-115, https://doi.org/10.1016/j.spl.2018.02.01510.1016/j.spl.2018.02.015Search in Google Scholar

Emmert-Streib, F., and Dehmer, M. (2019). Introduction to Survival Analysis in Practice. Machine Learning and Knowledge Extraction, 1, 1013-1038, https://doi.org/10.3390/make103005810.3390/make1030058Search in Google Scholar

Fabo, B. and Kurekova-Mytna, L. (2022). Forthcoming.Search in Google Scholar

Eiglsperger, M. (2019). New features in the Harmonised Index of Consumer Prices: analytical groups, scanner data and web-scraping. Economic Bulletin Boxes, 2, https://www.ecb.europa.eu/pub/economic-bulletin/html/eb201902~a070c3a338.en.htmlSearch in Google Scholar

Han, S. and Anderson, C. (2021). Web scraping for hospitality research: Overview, opportunities, and implications. Cornell Hospitality Quarterly, 62, 89-104, https://doi.org/10.1177/193896552097358710.1177/1938965520973587Search in Google Scholar

Head, A., Lloyd-Ellis, H. and Sun, H. (2014). Search, Liquidity, and the Dynamics of House Prices and Construction. American Economic Review, 104, 1172-1210, https://doi.org/10.1257/aer.104.4.117210.1257/aer.104.4.1172Search in Google Scholar

Lee, E. and Wang, J. (2003). Statistical methods for survival data analysis. John Wiley & Sons.10.1002/0471458546Search in Google Scholar

Observatoire de l’habitat du ministère du Logement (2021). Simulateur des prix. Accessed on 05 February 2022 at https://logement.public.lu/fr/observatoire-habitat/prix-de-vente/simulateur.htmlSearch in Google Scholar

OECD (2021), OECD Skills Outlook 2021: Learning for Life. OECD Publishing, https://doi.org/10.1787/0ae365b4-en10.1787/0ae365b4-enSearch in Google Scholar

R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, https://www.R-project.org/Search in Google Scholar

Seaman S. and White I. (2013). Review of inverse probability weighting for dealing with missing data. Statistical Methods in Medical Research, 22, 278-295, https://doi.org/10.1177/096228021039574010.1177/0962280210395740Search in Google Scholar

Statec – Institut National de la Statistique et des Etudes Economiques du Grand-Duché du Luxembourg (2013). Note méthodologique sur la statistique des prix de vente des appartements. Observatoire de l’habitat du ministère du Logement, https://logement.public.lu/dam-assets/documents/observatoire-habitat/Methodologie-PrixVente-Appartements-2013-12.pdfSearch in Google Scholar

Whitaker, S. (2018). Big Data versus a survey. The Quarterly Review of Economics and Finance, 67, 285-296, https://doi:10.1016/j.qref.2017.07.01110.1016/j.qref.2017.07.011Search in Google Scholar

Zhang, Z., and Sun, J. (2010). Interval censoring. Statistical methods in medical research, 19, 53–70. https://doi.org/10.1177/096228020910502310.1177/0962280209105023Search in Google Scholar

Zhelyazkova, N., and Ritschard, G. (2018). Parental leave take-up of fathers in Luxembourg. Population Research and Policy Review, 37, 769-793, https://doi.org/10.1007/s11113-018-9470-810.1007/s11113-018-9470-8Search in Google Scholar

Received: 2022-02-25
Accepted: 2022-11-10
Published Online: 2022-12-31
Published in Print: 2022-01-01

© 2022 Gabriele Marconi, published by De Gruyter

This work is licensed under the Creative Commons Attribution 4.0 International License.

Downloaded on 26.2.2024 from https://www.degruyter.com/document/doi/10.1515/openec-2022-0119/html
Scroll to top button