Skip to content
BY 4.0 license Open Access Published by De Gruyter April 14, 2022

Who Lost the Most? Mathematics Achievement during the COVID-19 Pandemic

Dalit Contini ORCID logo, Maria Laura Di Tommaso ORCID logo, Caterina Muratori, Daniela Piazzalunga ORCID logo and Lucia Schiavon ORCID logo

Abstract

This article estimates the effect of school closures in the spring of 2020 on the math skills of primary school children in Italy, which was the first Western country hit by the COVID-19 pandemic, responding with a strict lockdown and total school closures through the end of the school year. Leveraging unique longitudinal data collected in the province of Torino, a large metropolitan area in northern Italy, we analyse the learning outcomes of two adjacent cohorts of pupils, the pre-Covid and the Covid cohort. The pandemic had a large mean negative impact on pupils’ performance in mathematics (−0.19 standard deviations). Learning loss was greater for girls and for high-achieving children of low-educated parents. Net of individual characteristics, the impact was harshest in schools with a disadvantaged social composition.

JEL Classification: I21; I24

1 Introduction

In a bid to contain the infection rate of the virus during the Covid-19 pandemic, most countries imposed a series of strict lockdown measures. Schools worldwide were closed for several months starting in spring 2020, raising concern over children’s development and the threat of increasing educational inequality.

School closures risk damaging children’s education through the replacement of regular school activities with distance learning – deemed less effective than in-person instruction, and dependent on parental involvement and the availability of digital devices at home – and because of dramatic changes in children’s peer interactions (Agostinelli et al. 2022; Andrew et al. 2020). Many pupils have also had to cope with parental job loss, disruptions in social ties, lack of after-school activities, crowded dwellings, illness or death of relatives, isolation, and stress related to the pandemic (Bacher-Hicks and Goodman 2021).

The effect of Covid-19 and school closures on pupils’ achievements has been investigated in a handful of papers on Anglo-Saxon and Western European countries. Most available studies compare achievement of a cohort exposed to the school closure and previous cohorts using cross-sectional data. Instead, in this paper, we take advantage of unique longitudinal data that we collected in the province of Torino, a large metropolitan area in northern Italy, to evaluate the impact of the pandemic during spring 2020 on the mathematics achievement of primary school pupils. Individual-level longitudinal data allows to account for possible pre-existing differences across cohorts, mitigating concerns of bias (Engzell, Frey, and Verhagen 2021; Werner and Woessmann 2021).

Existing studies report declining achievement and greater educational losses for disadvantaged children. However, the impact may vary across societies, school systems, and measures adopted to contain the pandemic. Italy is a case of particular interest, because it was the first European country to experience the outbreak and rapid transmission of the virus, and the staggering number of infections completely upended the lives of children and their families. The lockdown was accompanied by strict social distancing measures and the closure of business and service activities, with severe repercussions on employment. Italy experienced one of the longest periods of school closures in Europe (15 weeks against a European average of 10), while having a low degree of technological preparedness to support distance learning (European Commission 2020a). Teachers had low ICT skills and little experience with technology-enhanced teaching (OECD 2018).[1]

2 Data Description and Empirical Strategy

We compare the learning progress over one school year made by two adjacent cohorts: the ‘pre-Covid cohort’, made up of children who attended both grades 2 and 3 before the pandemic, and the ‘Covid cohort’, made up of children who attended grade 2 before the pandemic and grade 3 during the pandemic. The latter were exposed to distance learning instead of in-person classroom lessons from February 2020 until the end of the school year.[2]

We construct a unique longitudinal dataset linking data from the national assessment of children’s skills (INVALSI data)[3] – which includes standardised tests in math and Italian administered at the end of grade 2 (the pre-test), teachers’ marks, and socio-demographic variables – to the results from a novel standardised assessment administered by the research team at the end of grade 3 (the post-test).[4] Both cohorts took the pre-test at the end of grade 2, and the post-test approximately one year later. Children in the pre-Covid cohort sat the test at the end of grade 3 (at the end of April 2019). Because of school closures, children in the Covid cohort took the test at the start of the following school year (at the beginning of October 2020).

Data were collected for 2188 children attending primary schools where the pre-Covid cohort had participated in a randomized controlled trial on active learning math instruction (Di Tommaso et al. 2021). Since that intervention proved to benefit only girls, treated girls who had participated in the project were excluded from our main analyses to rule out possible confounding effects. The final sample contains 1539 children (Appendix B, Table B1 presents the descriptive statistics of the sample before dropping treated girls).

The effects of the Covid-19 pandemic on the math achievements of children were estimated with a difference-in-difference strategy fully exploiting the longitudinal nature of the data (Contini and Cugnata 2020):

(1) Y 1 i k j = β 0 + β 1 C k + β 2 Y 0 i k j + β 3 X i k j + β 4 D j + e i k j

where Y 1ikj is the post-test taken by child i of cohort k in school j; C is a dummy variable indicating the Covid cohort; Y 0 is the array of initial skills in grade 2; X is a vector of sociodemographic variables and D is the school dummy vector. Errors are clustered at the class level. The identifying assumption is that – conditional on the skills displayed in grade 2 – the math performance of children in grade 3 would have been the same in the two cohorts, had the pandemic not occurred.

3 Results and Discussion

Overall, the pandemic negatively affected children’s math skills, with an estimated average loss of 0.19 standard deviations in test scores (Table 1), which corresponds to the learning typically occurring in 3 months of school (Bloom, Black, and Lipsey 2008).[5] , [6] Assuming normality of the distribution, the average impact can be viewed as a downward shift in the children’s test score distribution of 4–5 percentile points. The result does not change much when controlling for class-level variables instead of school fixed effects (Appendix B, Table B2).

Table 1:

Effects of Covid-19 on children’s math achievements.

Overall Overall Low-edu parents High-edu parents Low-edu parents High-edu parents
Math score Math score Math score Math score Math score Math score
(1) (2) (3) (4) (5) (6)
Covid cohort −0.188*** −0.167*** −0.198*** −0.164** −0.133* −0.201**
(0.053) (0.056) (0.065) (0.073) (0.070) (0.085)
Female −0.226*** −0.189*** −0.215*** −0.240*** −0.105 −0.308***
(0.031) (0.055) (0.041) (0.066) (0.080) (0.080)
Covid cohort * female −0.056 −0.164* 0.110
(0.067) (0.092) (0.126)
Observations 1539 1539 1038 501 1038 501
R-squared 0.575 0.575 0.585 0.523 0.586 0.524
Initial abilities Yes Yes Yes Yes Yes Yes
Socio-demographic controls Yes Yes Yes Yes Yes Yes
School fixed effects Yes Yes Yes Yes Yes Yes

  1. Standardised math post-test score. Initial abilities: math and Italian standardised test scores in grade 2, teacher-assigned mark in math in grade 2. Socio-demographic controls: high-educated parents (at least one parent with a tertiary degree) and migratory background. Clustered standard errors at class level in parentheses. ***p < 0.01, **p < 0.05, *p < 0.1.

However, not all the children seem to have been affected equally. When we include classroom-level interaction terms in the model, we observe an increase in educational inequality across socioeconomic backgrounds, as children in schools where few parents hold a university degree suffered the greatest loss (up to 0.3 SD, Figure 1). This could be because better teachers may self-select into more advantaged schools (Barbieri, Rossetti, and Sestito 2011) or because teachers in advantaged schools were operating in an environment more conducive to benefiting from distance learning. Similar results were found in Maldonado and De Witte (2022).

Figure 1: 
Effects of Covid-19 on math achievements by the proportion of high-educated parents in the class.
In the model we control for context variables at the class level and not for school fixed effects. Confidence intervals at 95% based on standard errors clustered at the class level. The results are based on a parametric estimate of model (1), with the addition of an interaction term between the Covid cohort and the percentage of high-educated parents in the class. Full results in Appendix B, Table B2.

Figure 1:

Effects of Covid-19 on math achievements by the proportion of high-educated parents in the class.

In the model we control for context variables at the class level and not for school fixed effects. Confidence intervals at 95% based on standard errors clustered at the class level. The results are based on a parametric estimate of model (1), with the addition of an interaction term between the Covid cohort and the percentage of high-educated parents in the class. Full results in Appendix B, Table B2.

Having controlled for class composition, no significant differences appeared at the individual level between children of high and low-educated parents (Table 1). Instead, we find heterogeneity within children with low-educated parents. First, girls experienced a significantly greater loss than boys (Table 1).[7] This result is particularly alarming if we consider that even in ordinary times girls do worse than boys in math in Italy (Contini, Di Tommaso, and Mendolia 2017). One possible explanation for the increasing gender gap is that parents are aware that boys tend to spend less time doing schoolwork and try to compensate by providing additional help (Del Bono et al. 2021). Another explanation is that gender norms are even more influential when school is closed, particularly among children of low-educated parents. Second, among pupils from disadvantaged backgrounds, the ones with the most severe learning loss were those who scored highest on math tests in second grade (Figure 2). Thus, school closure speeds up the process highlighted in Crawford, Macmillan, and Vignoles (2017) of increasing inequality among high-achieving children from different social backgrounds.

Figure 2: 
Effects of Covid-19 on math achievements by initial math skills, by parental education.
Low-educated parents: no parent has a tertiary degree. High-educated parents: at least one parent has a tertiary degree. Confidence intervals at 95% based on standard errors clustered at the class level. The results are based on a parametric estimate of model (1), with the addition of an interaction term between the Covid cohort and the Math test score in grade 2, separately for children with low- and high-educated parents. Full results in Appendix B, Table B2.

Figure 2:

Effects of Covid-19 on math achievements by initial math skills, by parental education.

Low-educated parents: no parent has a tertiary degree. High-educated parents: at least one parent has a tertiary degree. Confidence intervals at 95% based on standard errors clustered at the class level. The results are based on a parametric estimate of model (1), with the addition of an interaction term between the Covid cohort and the Math test score in grade 2, separately for children with low- and high-educated parents. Full results in Appendix B, Table B2.

The schools in our analytic sample are more advantaged in terms of socioeconomic composition than others at the regional and national levels (Appendix B, Table B4) and the provision of digital technology is higher in the North than in the South of Italy (Istat 2021). Thus, we expect the negative effects of the pandemic on pupil achievement at the national level to be even greater, with harsher consequences on children’s skills and on inequalities across socio-demographic groups.

4 Conclusions

Italian children faced large learning losses in mathematics resulting from the Covid-19 pandemic and the school closures in the spring of 2020. The pandemic deepened existing inequalities between socio-economic groups, as children attending schools with lower shares of high-educated parents suffered a greater loss. Among children with low-educated parents, the learning loss was greater for those with higher prior mathematical skills; moreover, the loss suffered by girls was double that of boys. If we add to this the possible effects of other disruptions related to Covid-19 pandemic during the following school years and the cumulative effects that these initial losses could develop over time, we can expect dramatic long-term consequences for an entire generation of young people (Cunha et al. 2006). A reduction of about one third of the usual learning gains during grade 3 could yield a loss up to a full year of school by grade 10 (Kaffenberger 2021).

These findings call for urgent policy action. On the one hand, the education system must be given the necessary tools to face possible future crises. On the other hand, remedial measures should be introduced to limit the damage that has already occurred, supporting pupils at high risk of being left behind and encouraging the learning of well-performing children, especially from disadvantaged social backgrounds.


Corresponding author: Daniela Piazzalunga, University of Trento, Trento, Italy; IZA, Bonn, Germany; and CHILD-Collegio Carlo Alberto, Torino, Italy, E-mail:

Funding source: Collegio Carlo Alberto

Funding source: Università degli Studi di Torino

Funding source: Ministero dell'Università e della Ricerca, Italia (FISR 2020 COVID)

Award Identifier / Grant number: FISR2020IP_02236

Acknowledgments

The Online Appendix contains a detailed description of data, descriptive and robustness analysis. We thank the Piedmont Regional Board of Education for their support and INVALSI for data and fruitful collaboration. We also appreciate the valuable contribution of Francesca Ferrara, Giulia Ferrari, and Ornella Robutti, and thank the tutors, principals and teachers involved in the project.

  1. Author contribution: None.

  2. Research Funding: This study was funded by the financial support of the Collegio Carlo Alberto,Università degli Studi diTorino and Ministero dell'Università e della Ricerca, Italia (FISR 2020 COVID) (grant FISR 2020COVID), FISR2020IP_02236.

  3. Competing interests: None.

References

Agostinelli, F., M. Doepke, G. Sorrenti, and F. Zilibotti. 2022. “When the Great Equalizer Shut Down: Schools, Peers, and Parents in Pandemic Times.” Journal of Public Economics: 206: 104574. https://doi.org/10.1016/j.jpubeco.2021.104574.Search in Google Scholar

Andrew, A., S. Cattan, M. Costa-Dias, C. Farquharson, L. Kraftman, S. Krutikova, and A. Sevilla. 2020. “Inequalities in Children’s Experiences of Home Learning during the COVID-19 Lockdown in England.” Fiscal Studies 41 (3): 653–83. https://doi.org/10.1111/1475-5890.12240.Search in Google Scholar

Bacher-Hicks, A., and J. Goodman. 2021. “The Covid-19 Pandemic Is a Lousy Natural Experiment for Studying the Effects of Online Learning.” Education Next 21 (4): 38–42.Search in Google Scholar

Barbieri, G., C. Rossetti, and P. Sestito. 2011. “The Determinants of Teacher Mobility: Evidence Using Italian Teachers’ Transfer Applications.” Economics of Education Review 30 (6): 1430–44. https://doi.org/10.1016/j.econedurev.2011.07.010.Search in Google Scholar

Bloom, H. S., A. R. Black, and M. W. Lipsey. 2008. “Performance Trajectories and Performance Gaps as Achievement Effect-Size Benchmarks for Educational Interventions.” Journal of Research on Educational Effectiveness 1 (4): 289–328. https://doi.org/10.1080/19345740802400072.Search in Google Scholar

Contini, D., and F. Cugnata. 2020. “Does Early Tracking Affect Learning Inequalities? Revisiting Difference-In-Differences Modeling Strategies with International Assessments.” Large-scale Assessments in Education 8 (1): 1–27. https://doi.org/10.1186/s40536-020-00094-x.Search in Google Scholar

Contini, D., M. L. Di Tommaso, and S. Mendolia. 2017. “The Gender Gap in Mathematics Achievement: Evidence from Italian Data.” Economics of Education Review 58: 32–42. https://doi.org/10.1016/j.econedurev.2017.03.001.Search in Google Scholar

Crawford, C., L. Macmillan, and A. Vignoles. 2017. “When and Why Do Initially High-Achieving Poor Children Fall behind?” Oxford Review of Education 43 (1): 88–108. https://doi.org/10.1080/03054985.2016.1240672.Search in Google Scholar

Cunha, F., J. J. Heckman, L. Lochner, and D. V. Masterov. 2006. “Interpreting the Evidence on Life Cycle Skill Formation.” In Handbook of the Economics of Education, vol. 1, 697–81.Search in Google Scholar

Del Bono, E., L. Fumagalli, A. Holford, and B. Rabe. 2021. Coping with School Closures: The Changing Responses of Schools, Parents and Children during COVID-19. ISER Report. Also available at https://www.iser.essex.ac.uk/files/news/2021/little-inequality-homeschool/coping-with-school-closures.pdf.Search in Google Scholar

Di Tommaso, M., D. Contini, D. De Rosa, F. Ferrara, D. Piazzalunga, and O. Robutti. 2021. Tackling the Gender Gap in Mathematics with Active Learning Methodologies. Bonn: IZA DP 14572.Search in Google Scholar

Engzell, P., A. Frey, and D. M. Verhagen. 2021. “Learning Loss Due to School Closures during the COVID-19 Pandemic.” Proceedings of the National Academy of Sciences 118 (17): e2022376118. https://doi.org/10.1073/pnas.2022376118.Search in Google Scholar

European Commission 2020a. The Digital Economy and Society Index (DESI). Thematic chapter. Also available at https://digital-strategy.ec.europa.eu/en/policies/desi.Search in Google Scholar

European Commission 2020b. The Digital Economy and Society Index (DESI). Italy. Also available at https://ec.europa.eu/newsroom/dae/document.cfm?doc_id=66918.Search in Google Scholar

Istat 2021. Internet: accesso e tipo di utilizzo. Roma: Istituto nazionale di statistica. Also available at dati.istat.it.Search in Google Scholar

Kaffenberger, M. 2021. “Modelling the Long-Run Learning Impact of the Covid-19 Learning Shock: Actions to (More Than) Mitigate Loss.” International Journal of Educational Development 81: 102326. https://doi.org/10.1016/j.ijedudev.2020.102326.Search in Google Scholar

Maldonado, J. E., and K. De Witte. 2022. “The Effect of School Closures on Standardised Student Test Outcomes.” British Educational Research Journal 48: 49–94. https://doi.org/10.1002/berj.3754.Search in Google Scholar

OECD 2018. TALIS 2018 Results (Volume I). Teachers and School Leaders as Lifelong Learners. TALIS. Paris: OECD Publishing.Search in Google Scholar

Sloan McCombs, J., C. H. Augustine, H. L. Schwartz, S. J. Bodilly, B. McInnis, D. S. Lichter, and A. Brown Cross. 2011. Making Summer Count. How Summer Programs Can Boost Children’s Learning. Santa Monica, CA: Rand Education.Search in Google Scholar

Werner, K., and L. Woessmann. 2021. The Legacy of COVID-19 in Education. Bonn: IZA DP 14796.Search in Google Scholar


Supplementary Material

The online version of this article offers supplementary material (https://doi.org/10.1515/bejeap-2021-0447).


Received: 2021-12-07
Accepted: 2022-03-28
Published Online: 2022-04-14

© 2022 Dalit Contini et al., published by De Gruyter, Berlin/Boston

This work is licensed under the Creative Commons Attribution 4.0 International License.