In May 2020, RTI examined early grade reading data from 27 datasets across 8 countries (which we have since updated to 87 datasets from 15 countries). These datasets all contained data for students in consecutive grades (e.g., grades 1 and 2; grades 2 and 3; etc.) and were used as a starting point for us to model the impact of COVID-related school closures on learning loss.
This blog continues and expands on our initial analyses by predicting where students will be once schools have re-opened after the COVID shut down. Via targeted briefs, this work has already helped government officials, stakeholders, and teachers in several countries plan for needed remedial programming and catch-up programs for school re-opening. In order to understand student needs at all levels of performance, we have focused this work on student distributions at restart, rather than average scores (i.e. student means). Distributions help show how widely the students’ abilities vary, which in turn help ministries understand the numbers of students that fall into different reading levels. This is essential for planning successful remediation programs.
....students can be expected to return to school up to a half of a year behind where they were when the school year was interrupted and more than a year behind where they would have been without school closures.
While results vary, we found that for the majority of countries, students can be expected to return to school up to a half of a year behind where they were when the school year was interrupted and more than a year behind where they would have been without school closures. Two things contribute to the severity of this learning loss. First, the abrupt disruption of the school year meant that students missed out on the learning that would have occurred had schooling continued. If the mid-March closing of school occurred in the early part of the school year – as in countries that begin school in January – then a significant amount of learning opportunity was foregone. Second, not being in school, and not having access to alternative learning opportunities (the sad reality for most students), the skills students had will deteriorate, and deteriorate more as time away from school increased.
Students who are the least likely to have the means to learn while at home, and those with the least solidly developed skills, sadly, are the ones who will have regressed the most, as shown in our examples below.
Parameters for Prediction
Our model for predicting student reading ability at the restart of school is based on the following parameters:
- Proportion of the school year completed before shutdown. For example, ‘early disruption’ schools (i.e. those starting around in January or February 2020), had only 1-2 months of school before the COVID shut down. ‘Late disruption’ schools, on the other hand, typically had 5-6 months of school prior to shut down (with their school year beginning around August or September 2019).
- Length of school closures due to COVID (weeks). It should be noted that at the time of publication, the school restart dates have not been finalized in many countries, as not all schools have re-opened. In these cases, predictions are used.
- The known reading distribution at the end of year for the given grades. The actual distributions (real data) of the reading scores at the end of a typical school year.
- The learning annual gain between grades. For example, the amount of growth seen between the end of the year grade 2 and end of the year grade 3 students will tell us what grade 3 annual growth looks like in a typical year.
Looking at Country Specific Data
In this section, we present data from eight different countries: four where the COVID break occurred late in the academic year (i.e. ‘late disruption’) and four where it occurred early in the academic year (i.e. ‘early disruption’). We chose these specific countries for this blog because they vary in the length of school closure and reading ability by grade, while also providing broad geographic representation. Table 1 introduces the countries, and their specific parameters used in our model.
Table 1: School calendar inputs used to help determine where students will be once schools reopen.
Using the school closure information provided in Table 1 along with student performance data from a typical school year, we are able to create distributions of typical learning gains, expected learning attainment at the time of school closures and predicted losses suffered before returning to school. We use Jordan as an example in Figure 1.1-3, to show what these distributions look like and how we can understand the expected student reading ability upon returning to school.
Figures 1.1-1.3: Stepwise explanation of the cumulative distribution graphs.
Figure 1.1 presents the cumulative distributions of students’ oral reading fluency (correct words per minute) based on actual data from a typical school year. The blue line represents end of grade 2 scores, while the black line represents end of grade 3 scores. These data were collected at the same time point and therefore the gap between the two lines represents the collective learning gain of the grade 3 students.
Figure 1.2 includes the same data as Figure 1.1, with the addition of a black dotted line representing the predicted reading ability of grade 3 students at the COVID break. In this case, Jordan completed two-thirds of the academic calendar when schools closed, so the black dotted line is closer to the end of grade 3 scores than it is to the end of grade 2 scores.
Figure 1.3 has the inclusion of a fourth line. The red dotted line represents our model’s predicted student fluency when school reopen this is the distribution, we are most interested in. The gap between the two dotted line represents the learning losses suffered because of school closures. In this example, grade 3 students will have regressed back to where they were when they finished grade 2 (i.e. approximately where they began grade 3 the 2019-20 school year).
In Figure 2, we show these distributions for the late disruption countries.
Figure 2: Learning loss for countries that closed late in the academic year.
Note: School return dates are not been finalized in all countries.
Our model predicts that when late disruption schools return, the students will have regressed to a point somewhere within their same academic year, typically close to the beginning of the school year.
Our model predicts that when late disruption schools return, the students will have regressed to a point somewhere within their same academic year, typically close to the beginning of the school year. For example, Ethiopia and Liberia completed about two-thirds of their academic years but they were not out as long as Jordan (39 and 34 weeks, respectively). Our model predicts when these grade 3 students return, they will look slightly better than when they started grade 3 (the red dotted line is slightly above the blue solid line). Grade 3 students in the Philippines nearly finished their academic year prior to the COVID break. Our model predicts these grade 3 students will have regressed to about the mid-way point of their grade 3 academic year.
Figure 3: Learning loss for countries that closed early in the academic year.
Note: School return dates are not been finalized in all countries.
Because the early disruption schools completed about one-fifth of the academic year, our model predicts that most students will return to school looking more like students in the middle or close to the end of their previous grade.
Because the early disruption schools completed about one-fifth of the academic year, our model predicts that most students will return to school looking more like students in the middle or close to the end of their previous grade. For example, Uganda grade 3 students completed only 17% of the 2020 academic year and are expected to be out of school for 45 weeks. Our model predicts that when they return, they will look more like middle of the year grade 2 students. Grade 3 students in El Salvador will resemble students who have completed about 85% of the second grade. And grade 2 students in Kenya, having been out for 41 weeks, would resemble grade 1 students who have completed about three-quarters of grade 1. It is only in Tanzania where the learning loss looks minimal, given they were out of school for only 17 weeks. Grade 3 students in Tanzania are very close to resembling end of year grade 2 students, (start of grade 3)—much like the late disruption schools, above. However, our model predicts that Tanzania and the other early disruption countries will have an increase in non-readers when compared to the end of the previous grade (i.e. start of the current grade).
Non-Readers
One of the most alarming aspects of our model, is its prediction of the proportion of students who will not able to read a single word when they return to school (i.e. non-readers). We estimate these increases based on the proportion of students who did not attain high enough oral reading fluency scores to ensure a level of reading security. More specifically, students reading fewer than 10 correct words per minute at the time of school closures are estimated to regress back to nonreader status because of extended school closures. In seven of the eight example countries, our model predicts that there will be a larger proportion of grade 3 students not able to read when schools reopen than when schools started the year prior to COVID (as shown in Figure 4, where the gray bars are taller than the blue bars). In those seven countries, the predicted increase in the percent of nonreaders from the time of school closures to school reopening ranged from 14 (Ethiopia) to 38 (Uganda) percentage points (difference between the orange and gray bars). In the Philippines, we predict a much smaller increase in nonreaders (3 percentage points). This is due to the fact that students were nearing the end of grade 3 by the time of school closures and most students were reading with sufficient accuracy and fluency in order to at least maintain some basic reading skills that will allow them to return to school with some reading ability.
Figure 4: Predicted percentage of non-readers at the start of the school year, at school closure, and upon school reopening.
Impact
To date, we have met with stakeholders and/or ministry officials from Cambodia, Ethiopia, Jordan, Kenya, Philippines, and Nepal to discuss the implications of this work. We have presented our model and begun dialogues around how best to address the concerning findings. Ultimately, we are focused on two main messages.
First, students will not be at (or near) the same level of reading ability as when schools closed down in March 2020. When they return to school, they will require an intensified remedial program just to get back to where they were before schools closed. For early disruption countries will require significant remediation efforts to get students back to where they were at the start of the 2020 school year. These remediation efforts should be focused both on the starting point of lessons (i.e. returning to prior material), as well as the pace of lessons moving forward. In other words, it will not be sufficient to simply pick up where things left off, nor is it advisable to continue through the curriculum at a typical year’s pace—due to both the need for additional ‘catch-up’, as well as the impact COVID will likely continue to have on teacher and student psychosocial well-being upon returning to school.
Second, addressing the increased proportion of nonreaders and those at the lower end of the learning distribution should be a particular focus of all systems. These students are predicted to have suffered the most significant losses during school closures (due in part to their lack of access to the same learning opportunities typically available to less disadvantaged students). This has served to further widen the gap between lower and higher performing students. If this is not addressed at school restart, these students are at risk of continuing to fall further and further behind their peers.
Next Steps
We will continue to meet with donors, government officials and stakeholders to share results for their respective countries.
As schools reopen, we hope to collect data as soon as possible to validate our predictions with real data. We welcome all collaboration with interested parties. If you have learning loss data and are interested in collaborating, or if you have any questions, please email the authors.
This blog was co-authored by Christopher Cummiskey, Jonathan Stern, and Joe DeStefano.
Blog Series:
Blog #1: Calculating the Educational Impact of COVID-19: Closed Schools, Lost learning, Unequal Impact
