Learning at Scale Interim Report

The Learning at Scale study was designed to identify existing early grade reading programs with demonstrated impact on basic skills at scale and to conduct in-depth investigations of these programs to determine what makes them successful. After an extensive search, eight programs (spanning seven countries) were selected for inclusion in the study. Research on these programs has been conducted in order to answer the three overarching research questions, focused on understanding the components of instructional practices, instructional supports, and system supports that lead to effective instruction. Learning at Scale data collection activities for some of these programs were delayed due to COVID-19. However, with demand for information about how to implement effective interventions at large scale at an all-time high, we believe that the timely sharing of findings from Learning at Scale is essential. Accordingly, this interim report provides preliminary findings from our study to date, highlighting key high-level findings across all eight programs, as well as quantitative and qualitative findings from primary research for select programs. The Learning at Scale study is led by RTI International, as part of the Center for Global Development (CGD) education research consortium, funded by the Bill and Melinda Gates Foundation.

Understanding the Social Classroom: The basis of effective pedagogy?

Literacy instruction programs have arguably had limited success because they focus on the technical – but not the social – aspects of literacy instruction. Reform efforts in sub-Saharan Africa have regularly failed to shift pedagogy away from teacher-led whole-class direct instruction to activities that are more effective for learning. In part, the failure is due to a lack of recognition of the social nature of classrooms where teacher-child interactions are conditioned by cultural predispositions. New research from Tanzania identified such challenges to pedagogical reform and points to potential solutions. One approach focuses on the child - to develop their social and emotional competencies. Teachers in Mtwara, Tanzania - but not parents – think that confidence and curiosity are important for student learning and report that interactive teaching activities are less effective in rural areas where students lack these competencies. Evidence suggests that building students’ confidence to participate in class is achievable relatively quickly. A second approach is to adapt teaching activities. Teachers in Tanzania report reluctance to implement teaching activities that undermine the social goals of instruction, such as avoiding embarrassment and promoting a sense of fairness and togetherness in the classroom. Instruction would be more effective if activities are co-designed with teachers to achieve both the social goals and the cognitive/learning goals of teaching.

Is It Possible to Improve Learning at Scale? Reflections on the Process of Identifying Large-Scale Successful Education Interventions

Improving learning outcomes at scale is hard. That may seem obvious, but only recently have policymakers and donors become aware of just how dire—and broad—the learning crisis is. Most of their efforts to improve learning have been pilot programs, and although in some cases it has been possible to improve outcomes at this small scale, it is an entirely different challenge at scale, which can involve thousands of schools—the level at which change must happen to fix the crisis.

Contextualizing the goals of social and emotional learning curricula and materials

Programs to promote social and emotional learning (SEL) risk making assumptions about the global relevance of core competences. Because scholarship is lacking about SEL in many parts of the world, new approaches are needed to contextualize the goals of SEL programs in a realistic time frame. Previous work in anthropology and developmental psychology can help us predict which competences are likely to be valued, given the sociodemographic characteristics of a society. In rural environments where subsistence agriculture is common, for example, communities are likely to value social responsibility, respect, and obedience. Attention should look beyond the needs of the here and now, however, to speculate what competences today’s children will require in the future. Looking at the current variation of competences within a society—for example, the values that teachers, but not parents, place on confidence and curiosity—can help identify immediate pathways for developing new competences. In all of these considerations, the goals of an SEL program must be negotiated with the communities themselves in order to ensure relevance, effectiveness, and acceptance. The hope is that such considerations can help prevent global homogenization of SEL programs, instead ensuring that they genuinely meet the needs of the communities they aim to serve.

Effective pedagogy in cultural context: Preaching to the introverted [CIES 2019 Presentation]

Several forces are at play in determining whether pedagogical approaches are optimally adapted to the culture of children’s behaviour and of teacher-child interaction in the classroom. Teachers’ expectations for children’s behaviour may differ from the way in which children are raised at home (Jukes et al, 2018). Teaching activities may be designed by experts from outside the beneficiary education system or who may be removed from the culture of rural schools. The evidence for the effectiveness of recommended teaching practices may be based on children and teachers from a different culture. The current study sought to understand the role of cultural factors in the teaching of early grade reading in Tanzania. The aim was to investigate teacher’s pedagogical choices and the implicit theory of teaching and learning that underpinned these choices. We also aimed to understand teachers’ perceptions of students’ social and emotional skills and how this influenced their pedagogical choices. The current study took place in three regions of Tanzania – Zanzibar, Mtwara and Iringa – in the context of the USAID supported Tusome Pamoja (“let’s read together”) project. Researchers observed one lesson from each of 36 teachers and recorded key teaching activities. A subsequent qualitative interview with the teachers examined the decisions they made during the lesson and how their perception of students’ competencies influenced their decisions. Two themes emerged from the results. First, teachers said that children who did not participate in classroom activities lacked confidence and curiosity, particularly in rural areas. These two qualities - confidence and curiosity - were identified in previous work as traits valued more by teachers than parents in the context of learning. Second, teachers said the independent activities were not effective because pupils always cooperate with others in life. There was a strong preference for activities that involved the whole class and against students learning by doing, independent of the teacher. Schools can respond by conducting more activities that build curiosity and confidence. Children becoming more confident and curious in rural Africa represents a cultural change. But one that emerges from the community - not imposed upon it. Instructional design should consider the strong cultural preference for group-orientation and work with - rather than against - this preference where possible.

A rational approach to evidence-based decision making in education [CIES 2019 Presentation]

There is growing demand for policy based on rigorous evidence. Many consider the strongest evidence to come from studies that identify causality with high internal validity - such as RCTs - and systematic reviews of these studies. If policies are based strictly on such rigorous evidence there is a risk of bias towards simple, discrete, measurable interventions and away from complex interventions. Rigorous evidence is also better suited to some questions than others. Evaluations may provide stronger conclusions about impact than about the mechanisms, implementation, context, generalisability and scaling of interventions. For these reasons, policy-making does – and should – consider issues for which there is no conclusive evidence. However, there is little guidance as to how and when such inconclusive evidence can be used. We present a framework for considering inconclusive evidence applied to examples from evidence-based education in low- and middle-income countries. The framework involves a systematic consideration of the estimated costs, benefits and potential harm of a policy, along with the uncertainty in those estimates. This analysis is conducted using standard decision theory and an examination of the utility of policies. We argue that it is rational to pursue a policy with uncertain outcomes if there is a reasonable probability of large positive utility (compared to the cost of the intervention) and a low probability of negative utility. The decision to act under uncertainty is influenced by a number of other considerations including: the potential to improve the evidence base, the urgency of the decision and the analysis of alternative options. The framework also calls for systematic analysis of uncertainty associated with all components of a policy decision. For example, some interventions may have robust evidence of impact but considerable uncertainty associated with the generalisability of the evidence to a new context, or with the scalability of the intervention. We discuss our approach to measuring and reducing uncertainty in policy decisions and its implications for evaluation and research. The overall aim of this work is to make evidence-based decision-making more effective and applicable to a wider range of problems.

Benchmarks for Early Grade Reading Skills in West Bank Policy Brief

The Ministry of Education and Higher Education (MOEHE) conducted the first EGRA assessment in the West Bank in March 2014 among a nationally representative sample of Grade 2 students, followed by a benchmarking exercise in September 2014. In 2018, the Early Grade Reading (EGR) project conducted a project baseline using an adapted EGRA 1 and MELQO 2 3 . Following this assessment, the MOEHE expressed interest in revising the 2014 provisional Grade 2 benchmarks and developing Grade 1 benchmarks. EGR conducted a technical benchmarking workshop in November 2018.

Measuring social and emotional learning of young children in Tanzania

CIES 2018 Presentation, given by Matthew Jukes. There is an increased demand for assessments of social and emotional competencies of young children in low- and middle-income countries. These competencies are increasingly seen as important for children’s development and for their education. In the context of preschool and primary education, such assessments have a number of uses. They are used to evaluate the impact of programs on children’s social and emotional learning. They can also be used to monitoring individual children’s progress in such programs and to tailor interventions to their needs. We developed a tool to assess aspects of SEL that are important for children’s education in Tanzania. Using this work as a case study, we describe the challenges inherent in developing such a tool. The tool was developed as part of the USAID Tusome Pamoja preschool program in Mtwara, Southern Tanzania.

Making evidence-based decisions: the illusory quest for rigour and certainty- CIES 2018 Presentation

CIES 2018 Presentation, given by Matthew Jukes. Randomized controlled trials (RCTs) have been used increasingly in an effort to reduce uncertainty and improve rigour of impact estimates. However, RCTs do not provide certainty in other important aspects of the evaluation, such as its external validity and in understanding which components of a complex program are essential for its effectiveness. This paper reviews two additional approaches which are currently underutilized which may help researchers and policy makers make better use of uncertain evidence.

“I failed, no matter how hard I tried”: A mixed-methods study of the role of achievement in primary school dropout in rural Kenya.

Article published in the International Journal of Education and Development, Volume 50. From Journal abstract: "Initial access to school is nearly universal in Kenya, but many children who enroll drop out before completing primary school. In this mixed-methods study, we use quantitative data from a randomized control trial involving 2666 upper primary-grade students, as well as qualitative data from interviews with 41 schoolchildren, dropouts, and parents, to examine dropout. Poorer baseline performance on literacy and numeracy assessments predicted a higher risk of dropout. Interviews revealed that children are the primary decision-makers rather than parents. Together, these findings suggest that school quality interventions may be an effective means of reducing primary school dropout in this region."