Title: Right-fit EdTech: Leveraging Loquat and other (machine) learning to support foundational learning in LMICs Abstract The COVID-19 pandemic has severely disrupted global education systems: according to the Global Education Monitoring Team, over 1.6 billion students have lost significant instructional time, with many yet to return to school. But even before the pandemic struck, the state of education around the world was so poor that the World Bank was decrying a “learning crisis.” To address the crisis, the Bank called for both a renewed emphasis on what teachers actually do in a classroom and the strategic deployment of technology to improve teaching and learning. During the pandemic, remote-first instruction became a feature of basic, secondary, and tertiary education systems in many high-income countries. [European Commission, Rickles et al. 2020, Hodges et al. 2020, Gillis & Krull 2020] Coverage was not universal, as wealth and income inequality within high-income countries remains a persistent challenge. [Romm 2020, Blaskó & Schnepf 2020] Nonetheless, the market for educational technology firms boomed during the pandemic, with investment in the sector expected to exceed $80 billion by the end of the decade. [Gillespie 2021] The growth in remote schooling, while temporary, coincides with a long-standing and ongoing interest in so-called “artificial intelligence” or “machine learning” technologies and their applications in business contexts. [Sestino & de Mauro 2021, Benaich & Hogarth 2018, 2019, 2020, 2021] An increasing number of so-called “EdTech” (education technology) companies are riding this wave to attain “unicorn” status (a market valuation in excess of $1 billion), often touting their use of cutting-edge machine learning techniques to support their products or deliver their services. In many lower- and middle-income country (LMIC) contexts, transitions to remote learning were uneven, with significant variance in access, quality, and coverage. [World Bank 2020, UNICEF 2020] Indeed, in many LMICs even the most basic of modern technologies can be unavailable: as of 2015, up to 90 million children in Sub-Saharan Africa were studying in classrooms that lacked electricity. What is the relevance to these education systems of analytical techniques developed in contexts where computing devices are plentiful? This presentation will answer that question in two ways. First, by introducing Loquat, a novel application from RTI International that leverages automatic speech detection and the increasing ubiquity of low-end smartphones to give teachers accessible, affordable instructional coaching. Using machine learning, Loquat detects and classifies verbal interactions between teachers and students. Automated analyses translate these data into simple but powerful visualizations that, combined with guided reflection, provide tailored, actionable feedback teachers can use to understand their talk management and facilitate its improvement. Second, the presentation will review recent advances in machine learning and so-called “artificial intelligence”, detail their potential relevance to lower- and middle-income country education systems, and discuss how programs that aim to improve foundational learning outcomes could begin to leverage these powerful new tools.

Policy linking is a standard-setting methodology, long used in many countries, to set benchmarks (or cut scores) on learning assessments that allow those countries to determine what percentage of students in their country are meeting minimum proficiency requirements for key skills such as reading and math. While it is an old standard-setting methodology, its use has been extended to help countries set benchmarks that will allow reporting against global standards. Policy linking allows countries to use their existing national and/or regional assessments to report against Sustainable Development Goal (SDG) 4.1.1: “Proportion of children and young people in Grade 2 or 3 (4.1.1a), at the end of primary education (4.1.1b), and at the end of lower secondary education (4.1.1c) who achieve at least a minimum proficiency level in reading and mathematics.” It works by linking assessments to the Global Proficiency Framework (GPF), a framework developed by global reading and math content experts based on current national content and assessment frameworks across more than 25 countries. The GPF provides performance expectations/ standards for learners in Grades 1-9 in reading and mathematics. By linking existing national and regional assessments to the GPF, countries and donors are able to compare learning outcomes across language groups in countries as well as across countries and over time, assuming all new assessments are subsequently linked to the GPF. In this roundtable, we will share learning from policy linking work that has taken place this past year. Following a brief introduction to Policy Linking for Measuring Global Learning Outcomes by Dr. Saima Malik, from USAID in Washington DC, Dr. Asumpta Matei from the Kenya National Examinations Council and Dr. Enos Radeny of USAID Kenya will present the model of a Policy Linking workshop that was designed and implemented in order to build ministry capacity as well as set benchmarks for grades 2 and 3 in English and Kiswahili in Kenya, Dr. Abdullah Ferdous and Dr. Jeff Davis of AIR (co-developers of the policy linking approach) will discuss the importance of feedback in establishing defensible global benchmarks during the policy linking process and Dr. Jodie Fonseca from RTI will share practical example from Nepal where policy linking was used to align the national assessment to the Global Proficiency Framework and proved to be a more cost-effective way to measure early grade reading outcomes than an EGRA. Melissa Chiappetta of Sage Perspectives will serve as discussant of the panel.

This presentation was part of a CIES 2022 panel on measuring learning through play and child SEL outcomes across humanitarian and LMIC contexts. The presentation focuses primarily on the development of a new SEL tool that is being used as part of impact evaluations for five learning through play implementation programs across five countries.

Prior Early Grade Reading Assessments (EGRA) have been used to set reading fluency benchmarks in Tanzania for USAID report and for the Government of Tanzania (GoT). Since the EGRA requires one-on-one administration with trained enumerators, tablets, it is currently too expensive to be sustainable within the government system. The Group Administered Literacy Assessment (GALA) is an inexpensive and sustainable way to collect information about students’ reading abilities; is it group administered, does not require intensive training to administer, and is collected on paper, which is then entered into a database. Unfortunately, the GALA does not contain a fluency measure, which is still used as the basis of USAID reporting. The Jifunze Uelewe team created a study in order to identify the reading fluency equivalent benchmarks for the GALA on a subsample of the total GALA respondents. The study is administering the both the EGRA’s reading passage and the GALA to a sample of grade 2 and grade 4 pupils attending public schools in Tanzania. Data collection occurred in October 2021. Data collection was happening during the submission of this abstract, so no results are available for the abstract. But we will report the results and will discuss how well the linking process worked.

Play has the potential to transform the global learning crisis. In infancy and early childhood, play builds a strong foundation for later learning by improving brain development and growth (Goldstein, 2012). In education systems that lack capacity to support children effectively, play brings its own powerful engine to drive learning—the joyful, engaged intrinsic motivation of children themselves (Zosh et al., 2017). In this way, play contributes to the holistic development of children, helping to prepare them for the challenges of the current and future world. Accordingly, there is an urgent need to improve measurement of playful learning, to be able to add to the evidence base on what the benefits of play are, how playful learning takes place, and how it can be promoted at home and at school across the lifespan. This presentation focuses on a renewed conceptualization of playful learning and describes an innovative approach to measuring how settings contribute to playful learning for children ages 0 to 12, supported by the Lego Foundation. The settings we examine include homes, classrooms and ECD centers. Following Tseng and Seideman (2007), we view settings as consisting of social interactions (i.e. between teachers or caregivers and children) and the organization of resources (e.g. learning materials, games). First, we will present our conceptual framework which identifies six constructs to guide our measurement strategy. The constructs, such as ‘support for exploration’, represent the ways in which a setting supports playful learning. Next, we will present our contextualization framework which guides how we are adapting and modifying the measurement tools to different contexts. The tool consists of a protocol to observe adult-child interactions and survey measures conducted with teachers, caregivers and primary school pupils. As part of the development process for these measurement tools, observation and survey measures will go through a three-phase development process in Kenya, Ghana, Colombia, and Jordan. The Build phase involved collecting qualitative data from teachers, caregivers and students to understand their perception of playful learning and how it is supported at home and at school. Next, an Adapt phase took place where the initial versions of the measurement tools underwent cognitive interviewing, field adaptation, and a small pilot to adjust and extend the items in the tool. The third Test phase is a full pilot of the instruments, and the data will undergo rigorous psychometric analyses to review the validity and reliability of the tools in the four country contexts. We will use the results to adjust the instruments and to finalize the conceptual framework and contextualization strategies. The final toolkit will be publicly available towards the end of 2022 with supporting materials for contextualization, piloting, training and analysis. The toolkit will be available on a public platform designed to promote sharing of data collected using the tool and to collaborate to continually improve approaches to measuring support for playful learning.

Learning outcomes are low and instruction is poor in many low- and middle-income countries (LMICs). These shortcomings are particularly concerning given the substantial learning loss due to COVID-19 from which many systems are suffering. The Learning at Scale study identified eight of the most effective large-scale education programs in LMICs and now is examining what factors contribute to successful improvements in learning outcomes at scale (see list of programs on last page of this brief). These programs were selected based on their demonstrated gains in reading outcomes at-scale, from either midline or endline impact evaluations. The study addresses three overarching research questions, focused on understanding (1) the components of instructional practices (Brief 1), (2) instructional supports (Brief 2), and (3) system supports (Brief 3) that lead to effective instruction. This brief focuses specifically on system supports. It addresses the following research question: What system supports are required to deliver effective training and support to teachers and to promote effective classroom practices?

Learning outcomes are low and instruction is poor in many low- and middle-income countries (LMICs). These shortcomings are particularly concerning given the substantial learning loss due to COVID-19 from which many systems are suffering. The Learning at Scale study identified eight of the most effective large-scale education programs in LMICs and now is examining what factors contribute to successful improvements in learning outcomes at scale (see list of programs on last page of this brief). These programs were selected based on their demonstrated gains in reading outcomes at-scale, from either midline or endline impact evaluations. The study addresses three overarching research questions, focused on understanding (1) the components of instructional practices (Brief 1), (2) instructional supports (Brief 2), and (3) system supports (Brief 3) that lead to effective instruction. This brief focuses specifically on instructional supports. It addresses the following research question: What methods of training and support lead to teachers adopting effective classroom practices in successful, large-scale literacy programs?

The Learning at Scale study aimed to investigate factors contributing to successful improvements in learning outcomes at scale in eight of the most effective large-scale education programs in LMICs (see the map of programs on the last page of this brief). These programs were selected based on their demonstrated gains in reading outcomes at-scale, from either midline or endline impact evaluations. The study addressed three overarching research questions, focused on understanding the components of instructional practices (Brief 1), instructional supports (Brief 2), and system supports (Brief 3) that lead to effective instruction. This brief focuses specifically on instructional practices. It addresses the following research question: What classroom ingredients (e.g., teaching practices, classroom environment) lead to learning in programs that are effective at scale?

On January 18, 2022, the USAID’s Bureau for Asia collaborated with RTI International to reflect on the journey of early grade reading around the globe. The first presenter, Rosalina J. Villaneza, gave an introduction of national-scale early grade literacy assessments in the Philippines. The second presenter, Pilar Robledo, discussed the advent of USAID early grade reading programs, using the EGR Barometer to explore the impact of these programs. The final presenter, Luis Crouch, reflected on research and experience of early grade reading programs, suggesting the next steps on this journey to improve early grade literacy worldwide. View the recording below.

The Learning at Scale study was designed to identify existing early grade reading programs with demonstrated impact on basic skills at scale and to conduct in-depth investigations of these programs to determine what makes them successful. After an extensive search, eight programs (spanning seven countries) were selected for inclusion in the study. Research on these programs has been conducted in order to answer the three overarching research questions, focused on understanding the components of instructional practices, instructional supports, and system supports that lead to effective instruction. Learning at Scale data collection activities for some of these programs were delayed due to COVID-19. However, with demand for information about how to implement effective interventions at large scale at an all-time high, we believe that the timely sharing of findings from Learning at Scale is essential. Accordingly, this interim report provides preliminary findings from our study to date, highlighting key high-level findings across all eight programs, as well as quantitative and qualitative findings from primary research for select programs. The Learning at Scale study is led by RTI International, as part of the Center for Global Development (CGD) education research consortium, funded by the Bill and Melinda Gates Foundation.