Right-fit EdTech: Leveraging Loquot and other (machine) learning to support foundational learning in LMICs [CIES Presentation]

Title: Right-fit EdTech: Leveraging Loquat and other (machine) learning to support foundational learning in LMICs Abstract The COVID-19 pandemic has severely disrupted global education systems: according to the Global Education Monitoring Team, over 1.6 billion students have lost significant instructional time, with many yet to return to school. But even before the pandemic struck, the state of education around the world was so poor that the World Bank was decrying a “learning crisis.” To address the crisis, the Bank called for both a renewed emphasis on what teachers actually do in a classroom and the strategic deployment of technology to improve teaching and learning. During the pandemic, remote-first instruction became a feature of basic, secondary, and tertiary education systems in many high-income countries. [European Commission, Rickles et al. 2020, Hodges et al. 2020, Gillis & Krull 2020] Coverage was not universal, as wealth and income inequality within high-income countries remains a persistent challenge. [Romm 2020, Blaskó & Schnepf 2020] Nonetheless, the market for educational technology firms boomed during the pandemic, with investment in the sector expected to exceed $80 billion by the end of the decade. [Gillespie 2021] The growth in remote schooling, while temporary, coincides with a long-standing and ongoing interest in so-called “artificial intelligence” or “machine learning” technologies and their applications in business contexts. [Sestino & de Mauro 2021, Benaich & Hogarth 2018, 2019, 2020, 2021] An increasing number of so-called “EdTech” (education technology) companies are riding this wave to attain “unicorn” status (a market valuation in excess of $1 billion), often touting their use of cutting-edge machine learning techniques to support their products or deliver their services. In many lower- and middle-income country (LMIC) contexts, transitions to remote learning were uneven, with significant variance in access, quality, and coverage. [World Bank 2020, UNICEF 2020] Indeed, in many LMICs even the most basic of modern technologies can be unavailable: as of 2015, up to 90 million children in Sub-Saharan Africa were studying in classrooms that lacked electricity. What is the relevance to these education systems of analytical techniques developed in contexts where computing devices are plentiful? This presentation will answer that question in two ways. First, by introducing Loquat, a novel application from RTI International that leverages automatic speech detection and the increasing ubiquity of low-end smartphones to give teachers accessible, affordable instructional coaching. Using machine learning, Loquat detects and classifies verbal interactions between teachers and students. Automated analyses translate these data into simple but powerful visualizations that, combined with guided reflection, provide tailored, actionable feedback teachers can use to understand their talk management and facilitate its improvement. Second, the presentation will review recent advances in machine learning and so-called “artificial intelligence”, detail their potential relevance to lower- and middle-income country education systems, and discuss how programs that aim to improve foundational learning outcomes could begin to leverage these powerful new tools.

A Research Framework for Capturing Teachers' Decision-Making [CIES Presentation]

The purpose of this session is to articulate a research framework that centers teachers’ voices when trying to understand how teachers use curriculum materials in the classroom. Operating in the context of highly structured lesson plans, the approach identifies ways in which teachers exercise their professional discretion to modify the lesson and frames conversations to elaborate the motivations driving the teachers’ choices. The approach has been iteratively refined across three studies; taken together, the studies provide evidence for the value of listening to teachers and being responsive to their voices during implementation. The research framework uses the lens of modifications, or changes to the intended lesson plan implemented within one class period. Modifications can be large or small, additive or subtractive. For example, in a lesson with a section for independent student practice, a large modification might be skipping the practice section entirely. Or, during a lesson focused on blending of initial sounds using 3 example words, a small modification might be extending the exercise by adding extra words. Researchers observe the lesson, noting any modifications; after the lesson, the researchers select some of the modifications and ask the teachers why they made the choices they did. Analyses of teachers’ explanations highlight the importance of understanding why teachers make the choices they do. For example, a teacher who skipped the independent practice section because they don’t think their students are ready to do the skill on their own suggests that the teacher is exercising agency and using her knowledge of her students to inform her decision-making. Insights such as this one can guide decisions on projects. It may be that while the intention of the teacher was guided by knowledge of students, the end result is not desirable from the project’s point of view. Understanding why the teacher made this choice provides implementers with better and targeted ways to address choices that impact the overall goals of the project. In this presentation, we draw on data from an exploratory case study to understand use of new mathematics materials in Liberia, a more in-depth case conducted in Malawi on teacher use of reading materials, and finally, a systematic study examining how reading teachers use materials across four Sub-Saharan African countries. We use each case to highlight both an aspect of the research framework and instances of modifications to project implementation driven by teachers’ voices. By focusing on teacher voices, we disrupt the deficit notion that teachers are “resistant” to change, or do not “understand” new pedagogies. Instead, we aim to value teacher voices and integrate their insight into implementation programs. By doing this, we not only raise the likelihood of successful use of new materials and pedagogies, but we also develop more responsive pedagogy that better matches existing classroom cultures.

Short Message Service (SMS)–Based Remote Support and Teacher Retention of Training Gains in Malawi

Chapter 5 of the book Cultivating Dynamic Educators: Case Studies in Teacher Behavior Change in Africa and Asia. This chapter critically reviews the design, implementation, and evaluation of an attempt to study an exploratory short message service (SMS)–based intervention conducted under the auspices of the United States Agency for International Development’s (USAID’s) Malawi Early Grade Reading Activity (EGRA).1 The overall EGRA program, which was implemented from July 2013 to October 2016 in 1,614 schools across 11 educational districts, was designed to support Malawi’s Ministry of Education, Science, and Technology (MOEST) to improve reading outcomes in both Chichewa. and English languages among children in grades 1–3. The

Uganda/LARA: EGR Action Research Tools

LARA developed action research tools to measure the level of fidelity of implementation of the EGR methods and generate lessons learned to inform adaptations in EGR programming. The action research tools gather both historical and real-time data at the school. They include the EGR core methodologies action research tool; the remedial instruction action research tool and the intensive coaching action research tool (subdivided into two tools i.e. head teacher coaching event log and school based community of practice event log). The EGR core methodologies action research tool assesses the teacher’s perception of the Teacher Guide usability, level of macro pacing, implementation of lesson plan elements, implementation of core EGR methodologies and tracking of instructional adaptations by the teacher. The remedial instruction research tool tracks the teacher’s perception to remedial instruction, the implementation of group-based instruction as well as in-class assessment. The intensive coaching action research tool tracks teacher’s perception of intensive coaching and keeps a log of head teacher instructional coaching events in addition to school-based community of practice activities. The action research tools are designed to be deployed electronically in order to seamlessly incorporate extra data quality standards and innovations like the Stalling’s classroom observation snapshot (Stallings and Kaskowitz, 1974 ). The project also developed the action research process flow guidelines to guide data collection activities.

Measurement of Inequality in Learning Levels [Conference Presentation]

The presentation summarizes a paper by Tim Slade and Luis Crouch on the measurement of learning inequality before and after a successful reading project. The paper concludes that at least for the case studied, the project improved not only the averages but also reduced the inequality. The paper was prepared under the auspices of a conference on "Learning at the Bottom of the Pyramid" organized by IIEP and Dan Wagner of U Penn. This is the presentation that was delivered at vCIES 2020.

Instructional coaching and literacy improvement at national scale: Lessons from Kenya’s Tusome early grade reading activity [CIES 2019 Presentation]

The Tusome Early Grade Reading Activity is USAID’s flagship education program in Kenya. This CIES 2019 presentation shares the findings from research currently in progress to analyze Tusome’s 2017 lesson observation data, and shares lessons learned from: designing a coaching program to operate at scale; effectively combining incentives and sanctions to drive coaching activities; and effectively combining automated, moderately high-tech data pipelines with qualitative, low-tech feedback on coaching of coaches.

Is ‘summer’ reading loss universal? Using ongoing literacy assessment in Malawi to estimate the loss from grade-transition breaks

Published abstract: "Summer learning loss – decreased academic performance following an extended school break, typically during the period after one grade ends and before another grade starts – is a well-documented phenomenon in North America, but poorly described in sub-Saharan African contexts. In this article, we use the term ‘grade-transition break’ loss in lieu of ‘summer’ loss to refer to the period after one grade ends and before another grade starts. This study analyses data from early grade reading assessments in Malawi, estimating statistically significant average reductions of 0.38 standard deviations (SD) across several measures of reading and pre-reading skills during two grade-transition breaks. The data show the loss in reading skills during the extended breaks between grades 1 and 2 and between grades 2 and 3 in two consecutive years. The study found no gender-based differences in loss. The findings suggest a need for early grade reading interventions to develop and evaluate mitigation strategies lest significant proportions of within-year performance gains be lost over the break between academic years."

Low-cost, familiar tech for teacher support: Evidence from a SMS campaign for early grade teachers in Malawi

Presentation delivered at CIES2017 (Atlanta). Providing teachers guidance, mentorship and encouragement in between formal, face-to-face trainings or coaching sessions is challenging. While school directors and other peers may offer teachers support in some contexts, others may experience difficulties, isolation or discouragement in incorporating new practices into their classroom instruction. This paper presents new research from a controlled study in Malawi that sought to extend in-person professional development trainings with a targeted communication campaign over a familiar, low-cost and ubiquitous medium: SMS text messages.