Digital Assessment and Data Collection

Своевременные данные, поступающие в систему, необходимы для обеспечения подотчетности основных действующих лиц за осуществление структурированной педагогической деятельности, для оценки воздействия данной деятельности на педагогическую деятельность учителей, для определения и обоснования коррекции содержания программы и, что наиболее важно, какое воздействие, если таковое вообще имеет место, оказывает данная деятельность на ключевые целевые результаты программы (например, базовые навыки учащихся в области грамотности и счета).

This report summarizes the findings of additional analyses conducted to delve deeper and develop more insight into the piloting of the Self-Administered Early Grade Reading Assessment (SA-EGRA) and the Self-Administered Early Grade Mathematics Assessment (SA-EGMA). These tools were developed and tested by RTI International with the support and direction of Imagine Worldwide. Children complete these assessments independently on tablet-based software while in a classroom with their peers. An adult supervises the process.

This report summarizes the findings of an effort to develop and validate tablet-based, self-administered assessments of Chichewa-language foundational literacy and numeracy in the early grades in Malawi. RTI International developed the two assessments, known respectively as the Self-Administered Early Grade Reading Assessment (SA-EGRA) and the Self-Administered Early Grade Mathematics Assessment (SA-EGMA), with the support and at the direction of Imagine Worldwide. The assessments are deemed “self-administered,” because children complete the assessments independently in response to instructions and stimuli embedded in the tablet-based software. However, adults typically supervise the organization and conduct of the assessment as well as the collection of individual data from the tablets for analysis.

USAID funded “Okuu Keremet!” is a project (5 years, $19M) designed to help improve learning outcomes in reading and math among 300,000 students in grades 1-4 in 1,686 target schools in the Kyrgyz Republic. As part of its efforts to improve learner performance, the project supported development of two user-friendly applications in four languages (Kyrgyz, Russian, Uzbek and Tajik). These apps are SabakApp (classroom observation), and BaalooApp (formative assessment), each described below. These apps represent cutting-edge and innovative solutions to challenges that teachers in the Kyrgyz Republic face in classrooms and will play a critical role when it comes to providing teachers with real-time feedback on their teaching strategies as well as instant access to knowledge, best practices, and advice that teachers currently do not have. Regular use of these applications will allow primary school teachers to implement tailored and effective teaching strategies in reading and math that will result in stronger early grade students’ performance. SabakApp is a coaching application tool developed for use by national trainers (coaches), district level coaches, and school-based instructional support teams (IST) to support teachers to improve student outcomes. This app enables these stakeholders to conduct classroom observation, use data to provide focused feedback to teachers, and engage in dialogue with decision makers on how to plan additional coaching support. This app includes the classroom observations forms that are synced with the reading and math curriculum being taught and based on the observation provides immediate feedback to teachers of their strengths and weaknesses. This data then feeds into a dashboard that ranks schools by performance, and thus provides information to the system on which schools to target. To date, more than 13,500 teachers were provided with focused mentoring support at school and district levels by the school and district leaders and national trainers. Early reviews of data collected indicate that the quality of teacher instruction has increased in less than one year. BaalooApp is a formative assessment application developed for use by primary school teachers to identify strengths and weaknesses in student performance, and to support teachers to decide the best approach to remedy the challenges identified – via individual, group or whole-class instruction; to provide individual descriptive and actionable feedback to students; and regularly reflect and adapt their own instruction with the aim to improve student learning outcomes. At present, BaalooApp is being used in 1,686 target schools. According to teachers’ perceptions: “BaalooApp gives both the overall picture on students’ results and the dynamics of each student and helps the teacher to identify both strong and struggling students and gives recommendations on how to work with both each groups” (Primary school teachers, School #5, Jalal-Abad city).

This report summarizes the findings of an effort to develop and validate tablet-based, self-administered assessments of English-language foundational literacy and numeracy in the early grades. The tools described in the report were developed at the request of Imagine Worldwide with the support of the Jacobs Foundation. RTI carried out field testing and a pilot study to assess the tools' internal consistency, test-retest reliability, and concurrent validity with respect to "traditional" EGRA and EGMA. RTI International developed the two assessments, known respectively as the Self-Administered Early Grade Reading Assessment (SA-EGRA) and the Self-Administered Early Grade Mathematics Assessment (SA-EGMA), with the support and at the direction of Imagine Worldwide. The assessments are deemed “self-administered,” because children complete the assessments independently in response to instructions and stimuli imbedded in the tablet-based software. However, adults typically supervise the organization and conduct of the assessment as well as the collection of individual data from the tablets for analysis. The tools have been developed under an open-source license. The code can be viewed and downloaded for reuse or modification at https://github.com/ICTatRTI/SE-tools/blob/main/README.md. Users of RTI's Tangerine software may request that the SA-EGRA and SA-EGMA tools be added to their Tangerine groups via https://www.tangerinecentral.org/contact

In February 2022, ACR–Philippines initiated conversations with USAID and the Philippines Department of Education (DepEd) on developing a prototype technology to enable automated assessment and scoring of learners’ oral reading fluency, listening, and reading comprehension skills. The idea resonated with DepEd leadership for several reasons. During the school years of 2020-2022, the COVID-19 Pandemic made face-to-face assessments challenging, particularly in remote learning settings. Teachers were stretched in time and resources to assess learners one-on-one their reading skills against the most essential learning competencies. Further, other international assessments like PISA use a computerbased format, and this will be an opportunity to understand how well-prepared students are to take computer-based tests. In response, ACR-Philippines sought to produce a ‘proof of concept’ that explores the feasibility of a self-administered computer-based reading assessment (CoBRA) in English and Filipino for students in the Philippines. The technology would incorporate voicerecognition software to enable students to read directly into their device. The software would automate the score of the students’ reading scores through an artificial-intelligence (AI) algorithm designed to calculate words-per-minute (wpm) and reading accuracy rate. The platform will produce reports providing students, parents, and/or teachers immediate feedback on their performance. This is a report of that pilot experience.

This material discusses the various technological innovations the USAID/Read Liberia Activity employed for data collection, teacher training, assessment, coaching and even virtual support to teachers during COVID.

Existing research on the uptake of technologies for adult learning in the global South is often focused on the use of technology to reinforce in-person learning activities and too often involves an oversimplified “with or without” comparison (Gaible and Burns 2005, Slade et al. 2018). This MEL Framework for Technology-Supported Remote Training (MEL-Tech Framework) features a more nuanced perspective by introducing questions and indicators that look at whether the technology-supported training was designed based on a solid theory of learning; whether the technology was piloted; whether there was time allocated to fix bugs and improve functionality and user design; how much time was spent using the technology; and whether in-built features of the technology provided user feedback and metrics for evaluation. The framework presents minimum standards for the evaluation of technology-supported remote training, which, in turn, facilitates the development of an actionable evidence base for replication and scale-up. Rather than “just another theoretical framework” developed from a purely academic angle, or a framework stemming from a one-off training effort, this framework is based on guiding questions and proposed indicators that have been carefully investigated, tested, and used in five RTI monitoring and research efforts across the global South: Kyrgyz Republic, Liberia, Malawi, the Philippines, and Uganda (Pouezevara et al. 2021). Furthermore, the framework has been reviewed for clarity, practicality, and relevance by RTI experts on teacher professional development, policy systems and governance, MEL, and information and communications technology, and by several RTI project teams across Africa and Asia. RTI drew on several conceptual frameworks and theories of adult learning in the design of this framework. First, the underpinning theory of change for teacher learning was informed by the theory of planned behavior (Ajzen 1991), Guskey’s (2002) perspective on teacher change, and Clarke and Hollingsworth’s (2002) interconnected model of professional growth. Second, Kirkpatrick’s (2021) model for training evaluation helped determine many of the categories and domains of evaluation. However, this framework not only has guiding questions and indicators helpful for evaluating one-off training events focusing on participants’ reactions, learning, behavior, and results (as is the focus in Kirkpatrick’s model) but also includes guiding questions and indicators reflective of a “fit for purpose” investigation stage, a user needs assessment and testing stage, and long-term sustainability. Furthermore, this framework’s guiding questions and indicators consider participants’ attitudes and self-efficacy (based on the research underpinning the theory of planned behavior), as well as aspects of participants’ post-training, ongoing application and experimentation, and feedback (Clarke and Hollingsworth; Darling-Hammond et al. 2017; Guskey). Lastly, the framework integrates instructional design considerations regarding content, interaction, and participant feedback that are uniquely afforded by technology.

This case study examines a training targeting 18,000 head teachers and subject heads over seven days using IVR accompanied by electronic materials shared via WhatsApp conducted by the USAID-funded and RTI-implemented Malawi Early Grade Reading Improvement Activity (MERIT). While the training was conducted in June of 2020, this case study includes findings from a follow up study one conducted year later, to see what training content and skills participants retained, and the training elements that were most effective in their delivery. The case study applies a process-based and learning-oriented approach, drawing from the MEL-Tech Framework (Paper 1), to understand technology-supported remote teacher training introduced in response to COVID-19.

The All Children Reading–Asia (ACR–Asia) Return to Learning Cambodia Case Study was designed to research the Cambodia Ministry of Education, Youth, and Sport’s (MoEYS’s) response to the disruption in learning created by the COVID-19 pandemic during different phases of school closure and reopening in the 2019–2020 and 2021 school years. The study used qualitative interview methods to study the education system’s response, from the central ministry level down, to how school directors and teachers implemented expectations communicated to them. Forty-four participants were interviewed, including central MoEYS personnel, school directors, teachers, and caregivers. Participants were asked about the phases of MoEYS response to the pandemic and what procedures, implementations, and subsequent outcomes occurred The presenter will present findings that positive changes were implemented in the participating schools. Many stakeholders within the education system and its stakeholders act and react to an education crisis in ways that would not be typical of the system functions and individual behaviors experienced in a regular school year. Fundamentally, the education system and its actors showed that during COVID-19, they could carry out a set of core functions (RTI International, 2018). These core systems functions were to set and communicate expectations, monitor progress, provide targeted support to facilitate students’ access to continued education during school closures, and make stakeholders accountable for meeting the expectations.