Monitoring, Evaluation, and Research

An overview of the results of a nationally-representative Early Grade Reading Study conducted in 2019 in the Philippines. The full report is available.

LARA developed a set of qualitative tools to learn about the successes and challenges related to the implementation of Journeys and to understand what changes staff and pupils had observed since Journeys started in the program schools. The qualitative tools include individual interviews and focus group discussion (FGD) guides with head teachers, teaching and non-teaching staff, change agents and students. There are two individual interviews, one for the teachers and another for the head teachers. The individual interview for teachers investigates the value the Journeys program has brought to the teachers personally, to the school and the classroom, for example changes in the way teachers relate and interact with pupils and changes in disciplinary practices at the school. The individual interview for head teachers on the other hand investigate what has gone well and what the head teachers are struggling with regarding the implementation of Journeys for School Staff and Journeys for Pupils (the Uganda Kids Unite [UKU] Program). There are three FGD guides; (i) FGD guide for teaching and non-teaching staff provides information about the changes (for example interactions among students, teacher attendance, extent of SRGBV) in the school as a result of Journeys, initiatives undertaken by the school to make the school safe and positive and how the initiatives improved the school and/or reduced violence; (ii) FGD guide for head Teachers and school change agents (SCA) that gathers feedback on the successes and challenges associated with the implementation of Journeys program for the school staff and Journeys program for pupils as well as improvements needed to for the continuity of the Journeys program in the schools; and (iii) FGD guide for students that focuses mainly on what pupils enjoyed most about the UKU program and the specific UKU activities they loved. It also asks about what pupils did not enjoy in the UKU meetings, initiatives that UKU teams developed to improve the school, what pupils learned through the UKU program and how the school and classroom have changed since Journeys began.

This tool is used by school support actors (e.g., DEOS, DIS, MEOs, DIS, MIS, IS, CCTs, head teachers, project staff) to monitor and support implementation of Journeys for school staff and Journeys for pupils. The tool serves two purposes: (i) to gather monitoring data to understand how well schools are organized and supported to implement the Journeys program. Monitoring data is also used for reporting purposes. With regard to Journeys for school staff, the type of monitoring data gathered using the MSS tool include the number of school change agents (SCAs) in the school, number of SCAs that have Journeys handbooks, number of Journeys for staff activities that the school conducted, evidence that SCAs receive support from support actors, number of SRGBV cases that have been resolved out of the total cases reported. For UKU, monitoring data gathered include the number of UKU teams in the school, number of UKU teams that have attendance registers, whether UKU attendance registers are updated regularly and whether UKU teams have work plans; (ii) to supervise and support SCAs and Teacher Patrons. SCAs lead Journeys sessions for Schools Staff while Teacher Patrons lead Journeys sessions for Pupils/UKU. The MSS tool has an observation checklist for School Staff Journeys sessions and an observation checklist for UKU sessions. The checklists are used by the school support actors to record key information about the Journeys session they observed, including whether the SCA/Teacher Patron applied Journeys facilitation approaches, followed the rights steps to conduct the session and used the planning form and evaluation form correctly to initiate and appraise the session respectively. Each checklist is complemented with a feedback section where the school support actors document the positive things (areas of strength) the SCA/Teacher Patron demonstrated during the Journeys activity and area of improvement.

LARA developed action research tools to measure the level of fidelity of implementation of the EGR methods and generate lessons learned to inform adaptations in EGR programming. The action research tools gather both historical and real-time data at the school. They include the EGR core methodologies action research tool; the remedial instruction action research tool and the intensive coaching action research tool (subdivided into two tools i.e. head teacher coaching event log and school based community of practice event log). The EGR core methodologies action research tool assesses the teacher’s perception of the Teacher Guide usability, level of macro pacing, implementation of lesson plan elements, implementation of core EGR methodologies and tracking of instructional adaptations by the teacher. The remedial instruction research tool tracks the teacher’s perception to remedial instruction, the implementation of group-based instruction as well as in-class assessment. The intensive coaching action research tool tracks teacher’s perception of intensive coaching and keeps a log of head teacher instructional coaching events in addition to school-based community of practice activities. The action research tools are designed to be deployed electronically in order to seamlessly incorporate extra data quality standards and innovations like the Stalling’s classroom observation snapshot (Stallings and Kaskowitz, 1974 ). The project also developed the action research process flow guidelines to guide data collection activities.

The Early Grade Reading Assessment (EGRA) is a diagnostic instrument designed to quickly assess foundational skills for literacy acquisition of pupils in Early Grades of primary school. This diagnostic tool, whose content has been adapted for Uganda and Ugandan local languages, can include a number of subtasks depending on the grade being assessed. EGRA is administered to pupils both in their local language and in English via tablets using a software application (for example Tangerine) designed specifically to collect data on mobile devices. EGRA data can also be gathered manually using paper forms. Each instrument is administered by trained assessors in one-on-one sessions with individual pupils, and requires approximately fifteen minutes. The sub tasks in the English EGRA tools used by LARA include letter sound knowledge, oral passage reading, reading comprehension and vocabulary. LARA administers the English sub-tasks mainly to the P3 and P4 pupils. The subtasks in the local language (Luganda, Runyankore/Rukiga and Runyoro/Rutooro) EGRA tools include orientation to print, letter sound knowledge, segmenting, oral passage reading, reading comprehension, and listening comprehension. Apart from orientation to print that is administered to only P1 pupils, the rest of the local language sub-tasks are administered to P1-P4 pupils. The EGRA tools are accompanied by a pupil stimuli packet used to administer the letter sound knowledge and oral passage reading sub tasks. The subtasks used by LARA were adopted from the EGRA tools developed for the School Health and Reading Program (SHRP). SHRP adapted and vetted the tools to the three languages during a series of weeklong workshops that included researchers, primary school teachers, language board members, Coordinating Center Tutors (CCTs) and MoES staff. The adaptation workshop for Luganda was held in December, 2012, Runyankore/Rukiga tools were adapted in January, 2013 and Runyoro/Rutooro tools were adapted in August 2013. The tools were also piloted in each of the three language areas during the workshops. As part of EGRA data collection, LARA administers supplementary data collection tools to assess pupil context and instructional leadership. The tools also provide important contextual information on the teachers and schools participating in EGRA. These tools include; the pupil context interview, a head teacher interview, teacher interview and school inventory. The following provides a summary of each of the tool. Pupil context interview: Used to gather information on pupils’ preschool attendance, language spoken at home, possessions in the household, and support for reading in the home. Head Teacher Questionnaire: Used to gather information from head teachers regarding their instructional leadership, including their training and education background and their support to the teaching of reading at lower grades. Data from this instrument is used to inform training and targeted corrective actions intended to improve the managerial skills of head teachers and their support to the teaching of reading. Teacher Questionnaire: Used to gather information on the teachers’ education and experience and demographics, support and supervision received, and the availability of teaching materials. The information provides the basis of training and the provision of teaching materials to teachers to help them improve their pedagogical skills. School Inventory Form: Used to gather information on school basic infrastructure (for example water source, latrines and electricity) as well as the presence and use of a school library.

This report presents the findings of the Jordan Early Grade Reading and Mathematics Initiative (RAMP) endline survey conducted at the end of the 2018–2019 school year (in May 2019).

This report presents a cross-country analysis of three qualitative case studies completed in Jordan, Uganda, and Bangladesh in late 2019. It reflects a snapshot of information about the refugee experience of early childhood services, based on interviews, focus group discussions, site visits and policy document review. The crosscountry analysis investigates individual and group stories and experiences to synthesize common themes with the goal of identifying recommendations to improve the provision of early childhood services for young refugee children and their families.

This qualitative case study describes the experiences of young refugee children and their families accessing early childhood development (ECD) services in Uganda in late 2019. The study team collected data through key informant interviews with representatives of the Ugandan government, national non-governmental organizations, humanitarian agencies and service providers. Focus group discussions were held with refugee families living in Bidi Bidi and Nakivale settlements. The study team supplemented key informant interviews and focus group with policy document review. Analysis is presented along the lines of policy and practice, with a focus on the respective roles of the government and international agencies in delivering ECD services to refugee families.

This qualitative case study describes the experiences of young refugee children and their families accessing early childhood development (ECD) services in Bangladesh in late 2019. The study team collected data through key informant interviews with representatives of the Bangladesh government, national non-governmental organizations, humanitarian agencies and service providers. Focus group discussions were held with refugee families living in both older and newer (post-2017 influx) camps near Cox's Bazar. The study team supplemented key informant interviews and focus group with policy document review. Analysis is presented along the lines of policy and practice, with a focus on the respective roles of the government and international agencies in delivering ECD services to refugee families.

The presentation summarizes a paper by Tim Slade and Luis Crouch on the measurement of learning inequality before and after a successful reading project. The paper concludes that at least for the case studied, the project improved not only the averages but also reduced the inequality. The paper was prepared under the auspices of a conference on "Learning at the Bottom of the Pyramid" organized by IIEP and Dan Wagner of U Penn. This is the presentation that was delivered at vCIES 2020.