Monitoring, Evaluation, and Research

Dr. Benjamin Piper and Dr. Yasmin Sitabkhan presented findings at CIES 2019 on gender and math in the preschool years from the Kenya Tayari program.

There is growing demand for policy based on rigorous evidence. Many consider the strongest evidence to come from studies that identify causality with high internal validity - such as RCTs - and systematic reviews of these studies. If policies are based strictly on such rigorous evidence there is a risk of bias towards simple, discrete, measurable interventions and away from complex interventions. Rigorous evidence is also better suited to some questions than others. Evaluations may provide stronger conclusions about impact than about the mechanisms, implementation, context, generalisability and scaling of interventions. For these reasons, policy-making does – and should – consider issues for which there is no conclusive evidence. However, there is little guidance as to how and when such inconclusive evidence can be used. We present a framework for considering inconclusive evidence applied to examples from evidence-based education in low- and middle-income countries. The framework involves a systematic consideration of the estimated costs, benefits and potential harm of a policy, along with the uncertainty in those estimates. This analysis is conducted using standard decision theory and an examination of the utility of policies. We argue that it is rational to pursue a policy with uncertain outcomes if there is a reasonable probability of large positive utility (compared to the cost of the intervention) and a low probability of negative utility. The decision to act under uncertainty is influenced by a number of other considerations including: the potential to improve the evidence base, the urgency of the decision and the analysis of alternative options. The framework also calls for systematic analysis of uncertainty associated with all components of a policy decision. For example, some interventions may have robust evidence of impact but considerable uncertainty associated with the generalisability of the evidence to a new context, or with the scalability of the intervention. We discuss our approach to measuring and reducing uncertainty in policy decisions and its implications for evaluation and research. The overall aim of this work is to make evidence-based decision-making more effective and applicable to a wider range of problems.

This presentation is based on an activity that was designed to apply lessons learned and best practices from the recent EGRA Benchmarks and Standards Research Report (RTI International, 2018) to a five-language benchmarking activity for early grade reading in India.

Executive functions are a cognitive skill set that underlie our goal-directed, planning, and problem solving behavior, and include the components of working memory, inhibitory control, and cognitive flexibility. EF skills undergo the majority of development during the pre-primary years of a child’s life and have been shown to contribute to academic success. However, most of our knowledge about children’s EF skills have been based on research with children living in high-income countries. This presentation reports on findings from the administration of an EF assessment with children from a West African country. Students entering Primary 1 grade for the first time from Kindergarten class were sampled. All students were administered four pre-literacy tasks and a set of questions measuring socio-economic status. Half of the sample also received EF touch games, including two training modules, two tasks measuring inhibitory control and 1 task measuring working memory. The presentation will focus on the findings of the use of EF Touch with this sample of children from a West African country. First, a brief description of the process of adapting and revising the tools for use in Liberia is reported. Second, a descriptive analysis is presented in order to describe the feasibility of using EF Touch with young children in this context. Third, children’s performance on the three tasks is summarized and correlations among the scores on the three tasks is reported and discussed. Fourth, a model exploring the unique contributions of simple reaction time and demographic characteristics is presented. Finally, the overall contribution to the field of early childhood assessment and executive function measurement in LMICs is discussed.

Core EGMA instrument in English

This report summarizes the findings of the 2018 Early Grade Reading and Mathematics Initiative (RAMP) Lot Quality Assurance Sampling Assessment, measuring impact between 2017 and 2018. The United States Agency for International Development (USAID) and UK Aid have funded RAMP as a national effort designed to improve the reading and mathematics skills and performance of students in Jordan from kindergarten 2 through grade 3 (K2–G3). This five-year program began on January 1, 2015, and is scheduled to end on December 31, 2019. Her Majesty Queen Rania Al-Abdullah formally launched RAMP as part of the broader Ministry of Education (MoE) initiative to improve education. The Research Triangle Institute (RTI) International leads implementation with its partners: Queen Rania Teacher’s Academy, ChangeAgent for Arab Development and Education Reform, We Love Reading, The Kaizen Company, Mercy Corps, Dajani Consulting, and Prodigy Consulting. The RAMP team and the MOE conducted a Lot Quality Assurance Sampling (LQAS) assessment in November 2018. Approximately 200 assessors (most of whom were MOE supervisors) were trained to collect reading and mathematics performance data across all 42 field directorates in the kingdom. For this activity, an approach was used at the school level, which led to a final sample of more than 39,000 Grade 2 and Grade 3 students in 2,076 schools. Performance comparisons made on three key indicators (reading comprehension, oral reading fluency, and mathematics) provides illustrative evidence of the gains made by RAMP schools over one school year. Large gains were seen on reading comprehension with the percent of students reaching the comprehension benchmark, increasing from 43% in 2017 to 55% in 2018. Gains in terms of oral reading fluency increased from 13% in 2017 to 19% in 2018. The smallest gains were in terms of mathematics (where the percent of students reaching the benchmark improved only slightly from 28% in 2017 to 30% in 2018).

In partnership with the Ministry of Education and Higher Education (MOEHE), RTI International administered a baseline assessment of early grade reading skills in April and May 2018. The goal of the baseline assessment was to provide the United States Agency for International Development (USAID) West Bank Early Grade Reading (EGR) activity with a deeper understanding of the pre-literacy, reading, and writing abilities of kindergarten (KG), grade 1 (G1), and grade 2 (G2) students in the formal language of primary school instruction, Modern Standard Arabic (MSA). The baseline assessment also established the values of program performance indicators before the implementation of the EGR intervention, which will subsequently be compared to future values collected at midline and endline timepoints to evaluate the effectiveness of the program. The baseline assessment was conducted with 293 KG students, 1,492 G1 students, and 1,465 G2 students randomly selected and equally divided between girls and boys. The assessment measured the students’ pre-literacy and early grade reading ability and writing skills, including letter sound knowledge, word recognition, oral reading fluency (ORF), listening comprehension, reading comprehension, and writing.

Given the dearth of research on early numeracy interventions in low- and middle-income countries, this paper presents the instructional methodology and impact results of the Tayari program. Tayari is a preprimary intervention in Kenya (2014–2019) that prepares children aged four and five for entry into primary school by providing materials for students, training for teachers, and continuous in-classroom support. The Tayari methodology was built on the Kenyan government’s preprimary syllabus to produce instruction that was developmentally sequenced, linked to out-of-school experiences, and supportive of children’s number sense. Tayari was evaluated using a randomized controlled trial (RCT) and collection of longitudinal data from 2,957 children in treatment and control schools at three time points. Pupil assessment items were drawn from a growing body of research on preprimary numeracy in developing contexts, plus instruments and techniques from the Measuring Early Learning and Quality Outcomes (MELQO) program (UNESCO, UNICEF, Brookings Institution, & World Bank Group, 2017). The impact evaluation of the longitudinal RCT results showed statistically significant effects in the numeracy tasks of producing sets, identifying numbers, and naming shapes, while revealing no initial effects in the areas of oral and mental addition. We present recommendations for Tayari’s improvement in terms of mathematics instruction, as well as preprimary policy implications for Kenya and similar contexts.

This paper explores an innovative assessment framework for measuring children’s formal and informal mathematical knowledge. Many existing standardized measures, such as the Early Grade Mathematics Assessment, measure children’s performance in early primary grade skills that have been identified by researchers and policy makers as foundational and predictive of later academic achievement (Platas, Ketterlin-Geller, & Sitabkhan, 2016; RTI International, 2014). However, these standardized assessments only provide information on children’s mathematical ability as it pertains to skills and concepts that are a focus of school instruction, referred to as formal mathematics. While valuable, they leave unmeasured the mathematics that children use and develop as part of their everyday life, such as the strategies they use to solve simple arithmetical problems that arise as they move through their day (Khan, 1999; Saxe, 1991; Taylor, 2009). In this article, we draw from mixed methods studies which focus on capturing the informal mathematical skills that children develop outside of school in various contexts (Guberman, 1996; Nasir, 2000; Sitabkhan, 2009; Sitabkhan, 2015). We describe how the use of observations of children’s mathematical activities in natural settings and in subsequent cognitive interviews using mathematical tasks derived from those observations, can illuminate mathematical knowledge and skills that may otherwise remain hidden. We found that an assessment framework that focuses on both standardized measures of formal mathematical learning and contextualized measures of children’s everyday mathematics can provide a more complete and nuanced picture of children’s knowledge, and taken together can inform the development of curricular materials and teacher training focused on early learning.

Do children with deficits in both rapid automatic naming (RAN) and phonological awareness in Chichewa experience more difficulty with reading growth than children with single or no deficits? To answer this question, we developed a RAN measure of common objects and a phonological awareness measure and administered them four times during a school year. Students who scored poorly on one of these measures as compared to their peers were classified as having a deficit. A compelling finding was Single Deficit and Double Deficit groups have similar average scores at Time 1; by Time 3 and Time 4, the Single Deficit group pulls away, more than doubling the average score of Double Deficit counterparts at Time 4. This study presents a tool to help to identify children at risk for reading difficulties. Children with a Double Deficit profile may need differentiated instruction.