Evaluation

There is growing demand for policy based on rigorous evidence. Many consider the strongest evidence to come from studies that identify causality with high internal validity - such as RCTs - and systematic reviews of these studies. If policies are based strictly on such rigorous evidence there is a risk of bias towards simple, discrete, measurable interventions and away from complex interventions. Rigorous evidence is also better suited to some questions than others. Evaluations may provide stronger conclusions about impact than about the mechanisms, implementation, context, generalisability and scaling of interventions. For these reasons, policy-making does – and should – consider issues for which there is no conclusive evidence. However, there is little guidance as to how and when such inconclusive evidence can be used. We present a framework for considering inconclusive evidence applied to examples from evidence-based education in low- and middle-income countries. The framework involves a systematic consideration of the estimated costs, benefits and potential harm of a policy, along with the uncertainty in those estimates. This analysis is conducted using standard decision theory and an examination of the utility of policies. We argue that it is rational to pursue a policy with uncertain outcomes if there is a reasonable probability of large positive utility (compared to the cost of the intervention) and a low probability of negative utility. The decision to act under uncertainty is influenced by a number of other considerations including: the potential to improve the evidence base, the urgency of the decision and the analysis of alternative options. The framework also calls for systematic analysis of uncertainty associated with all components of a policy decision. For example, some interventions may have robust evidence of impact but considerable uncertainty associated with the generalisability of the evidence to a new context, or with the scalability of the intervention. We discuss our approach to measuring and reducing uncertainty in policy decisions and its implications for evaluation and research. The overall aim of this work is to make evidence-based decision-making more effective and applicable to a wider range of problems.

This paper explores an innovative assessment framework for measuring children’s formal and informal mathematical knowledge. Many existing standardized measures, such as the Early Grade Mathematics Assessment, measure children’s performance in early primary grade skills that have been identified by researchers and policy makers as foundational and predictive of later academic achievement (Platas, Ketterlin-Geller, & Sitabkhan, 2016; RTI International, 2014). However, these standardized assessments only provide information on children’s mathematical ability as it pertains to skills and concepts that are a focus of school instruction, referred to as formal mathematics. While valuable, they leave unmeasured the mathematics that children use and develop as part of their everyday life, such as the strategies they use to solve simple arithmetical problems that arise as they move through their day (Khan, 1999; Saxe, 1991; Taylor, 2009). In this article, we draw from mixed methods studies which focus on capturing the informal mathematical skills that children develop outside of school in various contexts (Guberman, 1996; Nasir, 2000; Sitabkhan, 2009; Sitabkhan, 2015). We describe how the use of observations of children’s mathematical activities in natural settings and in subsequent cognitive interviews using mathematical tasks derived from those observations, can illuminate mathematical knowledge and skills that may otherwise remain hidden. We found that an assessment framework that focuses on both standardized measures of formal mathematical learning and contextualized measures of children’s everyday mathematics can provide a more complete and nuanced picture of children’s knowledge, and taken together can inform the development of curricular materials and teacher training focused on early learning.

This brief presents Early Grade Reading Assessment (EGRA) data from a randomized control trial conducted in conjunction with the USAID/Uganda School Health and Reading Program (SHRP). SHRP supports reading in 3,700 government schools working through the systems that support these schools and was the catalyst for reform efforts which now cover 80% of Uganda's government primary schools. The assessment collected baseline data at the beginning of Primary 1 for 12 language cohorts and follow up at the end of every school year. These data collected in October, 2017 track progress at the school level (and differences between program and control schools) until the end of Primary 4, Primary 5 or Primary 6 (depending on the year the language was phased into the program). Learning for Uganda and beyond • Registering significant programmatic gains in reading achievement takes time. In the case of Uganda, this is partially attributable to the low baseline levels of reading (94% of learners across the 12 languages could read no words in English at the beginning of P1) and other systemic challenges within Ugandan schools not uncommon in other settings including teacher and learner absenteeism. • When learners learn to read in a local language, the gains realized in the local language are transferred to learning to read in English Major messages - When children learn to read (decode) in their local language, they are able to transfer this skill to decoding English. However, more work needs to be done to improve English comprehension. - Although SHRP has been able to move more learners to higher reading levels and they are on their way to becoming fluent readers, there are still too many learners not acquiring foundation skills. - It takes more time to move children from the foundation threshold than anticipated but when you move them from that level, the gains take off as can be seen from P3 onwards. - Reading gains in large scale interventions working through government structures take time due to systemic and contextual challenges.

CIES 2018 Presentation, given by Tara Weatherholt. A current challenge in the equitable access to quality pre-primary education for young children is the current lack of relevant and reliable data on pre-primary education to inform improvement of the education sector in developing country contexts. In 2015, the global Measuring Early Learning and Quality Outcomes (MELQO) initiative was formed by UNICEF, UNESCO, the World Bank, and the Brookings Institution to focus on facilitating feasible, accurate, and useful measurement of young pupils’ development at the start of primary school as well as the quality of pre-primary learning environments, particularly in low- and middle-income countries. The first nationally representative school readiness study utilizing the MELQO suite of instruments was recently completed in Tanzania. This presentation will report on the quality of pre-primary education environments in both Mainland Tanzania and Zanzibar, as found through the national MELQO study conducted in early 2017, and how this may inform sector improvement.

CIES 2018 Presentation, given by Yasmin Sitabkhan. The purpose of this presentation is to explore the applicability of the direct instruction (DI) model (also known as the “I do, We do, You do” model) for early grades mathematics instruction in low-income contexts through data gathered on two projects, the Tayari Early Childhood Project in Kenya and the Liberia Teacher Training Project (LTTP2). The results from the two studies point to the effectiveness of the gradual release of responsibility model for early mathematics instruction. The model as it is, with an explicit I do, we do, and you do” section for each concept, does not align with best practices in early mathematics instruction. However, the workshop model, which essentially is a “we do, you do” model, may foster more productive instruction that encourages critical thinking and problem-solving behavior.

CIES 2018 Presentation, given by Ben Piper. While there has been recent evidence pointing to the efficacy and cost-effectiveness of pre-primary interventions, there remains little known about how those interventions might work at medium or national scale, nor whether the theoretical impacts of the programs increase, decrease or remain steady over time, nor whether the gains persist into primary school. The endline results from two years of Tayari’s intervention provide longitudinal data on nearly 2500 children who have been evaluated across a range of early learning skills (literacy, numeracy, socio-emotional) at three data points, including their first year of Grade 1. The results presented will explain how Tayari program impacts change over these three data points and determine whether Tayari’s impact persists into Grade 1.

CIES 2018 Presentation, given by Ben Piper. To improve the quality of preprimary education in Kenya, the Ministry of implemented the Tayari program in collaboration with four county governments. The program works with government officers to train teachers and support them in the classroom on a daily basis. Teachers rely on Teachers’ Guides developed to guide them in their lessons and are encouraged to expand children’s learning through two core strategies: the use of materials and extended learning. This presentation details one way in which reading and mathematics can be intertwined on a project through consistent instructional messages in trainings, materials for teachers, and support visits.

CIES 2018 Presentation, given by Ben Piper. The Kenyan Ministry of Education (MoE) in collaboration with RTI International and the county Governments of Laikipia, Nairobi, Siaya and Uasin Gishu have been piloting the Tayari Early Childhood Development and Education (ECDE) model since 2014. Built on a Randomized Controlled Trial (RCT), RTI has been tracking a cohort of learners from both treatment 2 and control schools over the last two years: The baseline data was collected in January 2016; midline in October 2016, at the end of the academic school year; and endline a year later in October 2017. The overall effect of Tayari showed a larger effect at midline – after ten months of implementation – than was expected at end line, i.e. after two years of implementation. The gains in the Tayari group was about 50% higher than in the control group.

CIES 2018 Presentation, given by Sam Ngaruiya. The Ministry of Education (MoE) with technical support from RTI International are collaborating with four county governments to pilot the Tayari ECDE programme. The objective of Tayari is to develop a cost- effective, scalable, low-cost and quality ECDE model. To maximize the impact of Tayari, the MoE intends to scale-up Tayari. However, the ability to take-up and sustain Tayari rests on the counties’ capacity to ensure a viable ECDE system and meet the critical cost of the model. The ability of the counties to take on a Tayari-type programme requires ample institutional, systemic, organizational and fiscal capacity of the counties to take-up and sustain the model. MoE statistics show that, whereas, access has improved, the quality is affected by non-provision of curriculum-aligned instructional materials in the counties and unclear policy in hiring and continuous professional development of the ECDE officers and teachers. Equally, there are policies at the national level that can be used as guidelines for implementation and management of ECD, but there are still some gaps in policy guidelines for implementation; a study mapping the four Tayari counties and three additional counties found increased investment in ECD infrastructure by the governments This presentation explores how the umbrella policy from the MOE as well as evidence from the Tayari pilot programme’s longitudinal study has informed practice within the four Tayari counties, enabling them to be more responsive to the dynamic policy environment. It also showcases initiatives put in place by the counties to sustain the Tayari implementation in pilot counties.

CIES 2018 Presentation, given by Rehemah Nabacwa. This evaluation looks at a large scale reading program in Uganda using a randomized control trial to look at the impact of the School Health and Reading (SHRP) program. This study combines a rigorous external evaluation methodology with the use of a range of 12 different language-specific literacy assessments applied in the 12 languages and communities supported by SHRP. We conclude that significant reading gains are possible in complex, large scale mother tongue reading programs, even within the complex language settings of Uganda. The findings point to language complexity as an important predictor of reading acquisition.