Evaluation

CIES 2018 Presentation, given by Peter Muyingo. Education programming worldwide is beginning to include dedicated activities to support a positive school culture. There is evidence, mostly from high income countries, that a positive school climate is associated with improved learning outcomes and attendance, and reduced violence prevalence. In this presentation we will discuss some of the challenges we faced in adapting a school climate survey from the United States for use in an impact study for a program in Uganda that has a dedicated focus on building a positive school climate. The school climate survey was adapted as one of a variety of instruments to be used in the impact evaluation of the USAID/Uganda funded Literacy Achievement and Retention Activity. This Activity, a working partnership with the Uganda Ministry of Education, focuses on improving early grade reading and retention in 28 districts and 2698 schools in Uganda.

CIES 2018 Presentation, given by Matthew Jukes. There is an increased demand for assessments of social and emotional competencies of young children in low- and middle-income countries. These competencies are increasingly seen as important for children’s development and for their education. In the context of preschool and primary education, such assessments have a number of uses. They are used to evaluate the impact of programs on children’s social and emotional learning. They can also be used to monitoring individual children’s progress in such programs and to tailor interventions to their needs. We developed a tool to assess aspects of SEL that are important for children’s education in Tanzania. Using this work as a case study, we describe the challenges inherent in developing such a tool. The tool was developed as part of the USAID Tusome Pamoja preschool program in Mtwara, Southern Tanzania.

CIES 2018 Presentation, given by Matthew Jukes. Randomized controlled trials (RCTs) have been used increasingly in an effort to reduce uncertainty and improve rigour of impact estimates. However, RCTs do not provide certainty in other important aspects of the evaluation, such as its external validity and in understanding which components of a complex program are essential for its effectiveness. This paper reviews two additional approaches which are currently underutilized which may help researchers and policy makers make better use of uncertain evidence.

CIES 2018 Presentation, given by Luis Crouch. Given the prominence of learning and quality in the SDGs, much discussion has gone into how to measure in a manner that is reasonably comparable. Some have argued (or feared) that this would necessitate a single, dominant, global assessment. Aside from the political or ethical acceptability of this kind of imposition, one has to wonder how meaningful this could be, psychometrically or pedagogically. The paper will argue that unless one were to increase the cost of assessment tremendously, or make children sit through lengthy assessments or until highly adaptive computerized assessments can be used, a single or a few dominant assessments are an unlikely approach. Instead, a variety of assessments is a more likely solution. These might have better psychometric “resolution” for poorer countries with greater cognitive inequality than the OECD countries, might be closer to the children’s actual levels, and might be psychometrically more reliable if done properly. The ability to make the results comparable or equitable in some sense need not be lost, however, if some sort of universal learning scale is created, so that countries can peg themselves, with reasonable rigorous, to that scale. A global neutral arbiter can “sponsor” the scale and compile country-based reports using it.

This presentation was prepared for UNESCO Mobile Learning Week, 2018. It describes an RTI International internal research program to learn whether mobile gaming can be used to assess soft skills important for employability. The presentation was created by Lee Nordstrum and delivered by Sarah Pouezevara.

While the SDGs now officially call for global reporting on learning outcomes, many institutions and scholars had noticed, at least since the mid-2000s, that many children were not learning much, and were starting to respond by, as a first step, advocating and developing assessments that, they felt, was perhaps more appropriate to learning at the bottom of the pyramid, or at the left end of the cognitive distribution. The GMR sounded a clarion with their estimate that there are some 250 million children in the world hardly learning. This paper addresses an issue that can be put simply but is extremely hard to answer: “Is the array of assessments emerging helping researchers, policy-makers, and implementers get a more accurate sense of how much or how little the poorest children in the world know, and is it helpful in remediating the situation?” The paper specifically is not addressed at the question of global reporting—although it does touch upon the issue. The problem of interest here is what is most useful for countries to generate movement along the bottom of the pyramid.

Executive Function (EF) refers to a range of cognitive skills that are key to problem solving and a child's ability to “learn how to learn.” These skills experience an important growth period during early childhood; thus, between 2005 and 2010, researches at the University of North Carolina designed a battery of tools to measure these skills in pre-school aged children known as EF Touch. In 2016, RTI International – a research institute dedicated to evidence-based approaches to education policy and practice - identified EF Touch as a unique opportunity to integrate early childhood EF measurement in its battery of existing international education assessment tools. EF Touch was originally developed by UNC in note card format; then adapted to a paper and pencil format; and later transitioned to laptop and touch-screen monitor “station” requiring two assessors and an expensive third party scoring software. Although the laptop mediated approach was an improvement on paper-based administration, it was still clear that this model was not sustainable, portable, or cost effective for large deployment in the low and middle income (LMIC) contexts where RTI most often works – and where very little research on EFs in young children has been thus far conducted). Therefore, RTI sought to adapt Tangerine® - an open source software designed by RTI for low cost, large-scale, offline assessment – for use in the administration of EF Touch. More than 60 organizations worldwide have used Tangerine on tablets to conduct 1,500,000+ surveys in 70 countries and 100 languages. To accommodate the specific requirements of EF Touch, the Tangerine platform needed extensive modification to accommodate the specific needs of EF touch, including new A/V capabilities, unique millisecond time-stamps indicating reaction time, plus the development of complex skip logic commands tailored to individual performance. The resulting Tangerine-mediated battery of EF Touch tasks were designed to remain scientifically rigorous while significantly reducing costs (one assessor, one tablet); increasing scalability in developing country contexts (offline assessment); and reducing the burden of complicated data entry (data uploaded directly to the server). This presentation provides an overview of EF touch, including the adaptation process to Tangerine and recent results from two field deployments in Kenya. The attached presentation does not contain video files to due the attachment size. The full presentation with videos of task demonstrations can be found via the included google drive link.

The Institute for People’s Education (Institut pour l’Éducation Populaire, or IEP) designed the Read-Learn-Lead (RLL) program to demonstrate that the new official curriculum, if properly implemented and supported, can be a viable and effective approach to primary education, using mother tongue and a very specific pedagogical delivery approach. The RLL program sought also to demonstrate how the new Curriculum can be effectively implemented and supported, and what resources are needed to do so. RLL offers students and teachers carefully structured and systematic lessons, activities, and accompanying materials for instruction and practice on critical early reading skills in mother-tongue medium during the first years of elementary school. It is organized around three programmatic “results sets,” the first of which focuses on Grades 1 and 2 and is the subject of the present evaluation. This independent evaluation study, funded through a grant from the William and Flora Hewlett Foundation and carried out by RTI, explored the effectiveness of the RLL program’s Results Set 1 as applied over three school years (2009-2010 to 2011-2012) in the Bamanankan language and in other Malian national languages (Bomu and Fulfulde in all three years, and Songhai in 2009 and 2010).

Management Systems International (MSI) led the Tusome baseline study using multiple data collection methods, including an early grade reading assessment (EGRA); surveys of pupils, teachers, head teachers, curriculum support officers (CSO) and households; and classroom observation. The EGRA assessment tool was developed during the baseline and includes eight subtasks in English and six subtasks in Kiswahili. The midline included developing additional data collection tools, revising the baseline surveys, recruiting and training supervisors and enumerators, administering the tool and surveys in the same sample schools as the baseline, ensuring quality control, establishing the reliability of the assessment tool and analyzing the data. For the midline, the evaluation team assessed pupils from the same 204 schools sampled for the baseline. Through discussions with USAID, MOE and RTI, the evaluation team created the sampling frameworks and set up the design for a national sample in 2015. Using a three-stage cluster sampling procedure from a sampling frame of 22,154 formal public schools and 1,000 non-formal (or Alternative Provision of Basic Education and Training – APBET) schools, the evaluation team drew a clustered, random sample, resulting in a target of 4,896 total pupils comprising 2,448 boys and 2,448 girls divided equally between Class 1 and Class 2. The evaluation team reached the following conclusions: • The Tusome approach is having a strong, positive influence on reading outcomes, with relationships between project implementation and reading outcomes. • Reading outcomes for Class 1 and 2 pupils greatly improved during the one-year period between the baseline and midline evaluations. While impressive gains have been made, continuing with the Tusome approach will be critical to sustaining or improving on those gains. The Tusome project has achieved a high level of national implementation of the activities at each level of the education system. Given that project activities such as CSO observations, in-service training and access to materials are associated with higher ORF scores, the high level of implementation across all schools appears to be a key part of its success. The effect sizes seen during the PRIMR pilot have been at least sustained, and in most cases strengthened, in the national scale-up of Tusome. • The evaluation methodology and implementation resulted in valid, reliable data for the midline evaluation, including the changes from baseline to midline.

A collection of articles on Improvement Science, published in a special issue (Volume 25 Issue 1) of the journal "Quality Assurance in Education". Each article describes one of seven improvement methods, presents its history, and gives at least one example of it employed in the education sector. The seven improvement methods covered by these articles are: - Networked improvement communities - Design-based implementation research - Implementation science - Lean - Six Sigma - Positive Deviants - Deliverology