A Monitoring, Evaluation, and Learning (MEL) Framework for Technology-Supported Remote Trainings [CIES Presentation]

Existing research on the uptake of technologies for adult learning in the global South is often focused on the use of technology to reinforce in-person learning activities and too often involves an oversimplified “with or without” comparison (Gaible and Burns 2005, Slade et al. 2018). This MEL Framework for Technology-Supported Remote Training (MEL-Tech Framework) features a more nuanced perspective by introducing questions and indicators that look at whether the technology-supported training was designed based on a solid theory of learning; whether the technology was piloted; whether there was time allocated to fix bugs and improve functionality and user design; how much time was spent using the technology; and whether in-built features of the technology provided user feedback and metrics for evaluation. The framework presents minimum standards for the evaluation of technology-supported remote training, which, in turn, facilitates the development of an actionable evidence base for replication and scale-up. Rather than “just another theoretical framework” developed from a purely academic angle, or a framework stemming from a one-off training effort, this framework is based on guiding questions and proposed indicators that have been carefully investigated, tested, and used in five RTI monitoring and research efforts across the global South: Kyrgyz Republic, Liberia, Malawi, the Philippines, and Uganda (Pouezevara et al. 2021). Furthermore, the framework has been reviewed for clarity, practicality, and relevance by RTI experts on teacher professional development, policy systems and governance, MEL, and information and communications technology, and by several RTI project teams across Africa and Asia. RTI drew on several conceptual frameworks and theories of adult learning in the design of this framework. First, the underpinning theory of change for teacher learning was informed by the theory of planned behavior (Ajzen 1991), Guskey’s (2002) perspective on teacher change, and Clarke and Hollingsworth’s (2002) interconnected model of professional growth. Second, Kirkpatrick’s (2021) model for training evaluation helped determine many of the categories and domains of evaluation. However, this framework not only has guiding questions and indicators helpful for evaluating one-off training events focusing on participants’ reactions, learning, behavior, and results (as is the focus in Kirkpatrick’s model) but also includes guiding questions and indicators reflective of a “fit for purpose” investigation stage, a user needs assessment and testing stage, and long-term sustainability. Furthermore, this framework’s guiding questions and indicators consider participants’ attitudes and self-efficacy (based on the research underpinning the theory of planned behavior), as well as aspects of participants’ post-training, ongoing application and experimentation, and feedback (Clarke and Hollingsworth; Darling-Hammond et al. 2017; Guskey). Lastly, the framework integrates instructional design considerations regarding content, interaction, and participant feedback that are uniquely afforded by technology.

Project Reconnect - Final Report

This is the final report for Project Reconnect, a bold and ambitious effort conceived by Google Germany, Google.org, and NetHope in late 2015 that aimed to to help refugees as they rebuild their lives in Germany by facilitating access to online education, language learning, culture learning, and information resources. This report was written by RTI International and based on data from 50 grantee organization reports, 320 Chromebook location managers, and 304 refugee Chromebook users, highlighting achievements and lessons learned from the program.

eResources Review Toolkit

This toolkit was developed for the All Children Reading Philippines project to support Department of Education staff in developing and reviewing electronic resources for the DepEd Commons (online OER repository).

Mobile Learning and Numeracy: Filling gaps and expanding opportunities for early grade learning [Arabic]

The present study on Mobile Learning and Numeracy examines how mobile learning (m-learning) could influence and improve numeracy education at early grade levels (ages 4-10) especially in low-income countries. Key questions to guide the research include: 1) What are the benefits and challenges of integrating mobile learning into early grade numeracy education? 2) What is the role of a teacher with regard to mobile learning and numeracy education? 3) How can the community and the parents actively contribute to/participate in the child’s numeracy education with the use of mobile devices? and 4) How can mobile technology be used effectively in measuring/assessing numeracy gains? The conclusions and recommendations of this study have been informed by an international working group that met over two days during the first International Numeracy Conference in Berlin in December 2012. We would like to acknowledge the following participants of this working group for their thoughtful contributions: Michaela Brinkhaus (BMZ); Dorothea Coppard (GIZ); Melanie Stilz (Konnektiv Büro für Bildung und Entwicklung); Jens von Roda-Pulkowski (KfW); Abigail Bucuvalas (Sesame Workshop); Mr. Kann Puthy (Primary Education Department, MoEYS Cambodia); Edward Barnett (DFID).

Virtual Assessment and Making the Right Technology Choices (Presentation)

This presentation was held by Carmen Strigel during the second webinar of the Basic Education Coalition EdTech working group on April 27, 2020. The presentation is about using Tangerine for student self-study and self-assessment as well as family outreach. The presentation also introduces a new tool developed by RTI on considering access, user engagement, and content in making the right technology choices for your audience.

Going Virtual: Content Delivery Decision-making Tool

This tool was created to support projects who are considering delivering educational content remotely. The decision tree supports multiple aspects of instructional design and accessibility to suggest specific authoring tools and delivery platforms.

Education Policy and Strategy for Scaling EdTech in Philippine Schools

This topic brief was prepared by Liezl F. Dunuan, under a subcontract issued to the Foundation for Information Technology in Education (FIT-ED), Philippines. It is based on document review and interviews with officials and staff of the DepEd’s regional and division offices in the Cordillera Administrative Region and Baguio City, DICT, DOST-Science Education Institute, the United Nations Development Programme Philippines office, two private telecommunications companies (Globe and Smart), two private EdTech service providers, and two non-profit organizations working in EdTech. Relevant policy documents were also reviewed. Carmen Strigel (RTI International) contributed to the international policy analysis section.

Assessing Soft Skills in Youth Through Digital Games

The acquisition and use of so-called “soft skills”, including problem solving, resilience, and self-regulation have been associated with better performance at school and in the workplace [1], [2]. Problem-solving is defined as the ability to acquire or use prior knowledge in order to solve new problems. Strengthening this skill is a concern to educators and employers as the 21st century labor market is increasingly unpredictable and requires skills that go beyond mastering and executing familiar processes. Students need to identify and solve problems that they have never encountered before, formulate a solution plan specific to that problem, and execute the plan. Thus far, the body of research that has measured these relationships relies on traditional self-reporting measurement questionnaires. This methodology is prone to bias since youth may respond in a way they know is desirable, rather than the way they actually behave [3]. Stealth assessment attempts to gather more authentic measurement of skills by asking children to demonstrate them in a structured environment where data collection is unobtrusive [4]. Digital games can be used for stealth assessment, with data on decisions and strategies collected in the application during game play. Since 2017, RTI has been developing games that target a range of soft skills by simulating real-world tasks in a virtual environment. The game designed to measure problem-solving skills gathers metrics on task completion, time management, accepting instruction, problem identification, solution identification, and self-regulation. This paper describes the multi-year process of development and testing of this game, the results obtained from pilots in the Philippines and Morocco, and the implications for strengthening problem-solving skills among youth worldwide. Cite this paper: Pouezevara, S., Powers, S. Strigel, C., McKnight, K. (2019). Assessing soft skills in youth through digital games. ICERI2019 Proceedings. 12th Annual International Conference on Education, Research and Innovation (ICERI), Seville, Spain. p. 3057-3066. https://doi.org/10.21125/iceri.2019.0774

Chile Country Report: Scaling Access and Impact - Realizing the Power of EdTech

This series of reports was produced by Omidyar Network’s Education initiative, whose mission is to unlock human potential through learning by catalyzing people, ideas, and systems – so every individual thrives and contributes in a changing and interdependent world. The Omidyar Network team included Eliza Erikson, Erin Simmons, Rebecca Hankin, and Eshanthi Ranasinghe. The data underpinning this report come from interviews, surveys, site visits, and desk research by a team of researchers and EdTech practitioners led by RTI International, drawing on local expertise in each of the case study countries. The team conducted more than 100 interviews with teachers, school principals, education administrators, policymakers, and EdTech experts. This study sought to understand the conditions that have thus far enabled EdTech initiatives to scale in Chile. This case study is the result of more than 20 interviews and site visits and a document review conducted in Chile over a 2-month period in 2018. The case demonstrates that long-term, top-down vision implemented in partnership with university networks or other NGOs who specialize in adaptive management, active learning and knowledge sharing.

Inclusive Education Screening Tool and Pilot Activity – Desk Review

Children with disabilities are less likely to be enrolled in school in low- and middle-income countries in Asia and beyond. Further, in most school systems in these countries, children with disabilities are not identified as having a disability and thus do not receive necessary support, even when the country has an inclusive policy education. Reports and studies indicate gaps in the identification of children with disabilities in school systems, gaps in teacher professional development and support, and a lack of specialized materials and resources. This desk review aims to provide critical information to policymakers and practitioners in US Agency for International Development (USAID) focal countries in Asia on the landscape of inclusive education for children with disabilities. Four main sets of research questions guide this desk review: 1) What are the disability prevalence rates in USAID-supported countries across Asia, and where do gaps exist in available data on students with disabilities? 2) What are the primary and secondary school completion rates of children with disabilities? 3) What policies exist, how do they vary across countries in USAID-supported countries in Asia, and are they being effectively implemented? 4) What is the education landscape for children with disabilities in USAID-supported Asian countries? What interventions to date have shown promise in improving aspects of inclusive education for children with disabilities, and what are the costs associated with implementing such programs?