Monitoring and managing inputs at scale is not easy. How do you know if you are training the right teachers, if they are using their new skills, or if the books are getting into the hands of the learners?
For national reading reform efforts in Uganda–which include the USAID/Uganda School Health and Reading Program (SHRP) and Literacy Achievement and Retention Activity (LARA – both RTI implemented) and the Uganda Teacher and School Effectiveness Project (UTSEP)[1], implemented by Global Partnership for Education --adapting to improve results is the foundation of success, and at the core of everything we do. More specifically, collaborating, learning and adapting (CLA) includes regular program reviews, "all hands" technical meetings, before and after action debriefs, “stocktaking” in crucial programmatic areas and a robust research agenda which have led to dozens of program modifications. These activities guide the development of the national reading reform and improved reading achievement in the early grades of primary school in Uganda.
In this commentary post, staff of SHRP have captured their approach to building a unique and effective monitoring and evaluation (M&E) culture that permeates their program.
CLA really is just plain best practice M&E
Components of a CLA framework
CLA helps address the need to bring people together to identify challenges and look for solutions using research-based evidence in a coherent framework. Though CLA really is just plain best practice M&E, it provides a framework to help programs and organizations operationalize the CLA concepts. We have used every subcomponent in the CLA framework, creating a culture of openness attuned to continually learning and improving, with the genuine belief that this leads to a better program and a wiser use of resources.
Continuous learning and improving: We are continually asking ourselves “how can we be doing things better?” M&E brings us together as teams for regular program reviews where we take stock of where we are. This includes looking at activities outlined in the work plan, as well as what is going on in Uganda and within the education sector in general. Teams guide their thinking towards a theme such as “accelerating reading achievement” or, more recently, “bridging to sustainable systems.” With this theme in mind, teams come together with ideas and leave with a list of concrete, actionable steps to be taken, along with dates and roles assigned that will get us further in realizing our goal.
External collaboration: Collaborating is crucial, internal and external. Collaborating improves learning, better ensures that the challenges identified are the most salient, and helps guarantee that identified adaptation and change take place as a result of the learning. We have benefited from learning from our external evaluation partner NORC at the University of Chicago and we have served on joint Ministry of Education and Sports (MoES) research teams. We have always relied on a mode of “leading from behind” and local capacity building working through local systems. Our learning and research efforts are no different. We support the MoES to identify challenges (particularly low reading levels in one geographic area, for example) and look for answers together. Our steady collaborative efforts have led to the Uganda National Examinations Board undertaking their own Early Grade Reading Assessment (EGRA) with minimal external technical assistance under the UTSEP reading project.
Pause and reflect: Targets and deadlines often seem to dictate fast action. M&E in both the SHRP and LARA programs encourages “taking the time to do the careful work”: working hard, setting ambitious targets and deadlines, while also taking the time to pause on the way to activity completion and post-completion. All three education programs include massive teacher training components – together training a combined 10,000 teachers in a one month period before the beginning of the new school year. In addition to coordinating the training for the three programs, we hold a joint “after-action” debrief focusing on what went well, what did not go well, and what actions need to be taken before the next training to ensure an even better training. Some of the changes include a pre-training checklist and visits to training colleges to ensure that basic facilities were up to standards.
M&E for learning: M&E features centrally within all of the reading programs, beyond ensuring that activities are on track and that the programs are doing what they are intended to do. M&E leads program planning and brings technical teams together to discuss synergies between activities (materials development and teacher training, for example). M&E staff participate in all program activities, immersing themselves in the technical programming so they can better understand how best to monitor and create learning while at the same capitalizing on the moment for learning. Staff might manage a teacher training site and, at the same time, collect data on teacher transfers and teacher attrition.
Decision making: We want to make the most informed decisions possible, informed by experience, information and data. Decision makers and decision influencers need to be part of the CLA process. They need to be consulted and informed but the culture also needs to allow for decisions to be made at all levels throughout the chain of command. Scenario planning is an important tool for decision making that we use regularly—laying out costed options to guide the decision making process and promote adaptive management.
CLA has turned teams into questioners who are not afraid to fail…
Impact of CLA on SHRP and the reading programs in Uganda
CLA has turned teams into questioners who are not afraid to fail and this spirit is carried to our undertakings in the MoES, at the districts and at the schools. We have seen tremendous change even at the school level, where school inspectors used to visit schools only to find fault and fill in reports. Now, they go to schools to work with the teachers, discuss challenges and ways to overcome them. We have also seen more collaborative discussions and efforts.
Field assistants are the regular face of our program in the schools. Before the start of every school term, the technical teams meet with them to discuss challenges and share new approaches. With the CLA approach, field assistants have become more open in sharing challenges they face. One such challenge was a need to be more strategic about how we used parent-teacher and school management meetings to improve parent and school support of reading. With field assistant’s input and testing, we have since developed talking points and a parent “learner check” that are used to stimulate conversations with parents about their children’s reading.
We started to hold joint “M&E teas” which are collaborative platforms that provide a time for staff from SHRP and LARA to meet in an informal setting to discuss programmatic issues. The first tea focused on CLA with a particular emphasis on “adapting”. It was an opportunity to determine why, in some cases, adaptation doesn’t happen and to work together to ensure that it would be more likely to happen in the future. We also organized affinity groups around technical topics such as database development and use. The SHRP team developed a database that cut the time required to create teacher training lists from two weeks to two days, and the development of book production and packing lists from three weeks to a few hours. The database affinity group is working on adapting the database for LARA. 
CLA has helped the program ensure that:
- We reached the right teachers with training and that the teachers were more likely to stay in the classroom and use their new skills;
-
We produced and delivered the right number of books to the right schools and that the books were being used appropriately in the classroom;
-
And that learners were more likely to be reading. [See figure at right, from the Cluster 1 EGRA results, and more Uganda M&E reports and briefs here.]
CLA is a mindset, a way of doing business and a culture around it needs to be cultivated throughout programs and processes. SHRP was fortunate to have very supportive USAID counterparts. USAID/Uganda was receptive to and encouraged programmatic changes to work plans and strategy. The program also benefited from supportive COPs and program staff who believed that M&E was central to successful programming and that a high priority should be placed on planning and strategizing. Another enabling factor is strong M&E staff who see themselves as program “animators,”who recognize the M&E function as supportive, but also strategic and catalytic, and who help find those learning moments and time for reflection.
CLA is a mindset, a way of doing business, and a culture around it needs to be cultivated.
Advice about using CLA
Collaboration is key. Hearing from a wide circle of voices when planning and implementing improves the chances that appropriate changes can be envisioned and carried out. Though "adapting" may seem to be at the end of the process, it needs to be considered throughout. It is important to ensure that whoever will ultimately be responsible for adapting is part of the collaborating and learning. Few people (including dedicated passionate professionals) want to have learning handed to them with the intention that they do something with it.
Data gathered to envision the changes needs to be specific enough to lead to a specific change. General data and information can only lead to general actions. A crucial step is to do enough research to really understand the problem. For example, this statement “teachers are not following the reading methodology” is less helpful than “teachers are not allowing for individual reading practice during the reading lesson.” This specificity can be gleaned through many in-depth lesson observations.
Actions for change need to be spelled out very clearly. What may look like lack of motivation or interest in making a change can actually stem from a lack of clarity on how exactly to enact the change.[2] Actions are not always obvious. Teams need to sit down and discuss possible remedies for an identified challenge (the challenge having been identified collaboratively), agree on the way forward and discuss what the change would look like in practice. Again, the statement “schools have not received books” is less helpful than “M&E will share the list with logistics who will follow up with the distribution company to ensure these specific schools receive books by this date.”
Since the beginning of the program, SHRP has always been looking for ways to do things better. These “better ways” were the jumping off point for LARA and UTSEP – partners were able to channel learning from SHRP into the development and implementation of the national reading program. LARA and UTSEP used the same reading methodology and materials that had been developed under SHRP, but they were not adopted wholesale – they were adapted based on lessons learned, making them even better.
For more information please contact:
Tracy Brunette (tbrunette@rti.org)
Rachel Jordan (rjordan@rti.org)
Rehemah Nabacwa (rnabacwa@rti.org)
