Put simply, education development can be described as work dedicated to the incremental betterment of learners’ experience with and success in educational systems in lower-income settings. It is about making educational systems, and all the many moving parts that comprise them, work better to reliably produce quality outcomes for all children. These truisms would seem to make education development work fertile soil for the use of improvement science methods, which aim to use disciplined inquiry and scientific methods to make social systems work better for the people they serve. Though a nascent discipline in education, the goal of improvement science seems to go hand-in-hand with the goals of education development.
Image: RTI International (2007) Evaluating the use of ICT for education under the Jordan Education Initiative.
And yet, we in the education development field must humbly admit that much of our work may not match up to the lofty goals stated above. Rather, much of our work is about developing “new parts” to educational systems: producing or introducing new technologies, writing and introducing new curricula, building teacher training and support systems, distributing learner support materials, or evaluating existing programs. These actions make sense, particularly when these parts are broken or missing entirely from educational systems in which we have the honor to work. However, the addition of “new parts” will not, by itself, ensure that those parts function as a coherent system. These parts will not inherently shed light on how people, roles, materials, social norms, and processes interact to produce educational outcomes, identify instances where performance is lower than desired, or determine through disciplined inquiry how to ameliorate those instances of undesirable performance. As a result, we in the education development field could learn from the methods of improvement science, and attempt to apply these lessons (and, where appropriate, these methods) in contexts in which we work.
Improvement science methods aim to use disciplined inquiry and scientific methods to make social systems work better for the people they serve.
Improvement science is a catch-all term that describes a set of systematic methods which aim to improve outcomes for users of the systems – or, in educational terms, students – and to learn how to continuously improve. Improvement science should also be differentiated from both pure research and audit (or inspecting for fidelity of implementation). It has been defined as: “a body of knowledge that describes how to improve safely and consistently. Improvement science is not the same as research. Research is designed to find out what is possible. Improvement science is not the same as audit. Audit is designed to find out what is actual. Improvement science describes how to reduce the gap between what is actual and what is possible” (Health Foundation, 2011: 6). Shojania and Grimshaw (2005) add that quality improvement efforts made by organizations should be based on a high warrant of evidence. In other words, strategies for the utilization and adaptation of evidence-based improvement methods should themselves be based on a foundation of evidence. In this sense, improvement science seeks to discern what works best for addressing a particular problem, for whom, and under what set of specific conditions (Berwick, 2008; Bryk, Gomez & Grunow, 2010). It represents a field of study focused on the methods, theories, and approaches that facilitate or hinder efforts to improve quality in context-specific work processes, and centers inquiry on the day-to-day “problems of practice that have genuine consequences for people’s lives” (Bryk, 2009: 598; Health Foundation, 2011).
Improvement science first originated not in education, but in industry, then was translated to healthcare. Recently, a handful of educational organizations have begun to implement specific improvement methodologies, though to date this has mostly (though not exclusively) been in high-income countries, particularly the United States. While these locations and contexts differ substantially from those in which we work, there are, at least, several lessons that we can glean from the methods of improvement science.
- Decentralization of improvement work: The goal of many improvement science methods is to enable “frontline workers” (teachers, in education) to identify and solve problems of practice. The frontline workers are viewed, not as system inputs, but as actors with local knowledge and expertise. Improvement methods, such as Networked Improvement Communities, Lean, and Positive Deviance, provide specific tools and procedures to these local experts to identify problems, develop solutions, implement changes, and determine success. This is not the same as letting teachers fend for themselves to ply their craft alone in a classroom; rather, these methods provide opportunities for them to monitor their work, rigorously measure success, and determine how they can continue to improve.
- Understanding variation in learning outcomes: In an education context, we know that there will inherently be some variation in effect across program sites. Instead of focusing on differences in specific outcomes such as average effects on words per minute, we should assess the underlying causes of variation through the lense of improvement science (assuming that there are logical local adaptations to pedagogical practice or school management that can be studied, analyzed, and learned from). The application of the Positive Deviance method in Vietnam serves as an example. Using this method, researchers noted that some households in impoverished communities could adequately feed their children while others (with similar incomes and diets) had malnourished children. Upon investigating what the households of non-malnourished children were doing differently when preparing meals, they found that when scooping up bowls of stew for their children, they scooped deep down into the pot in order to get more of the vegetables and crabmeat. This meant more protein and calories for their kids. Researchers then held sessions where other parents of malnourished children could observe this behavior and practice it for their own children - enabling the “positive deviant” behavior to spread.
- Practical measurement for improvement, not just monitoring: As a field, we conduct a great deal of measurement, most of which is used for aggregation and reporting (projects to implementers, implementers to donors, donors to international organizations, etc.). However, many improvement approaches (Lean, Six Sigma, NICs) push measurement down to the base level of the education system- teachers. Using these approaches, teachers run small-scale improvement projects in their classrooms, defining what they plan to change and articulating how they will know whether that change is an improvement. These definitions are coupled with measurement that is practical, relevant, and not onerous to collect. While different methods have varying standards for what is measured and how (and some are more standardized than others), they all tend to include inputs, outcomes, processes (how inputs yield outcomes), and balancing (unintended and undesired consequences of the improvement). All measures are defined before improvement projects commence, are tracked throughout their duration, and are revisited regularly.
- Formal networks of improvement workers: While classrooms and schools are unique, many of the problems faced by teachers and school leaders are not. Rather, problems of practice are often shared by many others. Some improvement methods seek to create formal networks of professionals who collaborate to identify solutions for commonly defined and experienced problems. After problems are identified and defined, these networks hypothesize possible solutions for implementation on a small scale in their classrooms and schools, and monitor and collectively discuss results. Such networks do not inherently have to be comprised of teachers- schools and larger organizations (districts) could form improvement networks, if faced with a commonly defined problem. An example of this network design is STIR, an organization working in India and Uganda that promotes and facilitates teacher networks to engage in “Learning Improvement Cycles” that provide teachers opportunities to identify problems, implement changes, and evaluate results collectively.
- Scale: There is enormous pressure in education to implement changes at scale immediately or soon after the changes start to show promise. After all, wouldn’t it be immoral to withhold positive aspects of a program from children who would benefit from it? Improvement science methods view the problem differently: scale is not seen as an inevitable rolling out of a promising innovation, but as another variable that causes variation in processes and outcomes. For example, it is relatively simple to control how teachers in a single primary school enact new, ambitious pedagogical moves in their classroom, but becomes much more difficult when that innovation is scaled to 100 schools, or 10,000 across regions. Improvement methods allow changes to start at a very small scale, with controlled scalability of improvements. As innovations are scaled up, they are monitored closely (outcomes, inputs, processes, and balancing measures) for impact and to determine how the innovation is transferred from one location to another.
Improvement methods allow changes to start at a very small scale, with controlled scalability of improvements. As innovations are scaled up, they are monitored closely for impact and to determine how the innovation is transferred from one location to another.
A special issue of Quality Assurance in Education authored by myself with Carnegie Foundation explores the similarities and differences between seven improvement science methods that are increasingly used in education, both in the United States and internationally. The improvement methods covered in the special issue include:
- Networked Improvement Communities
- Design-based Implementation Research
- Deliverology
- Implementation Science
- Lean for Education
- Six Sigma
- Positive Deviance
The volume as a whole describes the seven improvement methods and the specific methods that comprise them, explicate how the method identifies and approaches problems, determines solutions, as well as provides short case studies of how the methods have been deployed in education organizations. It is also meant to help education practitioners determine which method may be appropriate to solve the problems they face, and to understand the methods’ strengths and limitations.
Tools such as this volume can begin to demystify the methods of improvement science and render them useful for practitioners and educators in the field. They may also be of use to the educational development community as we continue to strive to make social systems work better for the people they intend to serve.
View other analytical work by Lee Nordstrum in the SharEd collection.
Image (c) 2007 RTI International.
