BetterEvaluation FAQ: How do you use program theory for evaluating systems?

Although it’s sometimes referred to as program theory or program logic, theories of change can be used for interventions at any scale, including policies, whole-of-government initiatives, and systems. Evaluating at systems scale presents particular challenges and scope for using different tools and techniques.

In our recent blog post about using theories of change and logic models better in evaluation, we asked BetterEvaluation members to submit a question or challenge that they have in relation to creating or using theory of change for review by the BetterEvaluation team. One member asked us for advice on using a theory of change for systems change which we are very happy to share in this blog.

In answering this question, I think it’s useful to start by considering whether it’s helpful to think about the system as simple, complicated and complex – in particular distinguishing between what is complicated (lots of components that can be identified in advance) and complex (emergent and adaptive components, actors, goals and roles). (You can find more information about this distinction on our thematic page on complexity.) It might well be the case that it’s most useful to think of some aspects as complicated and some as complex.

Thinking about it as a simple system

One way of using theories of change for evaluating systems is to treat them the same as a project and use a very simple version of a logic model: inputs -> activities -> outputs -> outcomes -> impacts.

In the early days of the American Evaluation Association Topical Interest Group for Systems in Evaluation, we used to get a number of proposals that were about evaluating a health system, or an education system, or a justice system, but which really address the fact it was a system and not a simple project in how it was evaluated.

For example, if you were evaluating a new professional development intervention for teachers that is intended to improve the school system, especially in terms of reducing school drop-out rates, you could do simple logic model something like this:

The problem with this is that it doesn’t really acknowledge and address any systemic issues. While it hints at two systems (the education system that is focused on teaching and learning, and the welfare system that is partly based in schools and partly in community agencies) it doesn’t focus attention on how these need to be better connected and how they might need to exchange information and be responsive to each other.

I wouldn’t usually recommend using this type of logic model for evaluating a system, because it leaves so much out. But there are some situations where this might be an important starting point. For example, several years ago when I was working with the Public Service Commission of South Africa, we developed the following diagram to support their work contributing to improving the public service monitoring and evaluation system. Because this had such a large scope, we developed a very simple outcomes hierarchy, focusing on a few key issues, which could provide a common road map for discussions and planning (you can read more on this in the November 2004 PSC Newsletter, page 4).

 

This outcomes hierarchy was intended to focus attention on the behaviour change theories underpinning performance monitoring. They might partly work by identifying problems that can be identified (and many of the monitoring systems focused on this), but they also should aim to work by providing clear signals about what is valued, and by recognising and sharing information about successes – that could then be further investigated to find out how they were achieved. They also cannot work if they only involve reporting data – some process for follow up or response is needed.

This simple theory of change could be then used to plan, manage and evaluate a departmental monitoring system or a system across government.

Identifying and addressing complicated aspects

It’s often much more useful to think about the system as complicated – having lots of components and actors which can be identified in advance and whose contribution to the theory of change can be articulated. There are two main ways of doing this – hierarchical and parallel.

Going back to our school dropout example, a hierarchical theory of change would think through the connections between the classroom, the school and the school district or state. For example, in order for teachers to undertake additional activities with students, the school might need to provide additional resources or change policies or processes; in order for schools to be able to do this, school districts or states might need to also provide resources or change policies or processes.

 

A parallel theory of change would show the educational system needs to connect better with the child welfare system:

 

A few years ago, I contributed to the development of a monitoring and evaluation framework for a whole of government initiative around family violence which connected frontline services, housing, police, and courts. At the heart of the framework was a parallel version of an outcomes hierarchy which showed how different service systems contributed to achieving particular outcomes and therefore needed to be connected.

The following example from the National Health Service in Scotland uses a parallel pipeline logic model, showing how the outputs from different components contribute to producing overall outcomes of reducing tobacco smoking.

Treating it as a complex system

Complicated approaches work when the components and how they fit together can be articulated in advance. But sometimes it is not clear how a system should be defined (what is in and what is out), how it should change and who should be involved in doing this.

Some approaches to systems change do not yet have an explicit theory of change but have identified good practices that speak to an underlying implicit theory of change around connecting different perspectives and information.

For example, in a recent article in the Stanford Social Innovation Review, Mark Kramer discussed five effective practices being used by philanthropic organisations working to bring about systems change:

• First, these funders work both inside and outside the nonprofit sector, leveraging the market forces that drive for-profit companies and making efforts to improve the implementation and outcomes of existing government programs.
• Second, they build common ground among key actors by forging cross-sector coalitions.
• Third, they recognize the importance of the intangible narrative that underlies the public response to our society’s problems and work actively to change that narrative.
• Fourth, they elevate the voice of lived experience in shaping solutions.
• Fifth, they reconsider their own staffing, budgeting, and operations to address racial and cultural blind spots, focus on multifaceted problems rather than separate program areas, and develop more active leadership roles for their CEOs and boards.

(Mark Kramer, Systems Change in a Polarized Countryin the Stanford Social Innovation Review. Apr 11, 2017.)

Julia Coffman has identified elements that are needed in a theory of change for building systems (such as an early childhood system that brings together early learning, family support, health, mental health and nutrition and special needs/early intervention):

Another way of thinking about theories of change for systems change is to use specific theories of systems change. For example, the Berkana Institute Theory of Change draws on the two loops model which shows how the old system might change to a new system:

As one system culminates and starts to collapse, isolated alternatives slowly begin to arise and give way to the new.

In this video Deborah Frieze, Berkana’s former co-president, explains the two loops theory and speaks about the way that our work to name, connect, nourish and illuminate has fit into this model. She also identifies some of the different roles we might play to hospice the dying system, usher in the alternative system and make clear the choice between the two.

(The Berkana Institute, Our Theory of Change)

An example by Catherine Howe of a theory of change for systems change that’s expressed in the form of three principles:

1. Map your system levers and maintain this as a dynamic map
2. Cultivate and curate a network of active participants who want to work together differently
3. Create some kind of engine of change which can be brought to bear on system-wide problems which will further develop the system change map as well as the network of active participant

(Catherine Howe, Systems Thinking: Theory of Changein Curious Catherine Blog)

Catherine's diagram of a principles-based theory of change such as this looks very different to the classic pipeline model or an outcomes hierarchy:

 

Some points to consider when using program theory for a system

What are considered the boundaries of the system? Are there different views on this? What might be the implications of drawing the boundaries tighter or wider?

Is it most useful to think about the system overall as simple, complicated, or complex? Are there some aspects that would be useful to think of differently (for example, a simple aspect of a complex system, or a complex aspect of a complicated system)?

What do you think?

Are these options and examples for theories of change for systems change helpful? Do you have other techniques and examples to share? We'd love to see some of your own ways of representing a theory of change for systems change - contact us or Contribute Content via the website.

[UPDATE: 11:00am 23.06.17 AEST)

Thank you to those who've been in touch with us to share your thoughts or questions on this blog. In response, we'd like to add some resources for those who are interested in reading more about systemic thinking in evaluation. See Related resources below.