How to choose, develop, and support innovation in evaluation

This blog is an abridged version of the brief Innovations in evaluation: How to choose, develop and support them, written by Patricia Rogers and Alice Macfarlan.

It builds on a webinar delivered by Patricia Rogers in May 2018 as a joint project of UNICEF, BetterEvaluation and EVALSDGs. This blog opens up some of the issues and questions about why and how to adopt innovations in evaluation, while the brief goes into further detail about innovations that can be useful in addressing long-standing challenges in evaluation.

Why do we need innovations in evaluation?

Evaluation is challenging.  It is hard to get good enough answers to the big questions it asks, in time to inform decisions, within tight budgets, while ensuring ethical and consultative processes, and meeting the different needs of diverse users and other stakeholders.

And wherever there is a gap between what we can do and what we need to do, that’s when there is a space for innovation.

Monitoring and evaluating progress towards the Sustainable Development Goals (SDGs) has created demands for new information, more information and better ways of using it. This is why one of the measurable intermediate outcomes identified in the EvalAgenda 2020 is “Innovations in evaluation are encouraged and tested”.

What do we mean by innovation?

For the purposes of this discussion, innovation is defined as a change (at least in that setting) that adds value (or is intended to add value) – either doing existing things more effectively or with fewer resources, or doing new things of value.

When we use the word ‘innovation’ we often think of inventions – something brand new, especially technological. But there are other types of innovations that might be useful for you. Transferring or translating a method or technique that is new to your organisation. Gathering together a number of elements into a new approach, or systematising some existing practices that are currently ad hoc and tacit.

Table 1. Types of innovation

Where is innovation needed?

In a recent blog post on key evaluation questions to address the SDGs, Caroline Heider, Director-General of the Independent Evaluation Group of the World Bank, identified the following innovations needed to address the SDGs:

• Assessing inclusion/exclusion and distributional effects of interventions
• Understanding complicated cause-and-effect relationships
• Embedding evaluation in decision-making processes
• Actively supporting learning, including acknowledging mistakes and failures
• Using big data and other technology ethically and appropriately

Importantly, Caroline Heiders' identified areas for innovation cover processes in evaluation, as well as methodology. This is key, as often when we think about innovations in evaluation, we think about innovative ways of collecting data.  But we know there are many different tasks involved in evaluation, from framing its purpose to supporting use of its findings.

In the Rainbow Framework, we have organised information about more than 300 evaluation methods and processes in terms of more than 30 tasks involved in doing evaluation. These cover the whole rainbow of evaluation tasks, from managing evaluations, developing theories of change, framing the purpose and questions for an evaluation, answering descriptive, causal and evaluative questions, and reporting findings and supporting use of them. All of these have challenges and there is scope for innovation in all of them.

In the recent webinar on Innovations in Evaluation (delivered as a joint project of UNICEF, BetterEvaluation and EVALSDGs), participants were asked to identify the broad area where innovation was most needed in their work. Their answers covered all areas, with a concentration in reporting and supporting use.

Figure 1. Webinar survey results: Where is innovation most needed for you?

Considering possible innovations to adopt

Which innovations are likely to be adopted?

Everett Rogers’ classic work on the diffusion of innovations synthesised evidence from hundreds of studies on innovations to identify five factors that were important influences on what innovations were most likely to be adopted:

Table 2. Factors that influence the adoption of innovations (Rogers, E.M., 2003)

1. Relative advantage -  in terms of effectiveness or cost-effectiveness 
2. Compatibility with existing systems
3. Ease of use (and learning to use)
4. Trialability – possibility of doing it on a small scale and gathering credible and timely evidence about relative advantage
5. Potential for reinvention and further adaptation

In the brief, Innovations in evaluation: How to choose, develop and support them, we discuss eight challenges in evaluation and list some 40 innovations that can be used to address these (eight featured innovations that are listed below, and 32 additional options and approaches to consider).

Challenges in evaluation and possible innovations to address these

• Framing the evaluation around intended uses and primary intended users (featured innovation: Data rehearsal)

• Checking for equity effects (featured innovation: Disaggregating data)

• Measuring the hard to measure (featured innovation: Big data)

• Including the effect of other interventions and factors (Triple-row logic models)

• Communicating findings to time poor users (featured innovation: Layering reporting media)

• Involving the least powerful beneficiaries in conducting and decision-making about an evaluation (featured innovation: Children as evaluators)

• Investigating cause and effect when a counterfactual isn’t possible and/or when causality is complicated (featured innovation: Process tracing)

• Making values transparent (featured innovation: Rubrics)

Things to consider when adopting innovations

When thinking about whether or how to adopt these innovations to your own work, we encourage you to consider the following questions:

CHOOSING to adopt a new innovation

1. What are your particular challenges in evaluation?
2. Do you think this innovation might be useful for you to use?  Why or why not? What challenges might it help you with?
3. What other information and assistance would be useful to have in order to make this choice?
4. How might you get this?

USING a new innovation

1. What other information and assistance would be useful to have in order to use this innovation well?
2. How might you get this?

Join the discussion

BetterEvaluation works with the global evaluation community to curate and co-create knowledge about existing methods and approaches in monitoring and evaluation, as well as to document and share information about new innovations. We think it's really important to foster discussion about choosing and using innovations to address evaluation challenges so that we can learn from each other. Therefore, we'd like to invite you to share your thoughts on this topic.

Have you used any of these innovations? How did it go? Are there any other innovations to address these challenges that you would suggest? Or other challenges that you think are in need of innovations? Let us know Or found other ways of addressing these challenges? Let us know in the comments below, or if you have a resource you think would be worth sharing you can contribute content here.

Sources

Heider, C. (2018) In the SDG Era, what are the Key Questions for the Evaluation Community? Retrieved from https://ieg.worldbankgroup.org/blog/sdg-era-key-questions-evaluation

Rogers, E.M. (2003). Diffusion of innovations. Free Press. 5^th Ed.

Rogers, P.J. and Macfarlan, A. (2018). Innovations in evaluation: How to choose, develop and support them. Briefing paper from joint UNICEF-BetterEvaluation-EVALSDGs webinar held May 2018. Retrieved from:  https://www.betterevaluation.org/tools-resources/innovations-evaluation-how-choose-develop-support-them