52 weeks of BetterEvaluation: Week 19: Using the Rainbow Framework

How do we ensure we address all the important aspects of an evaluation when we're planning it? How do we manage to consider the different options without being overwhelmed?

This week I was pleased to join over 500 people in the first in our Coffee Break Webinar series with the American Evaluation Association to explore these issues.

In this blog post I discuss five ways I've used the rainbow framework for planning, managing and conducting evaluation, and answer some of the questions which were asked in the webinar.

You can watch the first webinar, access slides and links to free downloadable tools and get a full overview of the webinar series below:

Overview of Rainbow Framework for evaluation webinar, slides and tools

BetterEvaluation series of AEA Coffee Break webinars

I’ve found that the Rainbow Framework can help both evaluators and commissioners of evaluations to make thoughtful choices that are appropriate and feasible for their situation – and makes it easier for them to make sense of the many M & E resources that are available.

The framework outlines the key tasks involved in any evaluation and organizes them into seven clusters:

• Manage: Manage an evaluation (or a series of evaluations), including deciding who will conduct the evaluation and who will make decisions about it
• Define: Develop a description of what is to be evaluated and how it is understood to work
• Frame: Set the parameters of the evaluation – its purposes, primary intended users, key evaluation questions and the criteria and standards to be used
• Describe: Collect and analyze data to answer descriptive questions about what has happened
• Understand causes: Collect and analyze data to answer causal questions about what has produced outcomes and impacts
• Synthesise: Combine data to form an overall assessment of the merit or worth of the intervention, or to summarize evidence across several evaluations
• Report & support use of findings: Develop and present findings in ways that are useful for the intended users of the evaluation, and support them to make use of them

Using the Rainbow Framework

I’ve already been using the Rainbow Framework in five important ways:

1. Design and plan an evaluation

Working through the clusters and related tasks can give you the confidence and clarity you need to plan a good evaluation – whether you’re new to evaluation or have considerable experience. Systematically considering all the why, what, when, who and how questions also helps with transparency - developing agreements among key stakeholders about which options are best. 

2. Check the quality of an ongoing evaluation 

At various moments in the evaluation, you can return to the framework and check that you are on track to deliver a transparent, appropriate and good quality evaluation. If you have encountered a tricky situation of confusion or disagreement, then the framework can help you identify what the issue is that needs clarifying and can offer some options. 

3. Commission and manage an evaluation

When developing the Terms of Reference for an evaluation, the framework can help to make them clear and focused. When reviewing proposals, it can be used to assess their clarity and quality. Also, when managing an ongoing evaluation, the framework can help commissioners to keep a tab on core questions (see number 2 above).

4. Embed participation thoughtfully in evaluation

The aim of ‘engaging all stakeholders’ in evaluation is often not feasible or necessary, yet participatory evaluation should be more than just asking people for their opinions. The framework can be used to consider who needs to be involved in which stage and how, plus how to make this possible, to plan for an appropriately and meaningfully participatory evaluation.

5. Develop evaluation capacity

The framework can be used by organizations to map existing skills and knowledge, in order to identify gaps and options for filling those gaps. It can also help an organization gain insight into which aspects of planning an evaluation need to be strengthened. For individual evaluators, the framework can be useful as part of a lifelong learning endeavor. None of us know all the 200+ evaluation options described on the BetterEvaluation website. There is always more to learn. By scanning the clusters and tasks, evaluators can identify areas where they would like to bolster their own capacity and find clear options for doing this.

Questions from the webinar

Does the framework exist in other languages?

We are currently working on this. We’re delighted to announce that EvalMENA is working with us to develop Arabic subtitles for all eight webinars in the series, which will provide an introduction to the whole framework. Stay tuned for more news about languages – or contact us if you’d like to help with a particular language.

What processes are in place for quality control?

The material on BetterEvaluation is created by identified authors with experience in evaluation and expertise in using particular options. Where possible pages are written by experts who have used the option extensively or by drawing on advice from experts. All pages show the author, reviewer and steward (if active).

For example, we are delighted that our authors include Angela Ambroz and Marc Shotland, JPAL(Abdul Latif Jameel Poverty Action Lab) who are currently revising the page on RCTs (Randomized Controlled Trials), drawing on their long experience with this approach. And that our reviewers include Werner Ulrich, Martin Reynolds and Bob Williams who reviewed and improved our page on Critical Systems Heuristics, drawing on their combined experiences in developing and exploring the approach.

What is unique about the Rainbow Framework?

The ideas underpinning the framework will be very familiar to people who have been trained in evaluation. What the framework does is to embed what we know about good practice in evaluation into a simple schema that makes it accessible to evaluation managers and community members who have not been trained in evaluation.

The rainbow metaphor represents the principle of diversity and inclusion that underpins BetterEvaluation – we promote thoughtful selection of the appropriate methods for a particular situation, and therefore include all evaluation methods and a space for constructive and respectful discussion about which ones should be used when and where.