Adapting evaluation in the time of COVID-19 - Part 1: Manage

By
Alice MacfarlanPatricia Rogers
adaptive management
COVID-19
COVID-19 series
illustration of people wearing medical masks

Organisations around the world are quickly having to adapt their programme and project activities to respond to the COVID-19 pandemic and its consequences. We’re starting a new blog series to help support these efforts.

We’ll be exploring some of the key issues and questions to be addressed, and structuring these around the seven clusters of tasks in the BetterEvaluation Rainbow Framework: MANAGE, DEFINE, FRAME, DESCRIBE, UNDERSTAND CAUSES, REPORT AND SUPPORT USE. We’ll also be creating a complementary thematic area on the BetterEvaluation website to gather this information and associated resources in a more permanent and accessible manner. We see this as a work in progress – new guidance and resources are being developed rapidly as the evaluation community comes together to support one another in this global crisis.

Overview

Much evaluation work, and the interventions they evaluate, have to change in response to changing needs, priorities, constraints and opportunities.

This involves adaptation in terms of how activities are done, and in many cases adaptation to undertake different types of activities. There are in many cases also fundamental changes in terms of planning, implementation and evaluation, from linear processes of using knowledge to plan and check compliance to more adaptive management processes of acting under conditions of ongoing uncertainty and imperfect information.  Evaluation approaches that have been designed for these conditions, such as real-time evaluation and developmental evaluation, now have wider relevance.

We believe evaluation is critical to supporting effective adaption, and that effective evaluation at this time will need to be adaptive. This means rethinking how to approach all of the tasks in an evaluation. This new blog series at BetterEvaluation is designed to help you reflect on issues and methods to do with evaluation, and to bring together useful knowledge at this unprecedented time.

We’ll be structuring this blog series around the seven clusters of tasks in the BetterEvaluation Rainbow Framework: MANAGE, DEFINE, FRAME, DESCRIBE, UNDERSTAND CAUSES, SYNTHESISE, REPORT AND SUPPORT USE. We believe it is essential to consider all the tasks in an evaluation, and how they may need to be adapted. This will help ensure an evaluation can be completed safely and meet the changing needs of programmes and staff. This blog will focus on the first cluster of tasks in an evaluation: MANAGE.

Manage an evaluation or M&E system

This section of the Rainbow Framework sets out 9 key tasks involved in managing an evaluation, an evaluation system or ongoing evaluative activity. We’re going to look at each task in turn and consider some of the ways you may need to e adapt each task to meet the changing circumstances. You can find more detail on each of these tasks in the MANAGE section of the Rainbow Framework.

1. Understand and engage stakeholders

Identify who has an interest in the evaluation in addition to the primary intended users, and whose interests need to be prioritised and why. Ensure their engagement throughout the evaluation. View full task page in the Rainbow Framework.

Most approaches to evaluation emphasise identifying and engaging with stakeholders (those who have a ‘stake’ in what is being evaluated) during the evaluation process to ensure the evaluation will meet their information needs and appropriately represent their interests.  (This is a different task to collecting data from stakeholders, which we will discuss in a later blog).

With changes to where we can travel and how we can meet, the ways in which you had planned to engage your stakeholders may need to change. In addition, the needs and availability of your stakeholders may have changed. It is a reasonable assumption that many people will have reduced capacity or productivity at this time, and so we encourage you to check in and make adjustments to your engagement plans as necessary.

Key considerations:

  • How has the COVID-19 crisis affected the key stakeholders of your evaluation?

  • Have their roles changed? Do they have additional duties that will take away their focus from engaging with the evaluation?

  • Will you need to rethink your planned engagement activities?

  • If you will need to move your engagement activities from face-to-face to online, what conditions or support will be needed to make participation and engagement possible – time, resources/technology, capacity, facilitation?

  • How will you ensure that all your groups of stakeholders are able to have a voice during the evaluation process? (Including both those who make the decisions about an evaluation, as well as those who are affected by an evaluation)

Resources:

2. Establish decision-making processes for evaluation

Specify how decisions will be made about the evaluation—who will provide advice, who will make recommendations, and who will make the actual decisions. View full task page in the Rainbow Framework.

As your evaluation may need to adapt to rapidly changing circumstances it is critical to be clear about how decisions will be made and who will make them, and to revisit these processes to make them nimbler if needed.

Key considerations:

  • How have decisions been made previously in the evaluation?

  • Is it still possible or appropriate to make decisions in this way if you need to implement changes to the evaluation plan?

  • What sorts of decisions will you need to make now that you did not previously have to make?

  • Who will have authority to decide these sorts of decisions?

  • How will you document the decisions made – including large, formal decisions and smaller, ‘on-the-fly’ decisions?

Resources:

  • Manager’s Guide – Step 1: Decide how decisions will be made: The BetterEvaluation Manager’s Guide to Evaluation provides some advice about creating a decision making process.
  • Tool – Decision making matrix: This downloadable template will help you map out and document who will be involved in which decisions.
  • Tool – Decision journals: Reflecting on how decisions are made can help improve future decisions  A decision journal can help to identify patterns in good and bad decisions and build skills in making better decisions under these changed circumstances. 

3. Decide who will conduct the evaluation

Clarify who will actually undertake the evaluation. This might include people who are involved in what is being evaluated (such as implementers, clients and community members), an internal or external evaluator, or some combination of these. View full task page in the Rainbow Framework.

It’s worth taking a moment to reflect on who will conduct the evaluation in the changed circumstances.

Key considerations:

  • Are there new restraints or time burdens that will hinder external evaluators, internal staff or community from conducting the evaluation as planned?
  • If your evaluation design has to change, does your evaluation team still have the necessary skills? If not, how can you bring in additional external expertise or strengthen the capacity of your team? (See below – Strengthen evaluation capacity)

  • Might it be appropriate to consider a hybrid model where internal staff work with external evaluators?

Resources:

  • Discussion paper: Evaluating Innovation in Action - A GP for me – this article from Canadian Government Executive describes how the evaluation was undertaken by a hybrid team, involving staff from the contracting authority and an external evaluation team.  
  • Manager’s Guide – Step 4: Engage evaluators: The BetterEvaluation Manager’s Guide to Evaluation offers guidance on how to engage evaluators. ​

4. Determine and secure resources

Identify what resources (time, money, expertise, equipment, etc.) will be needed and available for the evaluation. Consider both internal resources (e.g. staff time) and external resources (e.g. participants' time to attend meetings to provide feedback). View full task page in the Rainbow Framework.

If you need to change the design of your evaluation, you may need to make changes to your evaluation budget as well. In addition to known or calculable costs related to things like staffing, consultant time and technology, adaptive management means you will need to be prepared for future unseen challenges and changes of course. If possible, have some flexibility in your budget to make adjustments as the need arises and ensure that you have clear decision-making processes so that these changes can be actioned as quickly as possible.

Economic conditions have changed dramatically in the last few months and as a consequence, funding for many organisations has been reduced. This is likely to have an impact on evaluation budgets. This comes at a time when important, evidence-based decisions are more necessary than ever, so it’s worth considering how you can make the most of your resources to ensure a high-quality evaluation.

Key considerations:

  • Will you need to change your evaluation design or processes?

  • What additional costs will you now have?
  • What costs will be reduced?
  • What resources do you have available to you? (Including people’s time, expertise, equipment and funding)
  • Is it possible to include an ‘adaption’ fund or contingency allocation in your budget to allow for flexibility?
  • Have you budgeted for capturing learnings about this process?
  • Who has the authority to make changes to the budget and activities?
  • If your evaluation budget has been reduced, where can you reduce your costs?
  • What parts of your process are “essential” and which are “nice to have”?

Resources:

  • Method page - Strategies to reduce cost: This page discusses ways to lower your evaluation’s budget 
  • Blog - Evaluation on a Shoestring: This blog by Patricia Rogers looks at options for evaluation on ‘no budget’ (or at least, a very small one) 
  • Blog - Budgeting for Developmental Evaluation: This blog by Michael Quinn Patton discusses the issue of budgeting in a developmental evaluation – rather than having a fixed TOR and a competitive quote to deliver it as specified, the evaluator takes an ongoing support role to help gather information as needed, requiring a different procurement approach.  

5. Define ethical and quality evaluation standards

Clarify what will be considered appropriate quality and ethical standards for the evaluation and what will need to be done to ensure these standards are achieved. View full task page in the Rainbow Framework.

On ethical standards:

In the current global health and economic crisis, there is a real risk of doing harm. It is essential for those involved in an evaluation to keep ethical standards front of mind as they make decisions and changes. This includes paying particular attention to equity issues in the evaluation.

The onus should be on evaluators to

a) find out whether and how things have changed for the most disadvantaged and marginalised, and

b) determine what they can do to ensure these issues are addressed in the evaluation design and implementation.

Without an understanding of what the current crisis means for disadvantaged and marginalised populations, then going ahead with an evaluation ‘as planned’ carries a real risk of causing harm.

The scope of evaluations at this time should be expanded to include any new or deepened issues related to equity that have emerged during the current crisis.

Key considerations:

  • In what ways could your evaluation put members of the community at risk and how can these risks be mitigated? If you need information from essential service providers for example, how can you ensure these interactions are kept as brief as possible?
  • How will any changes you make to your evaluation affect your most marginalised stakeholders? (for example, do all your stakeholders have equal access to technology such as mobile phones?)
  • How will new social distancing measures impact the involvement and participation of community voices? How can you mitigate any negative impacts?

Resources:

On evaluation standards:

Many people associate evaluation standards with a specific approach or methodology, or go in with a particular view of what constitutes "good" evidence, and believe that without this an evaluation will not be of a worthwhile quality. However, this is worth revisiting – especially given the current circumstances where important decisions are being made rapidly.

  • Given the fast pace of changes and information needs in the current crisis, will there be changes to the way you weight the different evaluation standards? For example, if a decision needs to be made now, you may want to think through the weighting of quality, large scale data, for some data that can be provided in a timely manner.

Resources:

We’ll be discussing approaches and methodology in more detail in a later blog but in the meantime, here are two good resources to help you consider whether evidence that is “good enough” will be sufficient:

  • Blog - MQP Rumination on the ‘so-called gold standard’: This is an abbreviated version of a "rumination" from the 4th edition of Qualitative Research and Evaluation Methods by Michael Quinn Patton, published in mid-November, 2014. It argues for contextual appropriateness when choosing methods and research designs instead of using a hierarchy of evidence which favours a "gold standard" method.
  • Discussion paper - What counts as good evidence: This paper, written by Sandra Nutley, Alison Powell and Huw Davies for the Alliance for Useful Evidence, discusses the risks of using a hierarchy of evidence and suggests an alternative in which more complex matrix approaches for identifying evidence quality are more closely linked to the wider range of policy or practice questions being addressed.​ Read more​

6. Document management processes and agreements

Develop any formal documents needed, including a brief and a Terms of Reference. View full task page in the Rainbow Framework.

Key considerations:

  • Will you need to formally amend the Terms of Reference for the evaluation and any contractual agreements?

Resources:

  • Tool - GeneraTOR: This interactive tool, developed as part of the BetterEvaluation Manager's Guide to Evaluation, walks you through the key sections in a Terms of Reference (TOR) document, and generates a downloadable document based on your answers that can be shared with others. You must register for a free BetterEvaluation account to use this tool.
  • Discussion paper - MANAGING TO ADAPT: Analysing adaptive management for planning, monitoring, evaluation, and learning​: This Oxfam research report uses a number of case studies to explore how donors and implementers have used adaptive management - including discussing how this has been used in planning and funding mechanisms.

7. Develop planning documents for evaluation

Develop a formal plan which sets out how an individual evaluation or a range of M&E activities will be undertaken. View full task page in the Rainbow Framework.

Planning documents can include formal evaluation plans (for a particular evaluation) and evaluation frameworks (an overall framework for evaluations across different programs or different evaluations of a single program), as well as work plans, aide memoires, and inception reports.

Key considerations:

    • Will you need to revisit and revise your evaluation plans?
    • Given the current global uncertainty, will your revised plans need to incorporate evaluation approaches designed to be adaptive and address complexity? (e.g. Developmental Evaluation, Outcome Harvesting)

    Resources

    • Approach - Developmental Evaluation: An approach designed to support ongoing learning and adaptation, through iterative, embedded evaluation.​ 
    • Approach - Outcome harvesting: An impact evaluation approach suitable for retrospectively identifying emergent impacts by collecting evidence of what has changed and then working backwards, determining whether and how an intervention has contributed to these changes​.

    8. Review evaluation (do meta-evaluation)

    Decide processes to review the evaluation process, findings, and conclusions drawn. View full task page in the Rainbow Framework.

    In order to adequately adapt your evaluation to the changing circumstances, it help to include reviews early and often.

    Key considerations:

    • Instead of waiting until the end of the evaluation for quality reviews, how can you incorporate a review processes into early stages (e.g. at the stage of developing the Terms of Reference, Inception Reports, Baseline / Mid-line Reports, etc.) in order to pick up and address any emerging quality issues?

    9. Strengthen evaluation capacity

    Choose ways of building on and developing capacity for managing, undertaking and/or using evaluation, which might involve human capital (knowledge and skills), organisational capital (such as technical and administrative infrastructure) and social capital (supportive networks). View full task page in the Rainbow Framework.

    Many people are finding their roles, duties and ways of working have changed significantly in the past few months. For many this is because the evaluation’s design has changed, leaving some clear skill gaps about how to conduct an evaluation (for example, how to do a real-time evaluation, or use a new approach such as outcome harvesting).

    But there are other important areas of evaluation where the terrain has rapidly changed, including commissioning, managing, reporting, and supporting use of an evaluation, which may require those involved in an evaluation to take a different approach to tasks than before. There are also a number of areas where we have to rapidly learn how to do things differently – such as using new technology and developing our social and communication skills to effectively work and meet remotely. It can be useful to think about the areas in which you and your team need to strengthen capacity. It’s important to acknowledge that people can feel awkward or vulnerable learning new skills and technologies and that it might take some time for everyone to become comfortable doing things in new ways.

    Key considerations:

    • What are the additional or changed tasks, roles, methods, processes, or approaches that you will need to address?
    • Where do you and/or your team feel comfortable in your ability to enact these changes?
    • Where do you and/or your team have concerns or worries?
    • What existing skills and knowledge can you draw on?
    • What skills and knowledge will you need to develop or strengthen?

    • What are the available resources for capacity strengthening activities (e.g. money, time – in terms time able to be spent on capacity strengthening, but also time before skills and knowledge are needed)

    • What different options are there for capacity strengthening activities and how would these suit your needs? (Courses or training are one option, but also think about things like peer support and review, mentoring, discussion forums, webinars, just-in-time expert advice, formal quality review processes)

    Resources:

    Help us expand this content

    We've tried to capture some of the main issues and provide some useful resources and starting points for adapting your evaluation processes. But we know there's a wealth of information that we haven't included, and that new advice and guidance is being developed to address these issues.

    This is where we need your help - head to the Contribute Content page to contribute a resource, event, blog, or to get in touch to suggest a different form of content.

    We're all in this together and we value your input into how we can support the adaption of evaluation to meet the challenges of COVID-19.

    Related content

    'Adapting evaluation in the time of COVID-19 - Part 1: Manage' is referenced in:

    Back to top