Footprint evaluation webinar 2: Entry points for environmental sustainability

In part two of this three-part webinar series, Jane Davidson and Patricia Rogers discuss several ways to get sustainability on the evaluation agenda, even for projects that have no explicit environmental objectives and where there is no mention of environmental considerations in the Terms of Reference.

The approaches explored include using evaluation criteria, such as the OECD DAC criteria and using ‘Footprint-savvy’ Key Evaluation Questions (KEQs).

Entry points for environmental sustainability webinar

You can download the slides for this webinar from the Resource link above or watch the recording of the presentation section of the webinar below. Rather than share the video of the discussion, we’ve included a summary of this below - with thanks to Patricia Rogers and Kaye Stevens for pulling this together.

Summary of comments, questions and discussion in Webinar 2:

We were delighted to welcome participants from across the Pacific, Asia, Africa and North America, including China, India, Fiji, New Zealand, Australia, Indonesia, Zimbabwe, Uganda, South Africa and Canada. Most participants conduct evaluations, some both conduct and commission evaluations and one was previously an evaluator and now trains evaluators. The presentation focused on two particular strategies for getting environmental sustainability on the agenda for evaluation – evaluative criteria and key evaluation questions. The discussion also explored other strategies.

Experiences with OECD-DAC (Organisation for Economic Co-Operation and Development, Development Assistance Committee) criteria and Key Evaluation Questions

Question: In development evaluation there are conventions around the criteria that we use to assess programs and investment, such as relevance, alignment, harmonization, coherence, efficiency. And I just wondered if anyone had any experience weaving the kinds of questions around footprint evaluation in with some of those existing DAC criteria that people are already working with.

Comment: Having some published case studies would be very helpful. If you have any references, please let me know.

Responses: The OECD DAC criteria were revised and reissued recently and the new documents specifically talk about environment.

We contributed to an evaluation of a National Private Sector Development Strategy where the Terms of Reference were framed in terms of the new OECD DAC criteria. It was possible to consider environmental considerations within the criteria - the DAC criteria provided an opening to make links to environmental sustainability. We also undertook a case study of the environmental implications of the provision of personal protective equipment during the COVID-19 pandemic, using the Key Evaluation Questions as a guide. These examples will be shared soon.

Evaluative criteria – Efficiency

Comment: I encourage my clients to take a broader view of resource use in VFM (Value for Money) assessments. Recognising that resources aren’t just money, but also (for example) human resources and natural resources. Getting pretty good traction as far as shifting the thinking to first base, e.g. development programs tracking their CO2 emissions in addition to travel spend.

Response: Advocates for policy change are pushing for the removal of externalities – environmental damage produced by an entity but isn’t counted as a cost to that entity. Policy changes can push costs back to the polluters who stay in business because they aren’t paying these costs. It’s very important that evaluations don’t just think about costs incurred by the program, or the entity running the program but also consider costs that are borne by others, often completely inappropriately. Getting these ‘externalities’ out as cost loopholes is critically important.

Creating new evaluative criteria

Comment: An initiative in South Africa is extending the six OECD criteria by adding two additional criteria that will become standard in all evaluations. The new criteria relate to 1) climate and environmental health associated with programs or program outcomes and 2) just transition equity and past harm for those impacted by programs. The new criteria are being tested.

Response: When you have material to share, we’d be really delighted to help share that.

Environmental Impact Assessments (EIA) and Strategic Environmental Assessments (SEA)

Question / Comments: There is a process before implementing a program to minimize negative impact or to develop a development strategy/policy, Environmental Impact Assessment (EIA) and Strategic Environmental Assessment (SEA). How might we draw on these for considering environmental sustainability in evaluations?

A limitation of EIAs is that they are initiated and funded by the proponent and geared to securing approval. Impacted communities may have perspectives, concerns that are not well addressed in EIA process.

Experience in Indonesia is that the SEAs have become an obligation for governments including local governments. However, they are being done with low budgets and a lack of understanding about strategic adverse environmental assessment. As a result, the SEA is a procedure that collects data that isn’t always useful.

Response: Environmental impact statements do have limitations, as noted. However, where EIAs have been done, these can be useful as a ‘crib’ sheet to identify what monitoring and mitigation strategies were needed and whether the monitoring and mitigation systems have been effectively implemented.

This experience raises two of the big challenges: how do you do it in practice and how to link it to use? To do a proper environmental impact assessment is a huge undertaking - it’s not something you can tack on to an evaluation with no extra resources. We need to show how it’s used for different types of decision making and how to be strategic and focus on the big issues – we need to produce approximate answers to the big questions rather than trying to measure everything, or just measuring what’s easy to measure.

What if there are no environmental outcomes in the TOC?

Comment: I find it sometimes challenging to get the evaluation to look at environmental impact and CCA (Climate Change Agreements) when there is no specific ‘environmental outcome’ in the project’s ToC (theory of change). Perhaps project’s design should be more intentional in looking at and addressing environmental root causes of issues to be addressed by the project…

Response: A project’s TOC is usually focused on human system outcomes. It is useful to think about identifying the points of nexus with natural systems in terms of inputs, resources and procurement as well as the environmental consequences of project outcomes. This was explored in a set of thought experiments that considered the environmental implications of four programs that were initially evaluated in terms of outcomes for people, and in the first webinar in this series.

GEF evaluations

Comment: GEF has a standard expectation of evaluation, once you have the criteria, there are a set of questions that have embedded environmental aspects to be covered that include relevance, effectiveness, efficiency that need to be calculated and rated. GEF projects have environment as an outcome so it’s a little bit easier when compared to a Maternal Newborn Child Health Project. The footprint evaluation initiative could probably advocate for UN commissioning agencies to require environmental aspects in different criteria.

Response: The GEF has environmental intended outcomes, so the challenge isn’t about getting environment on the agenda. GEF Evaluation reports could be very helpful in terms of methods for answering evaluative questions. I’ve been particularly interested in their approach to causal inference, because when we’re talking about landscape scale interventions or national interventions, we need approaches to causal inference that don’t depend on having a counterfactual.

Engaging with Indigenous and local expertise

Comment: If evaluations are genuinely engaging local communities and stakeholders on the ground in developing the criteria, the footprint criteria are likely to come up anyway?

Response: Yes - especially if the first people you speak with are the traditional custodians of the land, default thinking about preserving the environment for future generations is very natural. Local people can know a lot more about the local environment than so-called experts. So start with that expertise, bring in environmental scientists, even better find Indigenous environmental scientists.

Evaluators can get the environment on the agenda

Comment: I think evaluators have more power than they realize to push for the inclusion of environmental sustainability in an evaluation.

Responses: In some cases it can be appropriate to add an extra evaluation question and an extra criterion – give the commissioners what they want and also what we believe they need.

Key Evaluation Questions and criteria, whether DAC or the ‘5 Es’ (economy, efficiency and effectiveness, equity, cost-effectiveness) which look at the relationship between impacts achieved and resources used, are contextual and the definitions are re-written and adapted to fit the context. This involves thinking about what’s important in this context and what’s important includes natural systems.

There are many ways to infuse sustainability into whatever framework that you’re expected to use, such as the OECD DAC criteria or into a set of key evaluation questions in the terms of reference. As evaluators, you can get away with a little bit more than what you probably think, Key evaluation questions can be written in a way that are going to make sense to your audience, that’s going to be a quite a powerful way of saying, surely you would want to know this?

You can either infuse things into the way that you normally do your evaluations, or you can pull things out as separate criteria. This really highlights the issue but if you can’t add the extra criteria you build them into whatever you do. You need to get in whatever we need to save the planet and get a more equitable society. It’s part of our job to help people think that way for the good of the planet, and the people. Often there are stakeholders both internal to an organization or agency and external stakeholders, who are already pushing for this. We’re the outside voice that can help the people who are already trying to get this stuff on the agenda.

If you really care about whether it’s being efficient, you have to look at the real costs. A lot of evaluation work is done for government or large organizations. If one part of an organisation, or one government department, is trashing parts that another part of the organisation or another government department is going to have to pick up, then the government as a whole, or the organization as a whole should care about that.