Footprint evaluation webinar 3: From 'do no harm' to restoration - Working with a typology of Footprint Evaluation

In the last part of this three-part webinar series, Andy Rowe and Patricia Rogers introduce a typology being developed that will assist a wide range of evaluations in assessing the effect of interventions on natural systems and sustainability.

Does the intervention achieve the standard we need where no net harm is caused to natural systems, does it contribute to the restoration of natural systems, or is it harmful to these systems?

From 'do no harm' to restoration webinar

You can download the slides for this webinar from the Resource link above or watch the recording of the presentation section of the webinar below. Rather than share the video of the discussion, we’ve included a summary of this below - with thanks to Patricia Rogers and Kaye Stevens for pulling this together.

Summary of comments, questions and discussion in webinar 3

We were delighted to welcome participants from across the Pacific, Asia, Africa and North America, including Indonesia, Cote D’Ivoire, Romania, USA, Canada, New Zealand, Tanzania, Fiji and Australia.

Applying the typology – tools, methods, resources and types of evaluation

Question: Where can we learn more about the tool and methodology to categorise projects per type of impact on ecosystems. For instance, is there a tool available that could be applied to a wide range of projects?

Responses: We have a page on the BetterEvaluation website, which sets out material as we’re developing it. That’s our key way to share this material as it becomes available. We also have a discussion group to talk about these things and learn from people who are trying out some of these ideas or who have other suggestions on how to improve it.

We’re looking at a whole range of tools and methods to be used for different types of projects.

Question: How intensive it is (level of expertise and amount time required, additional cost to a typical evaluation…?) Is this applied at project design phase or midline / endline evaluation stage?

Response: Environmental sustainability is potentially important for all types of evaluation. Clearly there’s not much point waiting till the end of a six year program and then saying we’re in deep trouble. That’s not evaluation helping at all. It’s got to be included in all types of evaluation. Ex ante evaluation, where we’re looking at options and trying to see what when or where time and money should be invested. Design evaluation, how we should be constructing interventions, what are the monitoring systems? What are the risk management strategies that need to be in place to manage environmental risks, and to monitor that we’re implementing the strategies effectively? It is needed in real time evaluation, really working to support managers, and staff and others key stakeholders to have adaptive management to address emerging risks or opportunities - not just what you could predict upfront, but how you need to be changing things. And of course, impact evaluation, and systematic reviews. These ideas must be applied and adapted to the full range of types of evaluation. Some of the examples we’ve worked on have been impact evaluations and one was a midterm evaluation. It was very much geared up to say, where are we and what do we also need to put in place, a lot of what we recommended was about improving the monitoring system to do a much better job of monitoring risks and feeding that into decision making.

Questions: I was really curious to hear about the examples from IFAD. Were the cases endline evaluations? Can the additional resources that were allocated to include an environmental sustainability assessment be quantified? Or was this purely an environmental sustainability evaluation? Or was this a meta evaluation, looking at all outcomes, including human centered outcomes and environmental impact? What was the scope of the study?

Response: The evaluation was a thematic evaluation so it’s a very large-scale evaluation that looked at a huge portfolio, the entire portfolio of climate adaptation projects for small holders for over about a 10 year period. I was part of the design team, early on, I said shouldn’t we be including sustainability and natural systems in the evaluation? There was about a five second silence and the response from the team leader was, yes, absolutely, we will do this, I’ll get permission tomorrow. I think that in itself was significant, because I don’t think there would have been that response five years ago.

I spent perhaps 20 days on the actual “Nexus case study”. There was perhaps a quarter day from each of about 10 other members of the evaluation team. Because it was during COVID, there was no fieldwork. I do want to emphasize that we were at the upper end of the competencies you would expect in evaluation team able to do this because of the nature of the intervention, and the people who had been selected and people we subsequently brought on to the evaluation team.

Question: Was this evaluation relying on available data?

The data in the case studies was pretty comprehensive. We were able to do some project site work using national consultants in some countries, but it was heavily constrained because of COVID, none of us travelled. We started with a very extensive, detailed portfolio review that was itself quite illuminating. Then we reviewed all of the documents for each project. The lead consultant for each case study probably spent around 10 or 12 days on a case study, they were pretty in-depth case studies that involved a wide range of interviews maybe 15 interviews per case study, quite extensive document reviews and they were written up in an overall case summary usually around 20 to 50 pages each. There were extraction summaries focusing on long scaling, knowledge management, jobs, things like that. I did the extracts for the Nexus study and we went through the process that I’ve described.

Question: Were only the case studies focusing on environmental sustainability or the entire project?

Response: The entire project, the environmental perspective was added on to the terms of reference of the evaluation. 10 to 15 days of my time and a half a day of each of the case study consultants was the entire budget for the Nexus study.

It’s certainly a real question about what’s feasible in terms of the team and the budget, particularly if you’re talking about an evaluation where that there’s not a team, there’s a single evaluator - what skills and knowledge do they have? As part of this Footprint Evaluation Initiative project, we contributed a chapter on environmental sustainability to a midterm evaluation, we’ll be able to share that soon. One of the things we learnt was that there’s a lot of existing data on natural systems hazards and risks that’s available, but it’s not the sort of data that evaluators usually know how to access. With more resources of course you can do more, but there are things you can do within quite constrained budgets that would be an improvement.

When evidence isn’t enough - how can evaluations (and evaluators) influence change?

Question: What can an evaluation commissioner do when the local administration answers “it is the only source of living for the locals” when deforesting entire areas? Last week, I was in an area where half of the side of the mountain was deforested. The only businesses in the area work with wood, there are no other jobs, no other opportunities to develop. What can an evaluator or the commissioner do to help local administration to understand and find another solution, they say it’s our only source of income for the future. We don’t know. The indicators can be seen by anyone, not only by the experts, floods, landslides, garbage can’t be collected. In this case, it’s too late. But they keep saying it’s our only source of income.

Response: It’s currently and historically probably been the only source of income. It’s not necessarily true that it’s the only source of income that could have been generated. We have examples of agroforestry with limited harvest of trees and turning the forest landscape into productive land for agriculture. Obviously, tourism is always a possibility but is not an answer to anything and can be harmful. In evaluation we should be looking at the true cost or the net costs of the intervention. When you strip a hillside of trees, you’re doing a number of things. First, you’re eliminating sequestration of carbon so that’s an absolute loss. You’re leaving the soils very open to erosion. It probably depends on rainfall and the slope of a mountain and meteorological conditions, but chances are you’re going to get soils sliding down - causing harm and creating a need for future investments. You’re going to you’re going to get soil released into water bodies, those water bodies are going to be impaired because of soil suspended in the water which means that the interaction with light is going to be different and that water is going to be less able to support life and downstream that water is probably used for agriculture or for drinking or irrigation of various kinds, etc. With the argument that cutting forest trees is our only job and we have to keep cutting trees, well, you’re going to run out of trees, so it’s a very short-term thing. You open the timeframe, you open the connections of the forest, what else did the forest provide, and you include that in the evaluation.

Deforestation is clearly a bad thing - it still goes on where I live. There’s tremendous economic value in cutting down forests and it’s an uphill struggle, but the job of the evaluator is to understand the connectivity that the forest provided. Document what is going on, put it into this typology and say - this is extraction, this is down at the bottom of the scale, this is amongst the worst in the world. Because it is.

The analysis needs to also look at the impact on other values including ecosystem services. I can speak from my experience where we are still logging old growth forests, which have been reducing the value of the forest in terms of water catchment and habitat biodiversity. There are people trying to do these calculations and demonstrate that you’re in fact destroying value rather than adding value - you and I think that argument doesn’t need to be made. The other point is around the connection between equity and environment, I think that’s a key issue, it must be clear that environmental damage has equity impacts. Short term economic gain is at the cost of long term economic and human social harm. We need to be making that that argument as well.

Power relationships - role of evaluators in getting to win/win

Comments:

The argument about evidence of the typology working, coming from 20 case studies is convincing but the problem is much more complex than providing good evidence or excellent evaluation studies. It’s been done for over 50 years. We’ve spoken about thresholds, doomsday, ticking clock for decades. The problem is policies that guard the business of fossil fuels industry not because of lack of good evidence but because of money and growth focus. We have had natural resources experts alarming commissioners, policy makers, businessmen for years.

Where I live in Eastern Europe we also have wood cutting business and industry. We have lots of professionals working on nature, protection, and sustainable forest management - it’s not work that has not been done and it’s not that the evidence doesn’t exist. It simply about the power relations between those who are earning from it, and those who understand systems and long-term consequences. As evaluators, the question is do we have enough of legitimacy to say this is the evidence we need to follow and have an impact on others? That’s what I worry about.

How to include the elephant in the room in your work? The power relations between those protecting financial interests, knowing the evidence, and those focusing on nature protection? How to strengthen legitimacy of our work as evaluators by working with other profession(al)s who have been providing this sort of evidence for decades?

Responses: The process that Andy’s setting out is a way to engage that, it’s not a technical exercise for the evaluators, sit in a room collect quality data and come up with a score and expect people to be interested in it and committed to it. It’s about engaging those interests in the exercise.

We can think of it as two types of evaluations, evaluations that have something to do with natural resources - probably extraction, which is wood, cutting forests sort of stuff. And evaluations that don’t have natural resources as stated objectives where the programs and the evaluators aren’t aware of the coupling with the natural systems and that their interventions are harming the natural systems or affecting and affected by the natural systems. In the latter case, where natural systems are not explicitly or even thought of as connected to the evaluation. I think there’ll be some pushback from people who want to continue to draw a bunch of water for agriculture, or whatever.

It’s not a very pleasant journey. The best way to make progress from that extraction focus where things are posed to you in a mindset of either/or - you can either have the jobs from cutting trees, or you could have a nice park – is to seek win/win solutions. I think the job of the evaluator in the former situation, the later too but particularly in the former, where we’re actually evaluating extractive activities, programs, policies, projects, our job is to switch them from an either/or mindset to a win/win mindset, to where we can, working with all of the interests, remember, we pointed to the affected interests, and those affecting the interests of the logging industry, the truckers that the people are cutting, and people were dealing with the clearing of the land, and the tree planting after all of that in the downstream industries, all of them are part of the extraction side.

There’s a whole set of other interests, Indigenous people and local communities, some of them are working there, some are hunting, fishing, some just enjoy being in there, enjoy the protection, are aware of the benefits the forest brings. We need to get them all together, instead of fighting about what is, let’s turn them and say, what can we have? What do you as an extractor really need? How many board feet do you actually need from a forest to make it economical? And what kind of trees do you have to have, what kind of markets exist? We’re not saying you shouldn’t be there; we just want to understand that. For you who recreates and is in that area, what are the really important values that a forest brings to you. And we find the evaluator becomes facilitator, we find what is non-negotiable, what’s negotiable. We find ways of addressing everyone’s non-negotiable, what they have to have. That’s the way we move it forward. So we’re not analysts any longer, we’re facilitators of change. And all of this is evaluation in the end game, 2030 is only eight years away. Evaluators need to become much more involved and engaged in solution finding and solution acting, going towards win/win solutions, rather than becoming the judge in an either/or situation, which was never an appropriate place.

Question: What do you do when you use it [the typology] and your final report gets cut into pieces, so it doesn’t disturb local businesses and investors in the area from continuing deforestation as the only source of living?

Response: That’s the whole issue about integrity of the reports and who has the say over it is a big part. This process is about making it transparent so that it can’t just be squashed by one person.

Speaking ‘truth to power’

Question: Wondering whether evaluators could be supported to ‘speak truth to that power’ …if we can access a basket of knowledge - key policy docs, IPCC narratives and other credible international bottom line directives. As well as showing the links between environment and equity impacts.

Responses: One of the areas we’ve talked about is how do you get this on the agenda, and part of it is being able to make the case that this is urgent and important. This is certainly something where we’re wanting to put some materials together, we’re doing some reviews of the papers that came out after the COP 26 meeting that the journal Evaluation commissioned, the IPCC report are compelling ones but they’re not going to be magic.

The concept of speaking truth to power is an interesting concept. If you think about it, it’s very laudable as this image of bravery, courage, etc. But actually, you’re speaking to someone, you’re not speaking with someone, and you’re not speaking with them in a group of people who have different views that are also important to them. I think speaking truth to power is akin to reports where there’s a document that speaks the truth and right-thinking people will read it and change, but the evidence is that that rarely happens. Knowledge is a process and change is a process and as evaluators we need to be engaging more with those who care about the subject, evaluating for those who are involved in it and affected by it. And acting more as facilitators, allowing them to speak to each other, finding ways for that, and helping them come together around what they can agree on.

Comment: The latter case Andy speaks about requires really wise commissioners that also hold legitimacy and a powerful voice. What I lack in addition to technical support is the stakeholder analysis, tipping point, agent network analysis (call it as you like) to know where you have allies to work with and where you can speak the truth to power as a community that sees benefits. Win/win alternatives are much more convincing than just pure criticism. If this gets added explicitly, I agree with Andy’s arguments.

Capacity building & communities of practice

Comment / question: I’ve been working in evaluation of innovation systems and development for about five years. I’m back in my country, Mexico, and there’s a focus on agriculture. I’m looking at the generic topology that could be developed into specific areas and I’m sure that there’s a lot of information out there. My question is about capacity building and building communities of practices within the evaluation community. Also, a comment about change because maybe you’re moving into nurturing specific areas, we kind of have to be patient till we can empower and affect institutions, similar to the SDGs that are helping to create a pool of knowledge and experience that can help push that facilitation process?

Response: I can point to some innovative agricultural efforts in Latin America.