Footprint evaluation
Footprint evaluation aims to embed consideration of environmental sustainability in all evaluations and monitoring systems, not only those with explicit environmental objectives.
Footprint evaluation is not a specific methodology or a tightly prescribed approach but an emerging set of practices and principles developed through ongoing international collaboration and adaptation to suit different contexts. While the label comes from the notion of the 'footprint' humans make on the environment, footprint evaluation covers more than calculating the footprint of program interventions, it involves exploring different types of nexus (or 'connections') between human and natural systems.
Overview
Defining environmental sustainability
The International Panel on Climate Change (IPCC) defines sustainability as “a dynamic process that guarantees the persistence of natural and human systems in an equitable manner” (IPCC, 2018, p.559). That means sustainability is about simultaneously pursuing preservation and restoration goals for both the human systems (such as equity and food security) while preserving and restoring and natural systems (such as biodiversity and atmospheric carbon levels). Given the severe degradation of natural systems to date, the performance standards for evaluating sustainability need to, at a minimum, be set at not causing harm to the environment and ultimately move to a performance standard of the restoration of natural systems.
On this page, we refer to environmental sustainability as:
The ongoing ability of natural systems to support the equitable life of all species on earth.
Environmental sustainability includes mitigating climate change and addressing other environmental challenges, including biodiversity loss, over-exploitation, pollution (air, water, soil), deforestation, invasive plant and animal species, access to fresh water, and the restoration of natural systems.
Join the Community
The Footprint Evaluation Initiative, an international collaboration, is engaged in curating and co-creating knowledge about methods and approaches that can be used to evaluate the environmental sustainability of interventions.
If you'd like to be involved or stay up to date with this project, sign up for the Footprint Evaluation newsletter, and join the Footprint Evaluation Discussion Group.
Why should evaluations consider environmental sustainability?
The world is faced with numerous environmental crises with the potential for global catastrophe, including climate change, pollution, loss of biodiversity, ocean warming and acidification, and deforestation. To avoid, mitigate and address these crises, decision-making must be informed by evaluations that explore the actual and likely environmental consequences of all programs, projects, policies, and strategies.
However, as recent stocktakes have found (Todd, 2020; Rowe and DeLancy, 2021), most evaluations of change efforts primarily intended to benefit humans fail to consider environmental sustainability. This is so even for evaluations using frameworks such as the OECD DAC evaluation criteria, which include "significant environmental impacts" in the scope of impact evaluation, and in countries with global environmental commitments. It is also true in national settings where environmental regulations, standards, and international commitments rarely find their way into evaluations.
What types of evaluations should include consideration of environmental sustainability?
Environmental sustainability is relevant to all sectors, such as:
- community and public health
- education
- housing
- transportation
- primary industries (agriculture, horticulture, aquaculture)
- economic development
- community development
- tourism
- training and development
- organisational and whole-systems change
Environmental sustainability can and should be considered in all types of monitoring and evaluation, including:
- evaluations of projects, programs, policies, portfolios and strategies
- evaluations conducted before, during and after implementation
- evaluations conducted for learning and accountability
- evaluations conducted by external evaluators, internal evaluators, implementers, and communities
- evaluations that focus on inputs, processes, outputs, outcomes and impacts
- evaluations labelled as mid-term, end of term, ex-ante, ex-post, real-time and developmental.
Considering environmental sustainability in different types and stages of evaluation can help to inform a range of decisions, including:
- the choice between investment options or whether or not to proceed
- reactive management decisions if there is evidence of inadequate compliance with risk mitigation processes or a lack of effectiveness of these processes
- adaptive management to address emerging environmental risks or opportunities to improve the environmental sustainability of the intervention
- the continuation or scaling up of an intervention or the design of future interventions based on a better understanding of the environmental impacts (including intended, unintended, positive and negative impacts)
- the design of further interventions and inform broader strategy decisions
How to address environmental sustainability in an evaluation
You can read further about:
A. Emerging principles for footprint evaluations
B. Getting environmental sustainability on the agenda
C. Understanding the nexus of human and natural systems
D. Collecting data
E. Valuing environmental impacts
F. Further resources and references
A. Emerging principles for evaluations that include environmental sustainability
The following section describes an emerging set of principles for incorporating environmental sustainability into evaluations. These should be read as a work in progress that will grow with use and input from others throughout the Footprint Evaluation Initiative project.
- Starting point: Worldview and mindsets
- Know the place
- Expand spatial and temporal framing
- Use systems thinking and frameworks
- Focus on the big issues
- Draw on multiple types of evidence and expertise
1. Starting point: Worldview and mindsets
Embedding consideration of environmental sustainability in evaluation requires more than just knowledge about methods and tools. In many cases, a shift in worldview and mindset will be essential. A worldview is fundamentally about what is valued, what is essential or sacred, what has status and what might be treated as having no value or importance and so used freely.
We have identified three important ways of thinking when considering environmental sustainability in all evaluations:
- Recognise the value of both natural and human systems
- Adopt win-win mindsets
- Recognise that environment and equity are inextricably linked
Recognise the value of both natural and human systems
Footprint evaluation recognises that human and natural systems are strongly coupled; what we do in human systems draws from and affects natural systems. Human life is dependent on being able to draw from natural systems. This includes the necessities of air, water, and nutrition, as well as the importance of natural systems for physical and mental health and well-being. Beyond the value that natural systems have to humans, they also have value to other natural systems with which they couple. Damage to one system can have devastating ripple effects on other systems.
Much scientific evidence supports warnings about tipping points and global catastrophe if human development continues unchecked. Communities at the local, country, and regional levels are already experiencing devastating impacts of climate change, pollution, and other environmental crises.
Given this, our stewardship responsibility is to ensure the sustainability of human and natural systems. At a minimum, this entails ensuring that human actions are not harming natural systems. However, it is imperative that we also bring back to health the natural systems that human activity has badly damaged in the past. One of the jobs of evaluation is to recognise the burden human actions place on natural systems and contribute to finding and valuing alternatives.
Adopt win-win mindsets
Mindsets founded on competition between goals and actions have also been important in justifying or ignoring sustainability. Decision-making has often been framed along lines such as 'development vs equity', 'development vs sustainability', or even 'equity vs sustainability'. However, rather than viewing these oppositions as the default and inescapable position, it's vital to expand this framing to look for 'win-win' solutions. Consensus decision-making around win-win solutions aims to recognise and advance all of the key interests in a matter, whether these interests are part of human or natural systems. For example, the 2015 Paris Agreement adopted a worldview that valued human and natural systems and sought agreements that addressed the needs of both systems while applying a consensus approach to reach the agreement (Radoslav, 2016). Consensus approaches are well-established processes to reach decisions in complex human and natural system matters.
Recognise that environment and equity are inextricably linked
In many cultures, natural systems have historically been, and are largely still, considered foremost as resources for human use. Evaluation as a field is slowly recognising the underlying assumptions underpinning evaluation practice and theory, but there is much work to do to deconstruct and address historical and ongoing systemic biases in how we think about human and natural systems.
Focusing on the impact on natural systems, the result of this worldview is that planetary resources have been compromised and depleted and soon will be unable to support many species. Traditional owners and Indigenous communities with spiritual, economic, cultural historical, and other ongoing connections to and deep knowledge of the land are often excluded from decision-making processes about land, waters, air, plants and animals and management of human use of natural systems. It's essential for evaluation to actively deconstruct worldviews that place human systems over natural systems, recognise the interconnectivity of equity and environmental sustainability, and value the importance of a wide range of human and non-human interests.
2. Know the place
Understanding the place is essential - whether the 'place' is the site of a project, program, strategy or a policy. The place of an intervention can have physical, social, economic, cultural and other dimensions, and these often couple with dimensions from other places. Understanding the place is vital for identifying connections between natural and human systems and including natural systems and sustainability in the theory of change.
Most evaluations include a process of identifying interested parties and intended beneficiaries. Footprint evaluation widens this to include interests who might be affected by positive or negative environmental impacts and interests that can affect success of the intervention. Traditionally, because this wider group may not be strongly connected with the evaluated intervention, they may not otherwise have a voice in the evaluation.
Understanding the place also involves identifying the elements of natural systems that may be impacted. Going to the site could assist in identifying these interests and identifying the elements of natural systems that may be impacted. It allows direct observation of interactions between human and natural systems and engagement with different interests. It also enables sharing stories and other ways of learning about the place from Indigenous and local communities and others.
Optimally, physically going to the place is recommended. However, if this is not possible, visiting 'virtually', such as through a facilitated video call, creative use of GIS or ensuring that the evaluation team includes someone with knowledge of the place are alternatives. Whether physical or virtual, the place should be visited during the evaluation design phase when the approach can be adapted to better address nexus interactions and the place.
For global, regional or national interventions, multi-site programs and policies, a thoughtful sampling of 'places' will be required.
3. Expand spatial and temporal framing
When considering environmental impacts in an evaluation, the scope of the theory of change (TOC) will often need to be expanded. This includes:
- the spatial framing – expanding the TOC beyond a specific 'site' to the broader landscape
- the temporal framing – expanding the TOC beyond the short time frame of an intervention to consider longer-term temporal framing to accommodate impacts in natural systems and for future generations of humans
Few natural systems can be considered local. Their boundaries often cross the boundaries of human and other natural systems: a river passes through communities, territories, and countries; multiple nations can share fisheries; political and social boundaries do not confine air pollution and natural systems overlap, rarely align.
While human systems might operate within the administrative boundaries of a local township and the boundaries of federal department, management units often differ dramatically and do not align with, natural systems which frequently span multiple administrative boundaries (Rowe 2012). The effects of interventions inevitably engage spatial scales in natural systems that differ from the spatial scales in which the intervention and human systems are framed.
Likewise temporal scales within natural systems and between human and natural systems can vary radically. Consider the life spans of many insects (days or weeks) and turtles (up to a century) or trees (many centuries). Also consider human constructs such as results management which are often constructed on annual cycles even when applied to elements of natural system with can be unlikely to show any effect in the first year or even five years. Evaluation needs to be much more open to multiple temporal scales if it is to recognise and assess effects.
Considering longer-term consequences can include "the fix that fails" (an apparent solution that has negative consequences), ripple effects, and spill over effects. It's essential to explore long-term impacts that may not be evident during the life of an intervention or evaluation.
It is also important to choose appropriate measures.
4. Use systems thinking and frameworks
A system can be defined as "a set of connected things that work together for a particular purpose" (Macmillan Dictionary, 2022). Systems can be composed of human elements, non-human elements, or a combination. Systems can overlap other systems and can be made up of smaller systems. The planet has a host of different systems that couple and affect each other, are contingent, often mutually dependent, and are the foundation of the sustainability of all the systems' elements.
Systems thinking is at the core of understanding systems. It can be used to identify key mechanisms and threats and to develop responses. A forest system provides an excellent illustration. A forest is a complex natural system in which trees are the dominant life form. In keeping with modern ecology, humans are considered part of the system. Collectively the trees in a forest sequester carbon, reduce evaporation and filter and regulate the flow of water, thereby inhibiting flooding and flows of sediments and nutrients harmful to the water systems. A forest also cools and nourishes the soil through leaf, limb and tree fall, aided by the actions of fauna such as ants. These processes contribute to the well-being of the forest system and the systems for forest dwelling or visiting species of flora and fauna. Recent advances have revealed the intense and synergistic coupling of fungi and tree root networks that enable sharing resources and communications among these major forest dwellers (Bennett et al, 2017). The coupling and flows in the forest system and across the various systems associated with forests are affected by and can affect other systems. This can include other natural systems, such as atmospheric or water systems, that overlap with a forest system. It can also include human systems, which can overlap when humans pursue their interests in forests. Human interests include visiting a forest for mental health, recreational, spiritual and cultural value, or extractive activities such as medicines, cosmetics or logging.
Viewed through a systems lens, cutting trees is not just an economic activity but also, directly and indirectly, affects other aspects of human and natural systems. This can include disruptions to human systems, such as health, recreation and spiritual systems. It can also affect natural systems through increased flooding, soil and snow destabilisation, carbon sequestration reduction, and reduced habitat for species contributing to biodiversity loss. These and other effects flow from cutting down trees and associated forest management practices, directly affecting the sustainability systems.
When addressing environmental sustainability in evaluation, systems thinking is necessary to identify how human actions couple with other systems and how natural systems couple with human systems. This is key to mapping the reach and relevant scales for an intervention. Systems thinking is central to understanding how human and natural systems couple, how human actions affect sustainability, and how changes in natural systems affect humans. In his 2014 blog on BetterEvaluation, Bob Williams presents systems thinking as the "combination of three things: 1. Understanding interrelationships, 2. Engaging with multiple perspectives, and 3. Reflecting on boundary choices." When approaching the task of incorporating environmental sustainability, evaluations will need to address:
- the existence, importance and character of relationships across and within natural and human systems, and identifying the most significant of these
- engaging all of the interests representing the various systems and entities – including human and non-human interests
- incorporating the systems' different spatial and temporal boundaries, including thinking beyond human boundaries (such as property or district borders or project timeframes).
For reasons of feasibility, evaluations will need to prioritise which systems and aspects of these systems to focus on.
5. Focus on the big issues
Given the wide range of potential impacts on natural systems and the challenge of exploring these within an evaluation's limited resources, it is vital to choose what to focus on carefully. There could be a risk of overloading an evaluation with an unmanageable number of potential environmental impacts. Or the focus might be narrowed inappropriately and focus only on impacts that are easy to measure or do not conflict with performance on social and economic criteria.
There are two parts to this: identifying relevant issues and deciding the most important ones.
An evaluation should be open initially to all of the elements that affect the value of the intervention. This includes all the systems the intervention reaches directly and indirectly through its actions (see systems thinking above).
For example, an intervention might destroy or damage a wetland area. Doing so affects many flora and fauna and connects to important water quality and supply management matters, biodiversity, and important human values such as mental and spiritual health and recreation. Big issues can be local in nature as they are often connected to the locale of the intervention.
Some overriding non-local matters will often rise to the top of what counts. For example, whether the intervention contributes to worsening or mitigating adverse climate effects. Other overriding issues may involve contributions to a cumulative problem that cannot tolerate further worsening, even if relatively modest. For example, every breeding pair is critical for critically endangered bird populations, and every female to critically endangered whales.
Issues of interest can be captured with a map or another conceptual or descriptive means. This is best developed with representatives of the interests who can identify matters of greatest concern to them, helping the evaluation understand the strongest and most critical elements in the ToC. Identifying the 'big' issues is as much a social as a technical undertaking. More information on identifying relevant issues can be found below in the section Identifying points of nexus.
The process of determining the most critical issues to focus on is also important. Identifying the significant issues is best as a co-generative activity and consensus decision.
It's essential to engage all key interests to reach a consensus on the "big issues" to be addressed, how they should be addressed, and the processes to gain and interpret information. This can include consultation with local and Indigenous communities, natural systems scientists, environmental organisations, project planners, managers, and evaluators. This engagement is important for several reasons. First, engaging with a wide range of interests and knowledges increases the evaluation's ability to understand the range of potential issues and narrow these down to the most relevant. Broad engagement also increases the likelihood that the evaluation findings will be used in decision-making and helps to strengthen an evaluation's credibility. It also helps evaluation focus on issues salient to those that can affect decisions and help the timing of evaluation reporting to align with the timing of decisions. Credibility and legitimacy are particularly important when dealing with the evaluation of environmental sustainability. One of the common forms of resistance to evaluation of environmental sustainability is through "duelling sciences" arguments employed by interests seeking a more favourable evaluation or interpretation of an evaluation. Engaging with multiple knowledges (natural sciences, traditional and local knowledge, social sciences and economics) can help to counter or avoid these arguments.
6. Draw on multiple types of evidence and expertise
Evaluations that consider environmental sustainability need to draw on diverse evidence, insights and knowledge beyond that usually gathered. This includes evidence and expertise from different sources and disciplines, such as local and Indigenous communities, natural systems sciences, project planners, managers, and evaluators.
Specific expertise may be required to understand the relevant systems and obtain the necessary knowledge inputs for the evaluation, including access to data and identifying existing research and evaluation findings. Previous research and evaluation evidence can identify potential natural systems impacts (or the precursors of these impacts) that a footprint evaluation could investigate.
Example: Diverse data sources used in evaluating the environmental sustainability of a strategy
This case illustrates the range of data sources used to evaluate cross-cutting environmental issues in a National Private Sector Development Strategy. Collecting data from a variety of data sources and the analysis of available data was an iterative, ongoing process during the evaluation. Sources included:
- Government sources, including relevant planning documents, national, state and local policies, legislation and regulations, contracts, enquiries, and reviews
- International environmental commitments
- Academic research articles and university theses about environmental risks and impacts
- Articles in newspapers and trade association newsletters
- Google Maps and Google Earth
- A Geospatial information analysis conducted by a GIS expert identified potential risks to the environment and people from tanneries and industrial parks. This involved drawing on several data sources to create maps showing river networks, watersheds and water flow, prevailing winds and population densities in the vicinity of industrial parks and tanneries.
Source: Footprint Evaluation Initiative 2022. Evaluation of Environmental Sustainability Aspects of a National Strategy.
B. Getting environmental sustainability on the agenda
What can be done during the commissioning and early phases of an evaluation
Many commissioners, evaluators and users of evaluations are increasingly open to and welcoming of addressing environmental sustainability in evaluations. However, it can still be challenging to get the consideration of natural systems onto an evaluation's agenda if it is not a stated goal of an intervention.
The commissioning phase provides opportunities for bringing sustainability into the evaluation and includes groups that will be increasingly open to addressing sustainability. Evaluators and evaluation commissioners have different responsibilities and scope for making decisions and advocating for particular choices in an evaluation.
Typically, an evaluation commissioner and departmental, organisational or program staff involved in managing evaluation recruitment and contracting can:
- Ensure that environmental sustainability is explicitly included in the scope of the evaluation in the Terms of Reference
- Discuss with primary expected and potential users of the evaluation the relevance of information about environmental sustainability for meeting their needs in terms of accountability and learning
- Draft Key Evaluation Questions which include environmental sustainability - potentially drawing on the OECD DAC criteria and/or the Footprint Evaluation Key Evaluation Questions to inform these.
- Ensure adequate knowledge in natural systems is available by requiring this as part of the evaluation team or by providing access to technical advice and review
Typically, an evaluation team can:
- Design and conduct the evaluation using processes that are co-generative of knowledge and involve a wide range of interests directly in evaluation processes
- Engage expertise in natural systems through expanding the team or by securing access to technical advice and review
- Advocate to include consideration of environmental sustainability in the scope of the evaluation, including adapting Key Evaluation Questions and expanding sources of evidence
- Engage expertise in natural systems through expanding the team or by securing access to technical advice and review
- Present evidence relating to natural systems and support intended users to understand it
In more collaborative evaluation approaches where the potential and likely users of an evaluation are engaged early in the evaluation process, they can:
- Advocate to have the evaluation address matters important to them (e.g. local community processes and effects, Indigenous ceremonial and traditional values)
- Advocate to have the evaluation engage with key interests beyond those normally regarded as stakeholders (e.g. environmental agencies and organisations)
The commissioning phase provides opportunities for bringing sustainability into the evaluation and includes groups that will be increasingly open to addressing sustainability. Be clear about what decisions you can influence and who else should be included in these decisions. Also, consider how you might encourage others to make decisions that support the inclusion of environmental sustainability.
Safety and harm
It's also important to consider the context when considering when what, and how to advocate. Seek advice from a trusted source with knowledge of your context if advocating for or including environmental considerations in an evaluation could put yourself or others at risk of personal harm.
Strategies for advocating for the inclusion of environmental sustainability
Several strategies can be used to advocate for the inclusion of environmental considerations in an evaluation Terms of Reference (TOR) where this is not an explicit project or policy objective.
These strategies can also be relevant for evaluators advocating for changes to a TOR if environmental sustainability has not been explicitly included.
The following strategies are discussed below:
- Draw on the OECD DAC Criteria
- Build environmental sustainability into the Key Evaluation Questions
- Reference existing environmental commitments, policies and regulations and law
- Make the case that environmental challenges are critical and urgent
Draw on the OECD DAC Criteria
Many evaluations are organised around explicit criteria. In international development evaluation, these are often operationalised in terms of the criteria developed by the OECD DAC (Organisation for Economic Co-operation and Development, Development Assistance Committee) – such as relevance, effectiveness, efficiency, impact, sustainability and coher
The OECD DAC criteria were revised in 2019. These revisions aimed to support better evaluation and lead to better policies to advance the 2030 Agenda for Sustainable Development (including the Sustainable Development Goals) and the Paris Agreement within the United Nations Framework Convention on Climate Change.
The revised versions of the OECD DAC criteria explicitly or implicitly include consideration of environmental sustainability. Highlighting this can provide a valuable rationale for including environmental sustainability in the scope of an evaluation.
Build environmental sustainability into Key Evaluation Questions
Another way to include environmental sustainability in all evaluations is to build it into the high-level Key Evaluation Questions (KEQs) that guide the entire evaluation.
KEQs are often developed through discussions between the evaluators, managers and commissioners, and evaluation users as part of framing the boundaries of the evaluation. Even when KEQs are included in the evaluation's TOR, there can be scope for revision after the evaluation team has been chosen.
One way to do this is to add a specific question related to environmental sustainability. Alternatively, it can sometimes be a better approach to integrate the consideration of natural systems into several or all KEQs. The Footprint Evaluation Initiative has developed a generic set of KEQs incorporating environmental sustainability to assist with this.
The Footprint Evaluation KEQs are purposefully framed as generic questions that can be adapted to meet the needs of specific evaluations. While it may not be possible to include all the questions in an evaluation, considering and discussing the Footprint Evaluation KEQs can lead to informed decisions about what questions to include or exclude.
The Footprint Evaluation KEQs can be used as an advocacy tool by:
- Demonstrating what generic footprint KEQs look like
- As a starting point for exploring how to adapt existing KEQs to include environmental sustainability
- Acting as a focal point for a discussion about what to include or exclude in an evaluation and why
- Demonstrating how to operationalise the OECD DAC criteria, including where these criteria consider environmental sustainability (see above)
View the Footprint Key Evaluation Questions here.
For more information on using the OECD-DAC criteria and the Footprint Evaluation KEQs, we recommend viewing the following Footprint Evaluation webinar:
Footprint Evaluation Webinar 2: Entry points for environmental sustainability: In part two of this three-part webinar series, Jane Davidson and Patricia Rogers discuss several ways to get sustainability on the evaluation agenda, even for projects that have no explicit environmental objectives and where there is no mention of environmental considerations in the Terms of Reference. The approaches explored include using evaluation criteria, such as the OECD DAC criteria and using ‘Footprint-savvy’ Key Evaluation Questions (KEQs).
Reference existing environmental commitments
Understanding and referring to existing environmental commitments a country or organisation has signed up for can help get environmental sustainability on an evaluation agenda. These could be included in an evaluation as part of addressing criteria related to relevance or cohesiveness. Existing commitments can also be used as a standalone rationale for illustrating the importance of including environmental considerations in an evaluation.
Keyword searches of national and local policies, strategies and plans can help discover environmental commitments internal to a country or region. Tools such as the World Fact Book can also be a valuable source of information about what international agreements a government has committed to.
Example: Referencing environmental objectives in a National Development Plan
One of the stated objectives of the evaluation of Uganda's National Strategy for Private Sector Development (NSPSD) was to "Realign the NSPSD with National Development Plan III, especially the Private Sector Development Program goal and objectives." The National Development Plan includes a number of environmental objects. Therefore, the evaluation report listed these objectives and assessed the extent that the NSPD had addressed them.
Source: Footprint Evaluation Initiative 2022. Evaluation of Environmental Sustainability Aspects of a National Strategy
Make the case that environmental challenges are critical and urgent
When broaching the subject of including environmental considerations in an evaluation, it can be helpful to present a clear, logical and supported argument.
To do this means making the case that including environmental considerations in evaluation is important and urgent because threats to the environment and human health and well-being are important and urgent.
Extensive literature supports arguments about the importance and urgency of global environmental crises. Much of this also discusses how people and nations with fewer resources will have less capacity to implement climate change mitigation strategies. Extensive evidence also points to how equity and environmental sustainability intertwine. This includes analysis that explores the distribution and consequences of environmental harm, such as pollution and climate change or losing access to traditional territories, which disproportionately affect people who are marginalised and disadvantaged.
One recent and widely reported source that can be helpful to draw on is the IPCC's Climate Change 2022: Impacts, Adaptation, and Vulnerability. Contribution of Working Group II to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. Six key messages to take away from this report are:
- Climate impacts are already more widespread and severe than expected.
- We are locked into even worse impacts from climate change in the near term.
- Risks will escalate quickly with higher temperatures, often causing irreversible impacts of climate change.
- Inequity, conflict and development challenges heighten vulnerability to climate risks.
- Adaptation is crucial. Feasible solutions already exist, but more support must reach vulnerable communities.
- But some impacts of climate change are already too severe for adaptation. The world needs urgent action now to address losses and damages.
Source: Levin, Boehm and Carter, 2022 6 Big Findings from the IPCC 2022 Report on Climate Impacts, Adaptation and Vulnerability. https://www.wri.org/insights/ipcc-report-2022-climate-impacts-adaptation-vulnerability
C. Understanding the nexus of human and natural systems
What is 'nexus'?
The concept of nexus is central to including environmental sustainability in evaluation. For Footprint Evaluation, 'nexus'2 refers to the causal connections and interactions between human and natural systems.
These can include:
- Causal interactions where human actions affect natural systems
- Causal interactions where natural systems have impacts on human systems
- Interdependencies, where human and natural systems depend on and affect each other
An example of interdependencies is restoring wetlands, which benefits both natural systems (increased wildlife habitat) and human systems (filtering, cleaning and storing water for use by urban human populations).
Benefits for one system can also come at the cost of the other, for example, extractive activities such as logging or mining, unless managed with a win-win mindset.
Nexus interactions between human and natural systems are often complex and dynamic, entangling systems with multiple nexus points, including reciprocal effects and feedback loops. Nexus relationships are usually non-linear and can involve thresholds, surprises and legacy effects. The systems that come together at nexus will have different temporal and spatial scales.
A well-described nexus setting from the Coupled Human and Natural Systems Network (CHANS) describes an intervention to protect giant pandas (Liu et al, 2007). This required conserving and expanding types of bamboo that only reproduce once every hundred years. Tourism to view giant pandas and their habitat was introduced to compensate local communities that drew heavily from the bamboo forests for fuel and building materials. However, tourism introduced new needs for fuel and further contributed to breaking up the contiguous character of the bamboo forests necessary for giant panda habitat.
A single intervention will involve human and natural systems connecting and interacting at one or multiple points. The causal character of these connections is why many refer to the systems as "coupling" rather than connecting.
The concept of nexus and connected or coupled human and natural systems is found in many disciplines, including economics (eg Ostrom, 2007), sustainability-science eg Clark and Dickson, 2003) , and contemporary ecology. Juha Uitto was an early user of the concept in evaluation, such as in the evaluation of two decades of UNDP work at the poverty-environment nexus in global, regional and national initiatives (UNDP, 2010).
The key is that all interventions will engage human and natural systems and that their nexus points are essential for bringing sustainability into evaluation.
Some ways of identifying nexus and possible consequences
Identifying nexus and possible consequences should draw on several complementary sources, processes and frameworks. These might include:
- Key informant interviews – including representatives of natural system interests, local communities, resource users and Indigenous people;
- Theories of change – to identify the systems involved;
- Existing research – for example, the case on Personal Protective Equipment during the COVID-19 pandemic (being finalised for publication) drew on a range of existing research;
- Previous evaluations of similar interventions – including Environmental Impact Assessments (EIA) /Environmental Impact Statements (EIS) – for example, the case on National Private Sector Development Strategy (LINK) drew on EIA reports for some of the industries being expanded under the Strategy;
- Rich pictures – a way to explore, acknowledge and define a situation and express it through diagrams to create a preliminary mental model. A rich picture helps to open discussion and come to a broad, shared understanding of a situation. Stakeholders and interested parties can be invited to develop a diagram or map representing the situation as they see it. This can include connections and interactions between human and natural systems without being constrained by particular formats;
- Existing regulations, guidelines and policies – can provide relevant and extensive, well-researched details about environmental risks and how they might or ought to be managed. For example, the case on the Private Sector Development Strategy drew on UNIDO's guidelines for industrial parks to identify potential environmental risks and how they should be managed;
- Lifecycle analysis – considers issues at different stages of a product or program lifecycle – for example, procurement, design and manufacture, distribution, consumption, disposal, or construction and operation;
- Ecosystem services/ecosystem functions framework – identifies and generates a value for the services ecosystems provide to human systems. These services can include regulating water flow and reducing the risk of flooding or filtering water used for drinking. Exploring ecosystem services, especially if put at risk by an intervention, can help identify potential costs in terms of negative impacts and provide a more accurate picture of the costs and benefits. The ecosystem services framework has the advantage and disadvantage of framing the cost of negative environmental impacts in terms of the associated negative effects on services provided to human systems. This risks perpetuating and reinforcing the human-centric worldview. For this reason, the label 'ecosystem functions' is sometimes used. In some situations, however, this framework can be handy as it can speak to value systems that do not place an intrinsic value on natural systems;
- Planetary boundaries - nine identified planetary boundaries or global environmental limits within which humanity could develop and thrive: climate change, novel entities (plastics, antibiotics), stratospheric ozone depletion, atmospheric aerosol loading, ocean acidification, biochemical flows (nitrogen and phosphorus), freshwater use, land-system changes, and biosphere integrity;
- Key drivers of biodiversity loss – as identified by the IPBES (Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services) (PDF) include changes in land and sea use, direct exploitation of natural resources, climate change, pollution, and invasion of alien species.
D. Collecting data
Using existing evidence
Given the resource constraints for considering environmental sustainability, it is particularly important to use existing data wherever it is sufficiently accurate, relevant and feasible.
-
Existing monitoring data: This can include monitoring data of the implementing organisation or department's activities and risk management compliance or monitoring by third parties, such as Auditor General reports. Government department monitoring of environmental conditions, including pollution levels and biodiversity, may also be available. Non-government organisations may also monitor environmental risk management systems and environmental conditions.
-
Existing research and evaluation: Academic research projects or publicly available evaluation reports can assist with developing an estimate of the impacts. This can include research and evaluation conducted at the same location as your evaluation and research and reports of similar initiatives from other locations
-
News reports: Credible news reports can provide relevant evidence, especially of historical events before the evaluation began.
-
GIS mapping: Geographic information systems (GIS) can draw on multiple datasets to generate a visual representation of various factors influencing environmental risks. For example, datasets on population density, prevailing winds, watersheds and rivers can be combined to produce a visual map that illustrates potential risks to the environment and human populations that rely on affected water catchments
-
Footprint calculators: Several calculators have been developed which produce an estimate of certain types of 'footprint' (i.e. an estimate of the level of impact that particular activities have on the environment). These can be used to track the estimated effect of changes in activities or to compare the likely effects of different options.
Direct measurement and information gathering
-
Interviews: Interviews with community members, program staff, and/or technical experts can be used to understand potential environmental impacts. Approaches to interviews with individuals may be convergent and seek maximum divergence in the sample of people interviewed or focus on key informants. Interviews may be structured, semi-structured or unstructured and in-depth. Focus group discussions are a method for conducting group interviews.
-
Stories: Eliciting stories from community members and other relevant groups can facilitate reflection and provide insights into programme processes, show impact, demonstrate innovation, build understanding of contextual factors, and support numerical data.
-
Citizen science projects: Citizen science projects can involve the crowdsourcing of data collection and/or analysis. These projects often focus on environmental issues, such as water and air quality, tracking biodiversity markers, monitoring plastic, etc. Projects that are relevant to your evaluation's location or that explore similar environmental impacts may be useful complements to other data sources.
-
Physical measurement: Direct measurement of environmental factors, such as air quality, soil, and water quality.
Sampling
Sampling is particularly important and potentially more challenging for monitoring and evaluations that include consideration of environmental sustainability. The spatial and temporal frames for natural systems differ from those that apply to human systems (see discussion about temporal and spatial framing above). Evaluations will often need to draw on sampling methods used in natural sciences, especially when considering the broader temporal and spatial scales. More traditional evaluation sampling methods will also be needed to address matters more familiar to evaluators, such as where the evaluation includes a large geographic scope. This requires careful consideration of:
- What is to be sampled – including sites, time periods, and species. The footprint evaluation principle focus on the big issues helps to frame what is to be sampled. The principle know the place helps ensure that sampling considers likely causal relationships and the interactions of the intervention with natural and human systems.
- How this will be sampled – including purposeful sampling, structured samples such as geographic grids frequent in some natural sciences, random sampling, and multi-level and multi-system sampling. For example, the GEF Independent Evaluation Office (IEO) often uses GIS to stratify natural system effects of interest and mixed method case studies to explore nexus within strata. In the case of initiatives and programs that cover multiple industries, a smaller number of industries can be purposefully sampled to cover a range of possible impacts, and then particular sites for these industries were sampled.
E. Valuing environmental impacts: From 'do no harm' to restoration
Evaluations answer evaluative questions about the value of what has happened or of alternative options – whether it is good, bad, better or worse.
Given the urgency to significantly reduce the harm human activities are causing to threatened natural systems and the need to restore the health of these systems, evaluations need to go beyond 'do no harm' when making judgments about environmental impacts.
This need is already recognised in the standards set by a number of global development organisations such as the UNDP, IFAD, and UNIDO7 adopting a 'do no harm' to natural systems minimum. Indeed, investments by these organisations are starting to move into restorative territory, actively restoring natural systems rather than simply avoiding further harm.
The Footprint Evaluation Initiative, building on Andy Rowe's earlier work, has developed a standard for assessing sustainability. The typology has four positions reflecting the likely net effect of environmental impact:
- Destructive – Extractive and damaging practices cause serious harm
- Harmful – sustainability-aware aware practices limit environmental damage
- Neutral – Practices cause no harm OR restoration offsets any harm
- Beneficial – Restores the natural environment so that it thrives
An evaluation team can use this typology to generate an evidence-based assessment of whether an intervention has damaged, done no harm, or helped to restore natural systems. This evaluation team-led approach requires that the evaluation team includes, or consults with, people with sufficient expertise and understanding of natural systems. The Footprint Evaluation Initiative is working on a rubric-based approach to support the use of the typology in all evaluations, including smaller evaluations and evaluations of interventions that do not have specific environmental goals.
Example: Thematic evaluation of support for smallholder adaption to climate (IFAD)
Proof of concept for this typology was provided in its first application as part of the International Fund for Agricultural Development (IFAD) thematic evaluation of support for smallholder adaptation to climate (Silici et al. 2022).
This evaluation was undertaken by an evaluation team with sufficient familiarity and understanding of natural systems conducting case studies and then working together to identify the likely value of the intervention's effects on natural systems.
The process involved:
- Thirty national case studies – (undertaken during the COVID pandemic, so site visitation was constrained but occurred in about a third of the case studies)
- Detailed case studies running 50 pages or more
- Several synthesis extractions (e.g. scaling up, environment, knowledge management) and a Rapid Evidence Assessment on those three topics
- GIS mapping physically positioned the interventions relative to important natural system features such as protected areas or endangered species.
- initial assessment using the typology by the lead for the sustainability assessment based on the case study materials
- an initial assessment was reviewed by the team member leading the case study and adjusted accordingly.
- Peer review of approach and findings involving the IFAD evaluation office and external advisors.
- A subsequent review by country and regional staff and managers occurred as the draft report took shape. If there was evidence supporting shifting the assessment, it was done.
The result of the sustainability assessment using the typology was an important part of the reports and presentations to the various levels of management and in the final report. The assessments of three countries were revised during the evaluation team review. One additional assessment was revised with the review by country and regional staff and managers.
This effort by the IFAD IEO is an important early example of systematically assessing the effects of human interventions across a range of countries and natural environments.
Using the typology for collaborative sense-making
The typology can also be used for collaborative sense-making when planning an evaluation or interpreting results. Using the typology as a tool for collaborative sense-making can facilitate a shift to win-win thinking as people understand what is important to each other and the natural world. When using the typology for this purpose, it's important to involve a range of diverse interests and perspectives. The knowledge and perspectives of Indigenous people, local communities, those directly and indirectly impacted by the intervention, those planning and implementing the intervention, and natural scientists can inform the development of rubrics to operationalise the typology.
This is illustrated in the table below – showing four stages of working with a range of interests to develop a shared understanding of what each position of the typology would look like (destructive, harmful, neutral, and beneficial) within the context of a specific intervention. Doing this in the planning stages of an intervention is especially helpful, as it allows for upfront reflection on what is needed to move towards beneficial outcomes for the environment.
![Table showing steps, approach, output and example of participatory synthesis](/sites/default/files/2022-10/Example%20of%20participatory%20synthesis.jpg)
For more information on this typology, we recommend viewing the following Footprint Evaluation webinar:
Andy Rowe and Patricia Rogers introduce a typology being developed that will assist a wide range of evaluations in assessing the effect of the interventions on natural systems and sustainability. Does the intervention achieve the standard we need where no net harm is caused to natural systems, does it contribute to restoring them, or is it harmful to these systems?
F. Resources
Guides
Discussion Papers
- Sustainability‐Ready Evaluation: A Call to Action
To facilitate the development of sustainability‐ready evaluation, this paper by Andy Rowe provides an initial checklist and references to useful resources.
This paper by Lamia Renaud provides guidance on Environmental Approach for Generational Impact – a proposed approach to designing and evaluating programmes in a way that includes consideration of environment and biodiversity issues.
This brief guide by Scott Chaplowe and Juha Uitto, as part of the EVAL-SDGS Insights series, presents the arguments for mainstreaming environmental sustainability into all evaluations and sets out factors addressed in a successful organisational change to do so.
Case study examples
Describes four 'thought experiments' undertaken as part of this project which involved revisiting a past, real-life evaluation and walking through how this could have been done differently to incorporate considerations of environmental sustainability.
This case study looked at how environmental sustainability might be addressed as part of a mid-term evaluation of a national strategy for private sector development.
This case study explores the feasibility and value of considering environmental sustainability in the evaluation of personal protective equipment (PPE) provisioning during the COVID-19 pandemic.
Tools
The key evaluation questions (KEQs) are designed to support the inclusion of environmental sustainability by embedding consideration of the environment in each evaluation question rather than adding environmental considerations as a standalone question.
This resource, from the Footprint Evaluation Initiative, discusses how the six OECD DAC evaluation criteria can be used to get environmental sustainability on the agenda for evaluations and monitoring.
This is a simple step by step guide to using The World Factbook to find the international environmental agreements a country has signed up to.
Blog
- Memorial Day Week: Sustainability-Ready Evaluation by Andy Rowe
In this AEA365 blog, Andy Rowe argues that human system interventions have direct effects on the natural system; ignoring these direct effects causes evaluation to have a systematic and positive bias.
This resource, by Andy Rowe, Jane Davidson, Dugan Fraser, Patricia Rogers and Alice Macfarlan, provides an introduction to the work underway by the footprint evaluation initiative so that all evaluations address environmental impacts.
- Incorporating environmental considerations into UNEG Member Agency Evaluations by David Todd
A discussion of challenges faced by UNEG member evaluation where guidance from the UN Working Group on Integrating Environmental and Social Impact into Evaluations (ESI) would be particularly useful.
Video
- Integrating environment into evaluation
The session explores how evaluations, where the evaluand is not an environmental program, can take full account of the environment, drawing on practical experiences from the Footprint Evaluation project and considerations from non-environmental agencies.
Identifying points of nexus between human and natural systems: In part one of this three-part webinar series, Andy Rowe and Patricia Rogers discuss what was learnt during the Footprint Evaluation Initiative’s ‘thought experiments', as well as diving into what it means to evaluate ‘at the nexus’ of human and natural systems.
Entry points for environmental sustainability: In part two of this three-part webinar series, Jane Davidson and Patricia Rogers discuss several ways to get sustainability on the evaluation agenda, even for projects that have no explicit environmental objectives and where there is no mention of environmental considerations in the Terms of Reference. The approaches explored include using evaluation criteria, such as the OECD DAC criteria and using ‘Footprint-savvy’ Key Evaluation Questions (KEQs).
From 'do no harm' to restoration: Working with a typology of Footprint Evaluation: In the last part of this three-part webinar series, Andy Rowe and Patricia Rogers introduce a typology being developed that will assist a wide range of evaluations in assessing the effect of interventions on natural systems and sustainability. Does the intervention achieve the standard we need where no net harm is caused to natural systems, does it contribute to the restoration of natural systems, or is it harmful to these systems.
Other related initiatives
- Blue Marble Evaluation
Focuses on evaluating global systems change initiatives
- Transforming Evaluation
Looks at a range of ways evaluation needs to be transformed to support the transformations needed to reach SDGs.
- UNEG Working Group
On incorporating environmental and social impact into evaluation
- Global Footprint Network
Who have developed the Ecological Footprint calculator
- Impact Investing
Environmental Social Governance indicators in Impact Investing
- CES Report (PDF, 866KB)
On stocktaking for sustainability-ready evaluation
About the Footprint Evaluation Project
This thematic page is an output of the Footprint Evaluation Initiative - a co-creation and research collaboration supported by the Global Evaluation Initiative.
You can find out more about the Footprint Evaluation Project here, including the project's learning questions and planned outputs.
This thematic page is a work in progress and we will be sharing more resources and examples here. Join the Footprint Evaluation Discussion Group to add suggestions for resources or other organisations and events relevant to footprint evaluation.
Sources
Bennett, J. A., Maherali, H., Reinhart, K. O., Lekberg, Y., Hart, M. M., & Klironomos, J. (2017). Plant-soil feedbacks and mycorrhizal type influence temperate forest population dynamics. Science, 355(6321), 181-184.
Clark, W. C., & Dickson, N. M. (2003). Sustainability science: the emerging research program. Proceedings of the national academy of sciences, 100(14), 8059-8061. https://www.pnas.org/doi/10.1073/pnas.1231333100
Flanagan, Richard. (2021). Toxic : the rotting underbelly of the Tasmanian salmon industry. [Australia]: Penguin Books
Footprint Evaluation Initiative (2022). Evaluation of Environmental Sustainability Aspects of a National Strategy.
Hejnowicz, A., Barbrook-Johnson, P., Blackstock, K., and Staddon, C. (2018) The Nexus: A New Approach to Sustainability Transformations – What, Why and How. CECAN. https://www.cecan.ac.uk/blog/the-nexus-a-new-approach-to-sustainability-transformations-what-why-and-how/
IPCC (2022). Climate Change 2022: Impacts, Adaptation, and Vulnerability. Contribution of Working Group II to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change [H.-O. Pörtner, D.C. Roberts, M. Tignor, E.S. Poloczanska, K. Mintenbeck, A. Alegría, M. Craig, S. Langsdorf, S. Löschke, V. Möller, A. Okem, B. Rama (eds.)]. Cambridge University Press. In Press.
IPCC, 2018: Annex I: Glossary [Matthews, J.B.R. (ed.)]. In: Global Warming of 1.5°C. An IPCC Special Report on the impacts of global warming of 1.5°C above pre-industrial levels and related global greenhouse gas emission pathways, in the context of strengthening the global response to the threat of climate change, sustainable development, and efforts to eradicate poverty [Masson-Delmotte, V., P. Zhai, H.-O. Pörtner, D. Roberts, J. Skea, P.R. Shukla, A. Pirani, W. Moufouma-Okia, C. Péan, R. Pidcock, S. Connors, J.B.R. Matthews, Y. Chen, X. Zhou, M.I. Gomis, E. Lonnoy, T. Maycock, M. Tignor, and T. Waterfield (eds.)]. Cambridge University Press, Cambridge, UK and New York, NY, USA, pp. 541-562. https://doi.org/10.1017/9781009157940.008
Levin, K. Boehm, S. and Carter, R. (2022). 6 Big Findings from the IPCC 2022 Report on Climate Impacts, Adaptation and Vulnerability. World Resources Institute. https://www.wri.org/insights/ipcc-report-2022-climate-impacts-adaptation-vulnerability
OECD/DAC Network on Development Evaluation. (2019). Better Criteria for Better Evaluation: Revised Evaluation Criteria Definitions and Principles for Use. https://www.oecd.org/dac/evaluation/revised-evaluation-criteria-dec-2019.pdf
Ostrom, E. (2007). A diagnostic approach for going beyond panaceas. Proceedings of the national Academy of sciences, 104(39), 15181-15187. https://www.pnas.org/doi/10.1073/pnas.0702288104
Radoslav S. Dimitrov; The Paris Agreement on Climate Change: Behind Closed Doors. Global Environmental Politics 2016; 16 (3): 1–11. doi: https://doi.org/10.1162/GLEP_a_00361
Rowe, A. (2012). Evaluation of natural resource interventions. American Journal of Evaluation, 33(3), 384-394. https://journals.sagepub.com/doi/10.1177/1098214012440026
Rowe, A., & DeLancey, D. (2021, July 21). Mainstreaming Sustainability in Evaluation: Canadian Evaluation Society Stocktaking Report. Retrieved from European Evaluation Society: https://europeanevaluation.org/2021/07/12/mainstreaming-sustainability-in-evaluation-canadian-evaluation-society-stocktaking-report/
Silici, L., Knox, J., Rowe, A., Nanthikesan, S. (2022). Evaluating Transformational Adaptation in Smallholder Farming: Insights from an Evidence Review. In: Uitto, J.I., Batra, G. (eds) Transformational Change for People and the Planet. Sustainable Development Goals Series. Springer, Cham. https://doi.org/10.1007/978-3-030-78853-7_13
Todd, D. (2020). UNEG Working Group on Integrating Environmental and Social Impact into Evaluations Volume One Main Report. UNEG. http://www.unevaluation.org/document/detail/2951
Uitto, J. I. (2019). Sustainable development evaluation: Understanding the nexus of natural and human systems. In G. Julnes (Ed.), Evaluating Sustainability: Evaluative Support for Managing Processes in the Public Interest. New Directions for Evaluation, 162, 49–67. https://onlinelibrary.wiley.com/doi/abs/10.1002/ev.20364
UNDP (2010). Evaluation of UN DP Contribution to Environmental Management for Poverty Reduction: The Poverty-Environment Nexus. https://erc.undp.org/evaluation/evaluations/detail/4784
United Nations / Framework Convention on Climate Change (2015) Adoption of the Paris Agreement, 21st Conference of the Parties, Paris: United Nations
Wentworth, J. (2016, December). The Water-Energy-Food Nexus. POSTnote, Issue 543. Parliamentary Office of Science and Technology.
Williams, B. (2014). Week 36: Systems thinking. BetterEvaluation. https://www.betterevaluation.org/en/blog/systems_thinking
Zylstra, P., Lindenmayer, D. and Bradshaw, D. (2022, May 23). Coming of age: research shows old forests are 3 times less flammable than those just burned. The Conversation. https://theconversation.com/coming-of-age-research-shows-old-forests-are-3-times-less-flammable-than-those-just-burned-179571
Zylstra, P., Lindenmayer, D. and Bradshaw, D. (2022, May 23). Coming of age: research shows old forests are 3 times less flammable than those just burned. The Conversation.
Expand to view all resources related to 'Footprint evaluation'
Resource
- Addressing environmental sustainability through the OECD DAC criteria for evaluation of development assistance
- Environmental approach for generational impact
- Environmental sustainability: 2022 NEC Conference session recording
- Equitable evaluation: Voices from the Global South
- Evaluating the environmental impact of personal protective equipment (PPE) in the COVID-19 pandemic
- Footprint evaluation case study: Evaluation of environmental sustainability aspects of a national strategy
- Footprint evaluation webinar 1: Identifying points of nexus between human and natural systems
- Footprint evaluation webinar 2: Entry points for environmental sustainability
- Footprint evaluation webinar 3: From 'do no harm' to restoration - Working with a typology of Footprint Evaluation
- Footprint evaluation: Thought experiments
- Guide to using the World Factbook to find a country’s international environmental agreements
- How can evaluation change the world? Empowering Young and Emerging Evaluators (YEEs) to contribute to transformational change (Webinar)
- Key Evaluation Questions (KEQs) to guide Footprint Evaluations
- Mainstreaming environmental sustainability in evaluation: EvalSDGs Insight #14
- Navigating sustainability: measurement, evaluation and action
- OECD-DAC principles for evaluation of development assistance
- Participatory tools and techniques for assessing climate change impacts and exploring adaptation options
- Sustainability-inclusive evaluation: Guidance for Global Affairs Canada to include consideration of the environment in all evaluations
- Sustainability-inclusive evaluation: Launch of Footprint Evaluation guidance (webinar)
- Sustainability-inclusive evaluation: Why we need it and how to do it
- When the world is in crisis, evaluation can provide answers
'Footprint evaluation' is referenced in:
Blog
Framework/Guide
- Evaluation career guide :
Method
Page