Sustained and Emerging Impacts Evaluation (SEIE)

Sustained and emerging impact evaluation (SEIE) evaluates the enduring results of an intervention some time after it has ended, or after a long period of implementation.

1. What is SEIE?

Sustained and Emerging Impacts Evaluation (SEIE) refers to an evaluation that focuses on impacts some time after the end of an intervention (which might be a project, policy or group of projects or programs) or after the end of participants’ involvement in an ongoing intervention.  It examines the extent to which intended impacts have been sustained as well as what unintended impacts have emerged over time (positive and negative).  

This type of evaluation is often referred to as ‘post-project’ impact evaluation to distinguish it from ‘impact evaluation’ done near the end or shortly after an intervention ends.  The label, Sustained and Emerging Impacts Evaluation, emphasises the focus on long-term impacts, and the need to include not only intended impacts but also unintended impacts.

An SEIE can identify the factors that have led to many of the activities and outcomes being sustained years later (for example, participants still retaining knowledge and practising particular behaviours) and to enduring impacts (for example, sustained reductions in malnutrition post-project). An SEIE can also document emerging or unexpected results such as the spread of knowledge and use to new communities or completely unexpected new directions the participants and partners have taken the project or their own new linkages from training provided.

It is most commonly done for interventions with a finite timeframe, such as projects funded or implemented by international development agencies, multilateral organizations or philanthropic foundations, or local projects funded for a period of time by a national government.

2. Why do SEIE?

SEIE provides the “missing piece” of evaluation in the project/programme cycle, as shown in the following diagram, developed by Valuing Voices:

As this graphic shows, when sustainability objectives are explicit in project design, they also become integral to monitoring and evaluation throughout the project cycle.  SEIE is thus ideally the culmination of a process for evaluating sustainability.

It is, however rare to return to evaluate years after funding has ended. Many ‘impact evaluations’ only evaluate impacts near the end of a project and therefore draw conclusions about the lasting value of an intervention based on incomplete evidence.

Sustained and Emerging Impacts Evaluation is important for accountability when it includes the voices of intended beneficiaries and institutions responsible for continuing initiatives after projects close out.  SEIE promotes learning about the lasting results and contributes to informing better interventions in the future. SEIEs can produce important lessons about the quality of past design, what works in what contexts and why. SEIE is particularly important when there is insufficient information about the likely trajectory of change – how impacts will change over time.

Findings at the end of a project are not always a reliable predictor of lasting benefit.  After an intervention ends, what might happen is that the impacts that have been achieved might be maintained, increase, or decrease. Additional important impacts might appear. An SEIE gathers evidence to understand which of these has occurred, and how. For example:

Sustained impacts: A child-centered livelihood project in Kenya that the international NGO returned to after nine years found that some investments were sustained. Water tanks and VIP latrines were maintained as were the health and financial services buildings and Irish potatoes (Plan International 2010).

Increased impacts: A food security project in Niger  fostered sustained agricultural practices that contributed to 30% improvement in food security three years after closeout. These sustained results were seen to be due to a handover process that involved partners and participants for over two years (CRS Niger 2015)

A microenterprise project that included training participants as ‘agents of change’ led to increased results after the end of the project, with microenterprise banks spreading and assets tripling (PACT Nepal 2006)

Decreased impacts:  A review of data on more than 200,000 water systems showed that ten years after installation roughly one quarter were no longer operational  (IDB 2016)

Emerging impacts: A drought/rehabilitation project that reduced women’s labour in water hauling had an unexpected positive result in terms of reducing gender violence (Lutheran World Relief/ Gates Foundation, Niger 2013)

Without this additional information, decisions about what to fund, next generation project design and strategies for implementation and monitoring rely on assumptions that are in many instances based on what was happening when resources were still flowing during and at the end of the project cycle.

If the impacts of intervention are only evaluated at the end of funding, it creates a number of risks, including:

• That choices will be made to achieve levels or types of shorter-term outcomes that cannot be sustained or that don’t lead to longer-term impacts

• That the value of ultimately more sustainable and effective approaches will not be appreciated and effective investments and approaches will be terminated too soon

• That emerging impacts are completely hidden from our understanding which robs us of lessons of what communities themselves valued enough to sustain with their own resources or created anew from what we catalysed. 

Such lessons are key for design and new funding, monitoring and evaluation.

Sustainable Development Goals

Given the international focus on Sustainable Development Goals (SDGs) for all countries, it is important to undertake evaluations that build knowledge about what is effective in producing enduring benefits. SEIE can be a valuable new contribution to achievement of the SDGs by illuminating what has been sustained long-term in particular contexts. Currently SGD indicators do not actually link the sustainability of interventions to the status of sectoral indicators and change over time. Adding SEIE data (as sentinel sites) to the SDG tracking on what interventions have or have not contributed to sustained improvement at the country level can be very illuminating for future progress and programming.

Fostering local ownership of results (in country or the region) by involving participants, stakeholder institutions, and local evaluators in the process of evaluation can encourage use of the results. 

3. When to do SEIE?

The timing for SEIE evaluation should depend on the expected change trajectory. It needs to be long enough afterwards to be able to see some change from initial impacts, but not so long that it is hard to collect data and not possible to use the information to inform decisions. 

Many SEIEs have been done between 2 and 10 years after an intervention has ended.  In some cases it has been much longer – for example, the evaluation of the long-term impacts of international development projects in the Koshi Hills area of Nepal reviewed the combined and enduring impact of various projects and programs that had been implemented in the area over a period of 30 years.

4. Who should be engaged in the evaluation process?

A number of decisions need to be reached about the evaluability, terms of reference, profile of the evaluation team, and management of the evaluation.

Decision-making about the scope of an SEIE might be done by:

• the original funder
• the local implementing agency or agencies
• a local community organisation
• the national- recipient- government
• a research organisation
• a combination of the above.

Those involved in conducting the evaluation (in terms of collecting, analysing and reporting data, and clarifying the evaluative criteria that should be used) might include:

• the original funding agency
• the local implementing agency
• an external evaluator or evaluation team not based in the country or community
• a local evaluator or evaluation team
• the local community
• a combination of the above

For example, the Koshi Hills SEIE was managed jointly by the Government of Nepal and the UK Department for International Development, and undertaken by a consortium of external evaluators, including international evaluators working in partnership with local evaluators.  Many SEIEs have been privately funded by international NGOs, managed internally, and carried out by external evaluators.

Involving local implementers, community members and local evaluators is important in order to understand the local context and factors affecting the sustainability of impacts. A combined evaluation team of local and outside evaluators can provide opportunities for mutual learning, act as valuable sounding boards for expectations and assumptions, and situate findings in terms of international trends. 

Read more on who should be engaged in the evaluation process:

Participatory evaluation

5. What processes and methods can be used to do an SEIE?

As with any evaluation, there are different methods and processes that can be used, and they should be chosen carefully to suit the purpose of the SEIE, the nature of what is being evaluated, and the availability of time, money, data, and other resources. And ideally planning for an SEIE is incorporated into planning the monitoring and evaluation system for the intervention.

Define | Understanding what is being evaluated

It can be helpful to develop a theory of change which includes a theory of sustainability – how will the intended impacts be sustained?

• Did the design explicitly anticipate sustainability of impacts or shorter-term outcomes?
• Will the intervention need ongoing funding (for example, will local users pay to maintain water infrastructure)? Or will a local agency provide ongoing funding or staff to maintain a service?
• Were participants capacitated to sustain activities themselves (for example, knowing value-chain individuals to sustain marketing?)
• Or will the impacts endure or even surpass expectations even if the activities don’t (for example, transformative change in social norms or impact of technological change).

For example, USAID/Food for Peace has adopted a four pillared exit strategy and sustainability framework that considers capacity, resources, motivation, and linkages as “catalysts for change” beyond the donor funding.

Develop Theory of Change: this page has guidance on developing a theory of change, along with a number of resources for further reading.

Frame | Intended users and uses, key evaluation questions and determine what ‘success’ looks like

There are a number of options that can be used to identify possible criteria and standards that could be used in an evaluation, drawing on formal and informal sources. There are also a number of options that can be used to negotiate which should be used and how they should be weighed.

Determine What ‘Success’ Looks Like: this page has links to a number of these methods.

One approach that is often recommended for use in an SEIE to determine what success looks like is a participatory approach. When using a participatory approach, the values used to define “success” to make evaluative judgements, will ideally not reflect only the funders’ criteria and standards, but particularly the community who are intended to be the beneficiaries of the intervention. Their different views on success should ideally be taken into account when determining:

• which impacts should be included in the evaluation,
• what levels of performance will be considered a success, and
• how performance will be synthesised into an overall judgement – in particular, whether positive impacts will be outweighed by negative impacts, and whether the distribution of benefits is a consideration (for example, did the most disadvantaged receive any benefit from the project, some benefit, the most benefit?). 

Describe | Answer descriptive questions, describe impacts

There are many options for measuring or describing impacts and how conditions have changed over time, including:

• Big data, large data sets that cannot be analysed using conventional methods, often produced as by-products of engagement, such as social media data
• Direct measurements(for example biophysical measurements of water quality or power generation)
• Interviews of individuals, communities and key informants, focus group discussions
• Observation – through Field Trips, Non-participant Observation, Participant Observation, Photography/video, Transect
• Project records and review of reports and secondary data
• Questionnaires using email, face to face, internet, mail, mobile, or telephone delivery
• Official statistics – e.g. census data, social statistics, HMIS, EMIS data
• Sketch mapping - creating visual representations ('map') of a geographically based or defined issue.

An effective mix of primary and secondary data collection and analysis methods, and appropriate sampling and adequate response rates, is ideal in order to triangulate between the limitations of any one data source.   Desk reviews, involving reviewing self-reported data from organisations, are unlikely to be sufficient by themselves. 

Given the challenges of conducting an SEIE after some time has passed since the end of an intervention, some creativity may be required in method selection when participants or key informants (including implementing staff) have moved, complete baseline or endline data from the intervention or participant lists are unavailable.  Selection of approaches need to consider whether the evaluation may meet resistance from stakeholders because of project commitments that may have not carried over into post project realities.

It is particularly useful to have data on the original project results, participant and partner lists to find and follow up with these respondents or others who were closely involved during the implementation period – as well as know how the area and communities have changed during the intervening years to now.  Very often, at the end of an intervention, data is aggregated and buried in implementing agency data-archives and reports retained in PDF format in donor databases with little local access. Only recently has some project data begun to be shared in open-data sets that are nationally available to local evaluation teams. 

Understanding causes | Methods for answering causal questions

A major challenge for SEIE evaluations is causal inference – explaining whether, or to what extent, observed changes are attributable to a particular intervention.  This is particularly difficult over a long time period, where projects with the same objectives have come on stream in the same localities in the intervening period, or where the combined impact of several projects are being examined.  

In some cases, a counterfactual design has been established from the beginning or can be created for the SEIE evaluation. 

For example, a long-term evaluation of the Head Start program followed up children who had been randomly assigned to participate in Head Start or to be in a control group (US DHHS, 2010). 

In some cases it might be possible to use a quasi-experimental design, such as a regression discontinuity approach, and compare those just above or below a threshold for inclusion in an intervention, or to find adequately matched sites to create a comparison group.

In many cases, however, it may require some creativity to create a credible counterfactual group either prospectively and especially retrospectively.  If it is not possible to create a credible counterfactual, non-experimental options will be needed – for example, process tracing, comparative case studies or Qualitative Impact Protocol, which use a range of data to test the plausibility of the causal inference. These methods and approaches can be particularly useful for investigating the contribution of a specific project within a crowded implementation environment with many other projects.

Synthesise; Report and support use | Combining and using evidence

Before starting an SEIE it is important to anticipate how findings will be shared with those who may benefit from the findings.  This includes the implementing agencies, as well as other agencies that implement or fund similar programs in the same context.  It is of course also vital to share findings back with those being evaluated – communities, local partners and local governments - for veracity confirmation, for knowledge management and to improve future programming, post-SEIE. 

A variety of reporting formats might be needed to make findings accessible to different audiences.  Report and Support Use has some ideas on different ways of reporting.