Making rigorous causal claims in a real-life context: Has research contributed to sustainable forest management?

This article discusses an impact evaluation that examined the contribution of two forestry research centres - the Centre for International Forestry Research (CIFOR) and the Centre de Coopération Internationale en Recherche Agronomique pour le Développement (CIRAD) - to sustainable forest management (SFM) practices in the six countries in the Congo Basin.

A detailed theory of change was developed through stakeholder interviews, consisting of five different contribution pathways, or ways in which research activities could plausibly contribute to the intended impacts:

• Framing - CIFOR and CIRAD frame the forest issues in the Congo Basin, propose concepts that help in understanding and addressing these issues and contribute to raising the profile of forestry issues among stakeholders.
• Data- CIFOR and CIRAD provide country or region-specific data which helps stakeholders develop solutions adapted to the region’s context and needs.
• Solutions - CIFOR and CIRAD experiment with new solutions in local projects and provide operational solutions that stakeholders can implement.;
• Capacity - CIFOR and CIRAD train and support stakeholders in designing and implementing forestry-related activities and contribute to the exchange of good practices among them.
• Operationalisation - CIFOR and CIRAD provide guidance and data helpful to the development of adequate management plans and train people who sometimes become consultants or work at timber companies.

The evaluation presents an example of a rigorous non-counterfactual causal analysis that describes how different evidence and methods were used together for causal inference without a control group or comparison group.

It includes discussion of whether some contributions of the project could be qualified as necessary for the outcomes and explores potential generalisability of the findings.  The example brings together contribution analysis, process tracing and realist evaluation, using techniques such as identifying and ruling out alternative explanations and looking for "signature" processes that indicate what has produced outcomes.

This resource and the following information was contributed by Patricia Rogers.

Authors and their affiliation

Thomas Delahais and Jacques Toulemonde.

Key features

This resource provides a detailed example of using rigorous non-counterfactual causal analysis – that is, doing an impact evaluation without a control group or a comparison group.  This example explains how a number of methods were used together; a theory-of-change (program theory); contribution analysis; process tracing; modus operandi (or signature)

How have you used or intend on using this resource?

I haven’t yet used the resource  but plan to add it to my set of examples to use in evaluation capacity strengthening.  It would be ideal to develop some teaching support material around this case.  I think people learn a lot from engaging with real data from real cases.

Why would you recommend it to other people?

While there is increasing recognition, at least in some organisations, that it is not always possible to create a credible counterfactual (an estimate of what would have happened in the absence of a program – usually by creating a control group or a comparison group), and there is information about alternatives, there are few good examples available for people to both learn from and use to demonstrate their feasibility. 

This example could be used in evaluation training and advocates for a broader range of designs for impact evaluation as an example of a rigorous impact evaluation without a counterfactual, by evaluators providing more details about how to operationalise non-counterfactual options and approaches; and by commissioners and managers of evaluation as an example by which to hold accountable consultants who promise to produce a rigorous impact evaluation without a counterfactual.