Impact evaluation: A guide for commissioners and managers

This guide, written by Elliot Stern, aims to support managers and commissioners in gaining a deeper and broader understanding of impact evaluation.

The resource builds on an initial report prepared for the UK Department for International Development (DFID) Broadening the range of designs and methods for impact evaluations.

The impetus for this guide came from a ‘cross-funders group’ interested in helping decision-makers within civil society organisations and those that fund them to better understand how to commission, manage and use impact evaluations. 

Extract

"This ‘design guide’, as the title suggests, starts from the assumption that:

• Evaluation design is a vital stage in the overall impact evaluation process. If neglected, it will have negative consequences down the line in terms of the relevance, validity and usability of evaluation outputs.
• It is important for those who commission, manage and use Impact Evaluations to have access to frameworks and guidance. These allow them to ask the right questions of the specialist evaluators who will in the end do the IE work that is needed.

The audience for this guide are those who:

• Draw up IE terms of reference
• Have to assess IE proposals that cross their desks
• Manage and steer ongoing IEs
• Wish to assess the strength of conclusions and recommendations reached by those conducting IEs
• Need to develop new programmes and policies that are ‘evidence-based’, ie, learn lessons from completed IEs

In depth evaluation and methodological expertise is not assumed in this guide – rather readers are expected to
have familiarity with evaluation issues and challenges; and with the demands of socio-economic development programmes. The guide signposts more specialist sources and references, but is mainly interested in equipping practical managers in the development sector with enough knowledge to allow them to have meaningful conversations with technical experts.

This guide builds on a major report funded by the Department for International Development that was published in 2012: Broadening the Range of Designs and Methods for Impact Evaluations. That report, which including annexes exceeded 120 pages, was intentionally more technical and more geared to evaluation specialists rather than managers and practitioners. The 2012 report provides an additional point of reference for those wishing to further deepen their understanding of IE. Some readers of this guide will undoubtedly wish to cross-refer to sections of the earlier report to pursue some issues in greater depth and this is signposted in the text."

Contents

• 1. Introduction and scope 2
• 2. What is impact evaluation? 4
  • Defining impact and impact evaluation 4
  • Linking cause and effect 5
  • Explanation and the role of ‘theory’ 7
  • Who defines impact? 7
  • Impact evaluation and other evaluation approaches 8
  • Main messages 9
• 3. Frameworks for designing impact evaluation 10
  • Designs that support causal claims 10
  • The design triangle 11
  • Evaluation questions 11
  • Evaluation designs 13
  • Programme attributes 14
  • Main messages 15
• 4. What different designs and methods can do 16
  • Causal inference: linking cause and effect 16
  • Main types of impact evaluation design 20
  • The contemporary importance of the ‘contributory’ cause 21
  • Revisiting the ‘design triangle’ 21
  • Main messages 23
• 5. Using this guide 24
  • Drawing up terms of reference and assessing 
  • proposals for impact evaluations 25
  • Assessing proposals 25
  • Quality of reports and findings 27
  • Strengths of conclusions and recommendations 28
  • Using findings from impact evaluations 29
  • Main messages 29
• Annex 30