The Triviality of Measuring Ultimate Outcomes: Acknowledging the Span of Direct Influence

This article, written by Giel Ton, Sietze Vellema and Lan Ge for the IDS Bulletin (Volume 45 Number 6), argues that instead of using household surveys to assess the effects of programs the focus should be on detailed measurement of more immediate outcomes through the use of contribution analysis. The authors use a detailed case study from Côte d’Ivoire to demonstrate their results.

Excerpt

"The private sector and market-led strategies have become increasingly central to development policy and practice. Moreover, non-governmental organisations (NGOs) are teaming up with companies or private–public partnerships. This shift from public to private-led development strategies is based on changing expectations of the role of trade versus aid for poverty alleviation. In many donor countries, this policy is increasingly based on the assumption that the private sector is more effective in reaching development goals than development aid through governments or NGOs. Accordingly, donor agencies have begun to re-allocate public resources to companies and private–public partnerships. From a public perspective, the obvious question for impact evaluation is how to demonstrate this assumed effectiveness.

In general, donor agencies prefer precise measurements of net effects in relation to the Millennium Development Goals (MDGs), with income generation and poverty reduction as main objectives (DCED 2010; DGIS 2011). This often translates into survey-based research designs, including baseline studies, randomised sampling, and comparison groups. This article challenges the exclusive emphasis on precise measurement of income effects in quasi-experimental evaluation designs. Net effects, especially those related to business performance and income, are influenced by a wide range of intervening factors that are impossible to control for under real-world conditions. This makes it difficult to attribute effects to the actual interventions and provides little information on the effectiveness of developmental activities. Based on our experiences with impact studies of certification-induced training programmes for farmers (Ton, Vellema and de Ruyter de Wildt 2011; Ton 2012; Waarts et al. 2012; Waarts et al. 2013a, 2013b, 2013c), we argue that there are good reasons to limit this dominant focus on measuring net effects in ultimate outcomes, and propose to shift attention to the domain of immediate and intermediate outcomes."

Contents

• Setting: impact evaluation of certification
  • Public accountability requirements
  • Harmonised indicators and rigorous measurement
  • Refined theories of change
  • The evaluation challenge in certification
• Case study: evaluation of training for certification in Côte d’Ivoire
  • Multiple actors, treatments and contexts
  • Sample sizes for rigorous measurements
• Discussion
• Conclusions