Using case studies to explore the external validity of ‘complex’ development interventions

Michael Woolcock's paper addresses the fact that rising standards required for assessing the impact of development projects have not been met with equally rigorous procedures for guiding decisions about external validity - "whether and how similar results might be expected elsewhere" (Woolcock 2013).

Woolcock offers a basic analytical framework for assessing the external validity of complex development interventions, focusing on the use of case studies to identify particular conditions present when diverse outcomes are found.

Excerpt

"Even if concerns about the weak external validity of RCTs/QEDs—or for that matter any methodology—are acknowledged by most researchers, development professionals still lack a useable framework by which to engage in the vexing deliberations surrounding whether and when it is at least plausible to infer that a given impact result (positive or negative) ‘there’ is likely to obtain ‘here’. Equally importantly, we lack a coherent system-level imperative requiring decision makers to take these concerns seriously, not only so that we avoid intractable, non-resolvable debates about the effectiveness of entire portfolios of activity  (‘community health’, ‘justice reform’) or abstractions (‘do women’s empowerment programmes work?’)6 but, more positively and constructively,so that we can enter into context-specific discussions about the relative merits of (and priority that should be accorded to) roads, irrigation, cash transfers, immunization, legal reform etc with some degree of grounded confidence—i.e., on the basis of appropriate metrics, theory, experience and (as we shall see) trajectories of change. 

Though the external validity problem is widespread and vastly consequential for lives, resources and careers, my modest goal in this paper is not to provide a ‘tool kit’ for ‘resolving it’ but rather to promote a broader conversation about how external validity concerns might be more adequately addressed in the practice of development. (Given that the bar, at present, is very low, facilitating any such conversations will be a non-trivial achievement.) As such, this is a paper to think with. Assessing the extent to which empirical claims about a given project’s impact can be generalized is only partly a technical endeavor; it is equally a political, organizational and philosophical issue, and as such useable and legitimate responses will inherently require extended deliberation in each instance. To this end, the paper is structured in five sections. Following this introduction, section two provides a general summary of selected contributions to the issue of external validity from a range of disciplines. Section three outlines three domains of inquiry (‘causal density’, ‘implementation capabilities’, ‘reasoned expectations’) that for present purposes constitute the key elements of an applied framework for assessing the external validity of development interventions generally, and ‘complex’ projects in particular. Section four considers the role analytic case studies can play in responding constructively to these concerns. Section five concludes." (Woolcock 2013, p.3)

Contents

• Introduction  1
• External validity concerns across the disciplines: a short tour  3
• Elements of an applied framework for identifying ‘key facts’  6
  • 'Causal density'  7
  • 'Implementation capability'  10
  • 'Reasoned Expectations'  12
• Harnessing the distinctive contribution of analytic case studies  15
• Conclusion  16