Week 8: Guest blog: Innovation in development evaluation

Development aid is changing rapidly – so must development evaluation.

This is the second post in our series of innovation in development evaluation. Thomas Winderl, an evaluation consultant and co-author of ‘Innovations in monitoring and evaluating results’ explains why evaluation needs to keep pace with an increasing understanding of complexity in development planning, why multi-level mixed methods will be the new norm, and why evaluators need to get more imaginative about primary data collection. Read part one in this series here.

1. Planning in a complex, dynamic environment requires more and different evaluations

Linear, mechanistic planning for development is increasingly seen as problematic. Traditional feedback loops that diligently check if an intervention is ‘on-track’ in achieving a pre-defined milestone do not work with flexible planning. In their typical form (with quarterly and annual monitoring, mid-term reviews, final evaluations, annual reporting, etc.), they are also too slow to influence decision-making in time.

A new generation of evaluations is needed: one which better reflects the unpredictability and complexity of interactions typically found in systems; one which gives a renewed emphasis to innovation, with prototypes and pilots that can be scaled up; and which can cope with a highly dynamic environment faced by development interventions.

This is an exciting opportunity for monitoring and evaluation to re-invent itself. With linear, rigid planning being increasingly replaced by a more flexible planning approach that can address complex systems, we find now that we need more responsive, more frequent, and ‘lighter’ evaluations that can capture and adapt to rapidly and continuously changing circumstances and cultural dynamics.

To do this we need two things: firstly, we need up-to-the-minute ‘real-time’ or continuous updates on the outcome level; this can be achieved by using, for example, mobile data collection, intelligent infrastructure, or participatory statistics that can ‘fill in' the time gaps between the official statistical data collections. Secondly, we need to use broader methods that can record results outside a rigid logical framework; one way to do this is through retrospectively ‘harvesting outcomes’, an approach that collects evidence of what has been achieved, and works backward to determine whether and how the intervention contributed to the change.

2. Multi-level mixed methods become the norm

Although quantitative and qualitative methods are still regarded by some as two competing and incompatible options (in his recent blog, Michael Quinn Patton compared them to two-year olds not yet able to play together), there is a rapidly emerging consensus that an evaluation based on a single method is simply not good enough. For most development interventions, no single method can adequately describe and analyze the interactions found in complex systems.

Mixed methods allow for triangulation – or comparative analysis – which enables us to capture and cross-check complex realities and can provide us with a full understanding, from a range of perspectives, of the success (or lack of it) of policies, services or programmes.

It is likely that mixed methods will soon become the standard for most evaluations. But using mixed methods alone is not enough; they should be applied on multiple levels.

Graph: Example of a multi-level mixed method to evaluate a language school

3. Outcomes count

There is broad agreement that what ultimately counts - and should therefore be closely monitored and evaluated - are outcomes and impact. That is to say, what matters is not so much how something is done (=outputs, activities and inputs), but what happens as a result. And since the impact is hard to assess if we have little knowledge on outcome results, monitoring and evaluating outcomes becomes key.

There is one problem, however: by their nature, outcomes can be difficult to monitor and evaluate. Typically, data on behaviour or performance change is not readily available. This means that we have to collect primary data.

The task of collecting more and better outcome-level primary data requires us to be more creative, or even to modify and expand our set of data collection tools.

It will no longer be sufficient to rely on non-random ‘semi-structured interviews with key stakeholders’, unspecified ‘focus groups’, and so on. Major primary data collection will need to be carried out prior to or as part of an evaluation process. This will also require more credible and more outcome-focused monitoring systems.

Thankfully, there are many tools becoming available to us as technology develops and becomes more widespread: Already small, nimble random sample surveys such as LQAS are in more frequent use. Crowdsourcing information gathering or the use of micro-narratives can enable us to collect data that might otherwise be unobtainable through a conventional evaluation or monitoring activity. Another option is the use of ‘data exhaust’: data which is passively collected from people’s use of digital services like mobile phones and web content such as news media and social media interactions.

Table: Eleven innovations potentially useful for innovative development evaluations

Source: Discussion Paper: Innovations in Monitoring & Evaluating Results, UNDP, 05/11/2013

So there is good reason for optimism. The day will soon come when it is standard practice for all evaluations to be carried out by mixed methods at multiple levels, with improved primary data collection enabling us to evaluate what really counts in our interventions: the outcomes and the impact.

Coming up next week: Innovations in advocacy evaluation and views from the African Evaluation Association conference on innovation in evaluation.

Image: PT Bambu - Ibuku Green School