Week 23: Evaluation of anti-corruption reforms and programmes – don’t give up before you even try!

By
Jesper Stenberg Johnsøn

Our blogger this week is Jesper Johnsøn, Senior Advisor to the U4 Anti-Corruption Resource Centre

Jesper highlights a frequent confusion among anti-corruption practitioners between the difficulty measuring levels of corruption and the evaluability of anti-corruption initiatives, and urges us not to give us on rigorous evaluation.

You rarely enjoy being (rigorously) evaluated. So you make up excuses, such as “I am different, so you can’t compare me to these other guys,” or “these assessments make no sense, so I won’t pay much attention to the results (except if they come out in my favour).” Some areas have additional excuses for why they are special. I often hear that “we cannot reliably measure levels of corruption because it is a hidden activity.” This is only partly true, but it has been the ultimate excuse for not investing in evaluations. This mistakes the difficulty of corruption diagnostics for an impossibility of evaluating, meaning donors and governments seldom evaluate their anti-corruption efforts systematically. They give up in advance. As my father would say: you have to at least try! 

We at the U4 Anti-Corruption Resource Centre noted a glaring evaluation gap in the anti-corruption field. This was highlighted by two meta-evaluations by the World Bank, and U4 donors and the Asian Development Bank. In response, we built up a work stream on evaluation and measurement to promote better evaluations and to improve the global body of knowledge on what works and why in anti-corruption. The two meta-evaluations found that anti-corruption interventions had either no theories of change, or very poor ones: our tool for programme design and evaluation can help with that. We also show how to use different types of evaluation methods for anti-corruption initiatives, and how proxy indicators could advance our ability to measure corruption. I will give you a few examples on how we can avoid being stuck trying to refine ways to diagnose corruption risks at the country level, and start paying attention to evaluations.   

The search for indicators

Various organisations have invested substantial financial and intellectual resources into measuring “corruption” as a social phenomenon at the country level. There are a range of cross-country, composite indices of corruption – the best known are Transparency International’s Corruption Perceptions Index and the World Bank’s Control of Corruption component of the Worldwide Governance Indicators. These are useful advocacy tools, but offer little for assessing a country’s trajectory of fighting corruption. Nor do they contribute to the global body of evidence on what works and why in anti-corruption.

Country-wide perception measures have long been used as the main effectiveness measure for anti-corruption activities. This is often not fruitful as the attribution gap back to the intervention is too large, and perceptions change only slowly over time. New, more experience-based survey tools can help us measure anti-corruption effects with higher accuracy, particularly for bribery and fraud. Ideally, the anti-corruption field would be blessed with census data and large-scale household surveys, but such costly investments are not made. Luckily, though, there are usually easily available alternatives to country-level perceptions-based measures, such as the Global Corruption Barometer.

Finding one indicator that can capture dynamics and changes in different types of corruption at the country level is very difficult. Composite indices are often biased towards bribery – just one type of corruption. U4 launched the Proxy Challenge initiative last year to think beyond existing, standardised, aggregate, cross-country corruption indices. We should explore how bespoke indicators work in specific country contexts – working bottom-up rather than top-down. I wrote about that in the Global Anti-Corruption Blog.

Evaluation, not measurement

We would all like one indicator that tells us everything, but that is an impossible quest. It is wrong to assume that corruption diagnostics alone will enable us to assess whether anti-corruption reforms are working at the country level. To do this we need to evaluate specific reforms and programmes. As we observe in a recent U4 paper on evaluation methods, most evaluations unfortunately rely on interviews as the sole source of data. The paper therefore shows how a variety of evaluation methods, from randomisation to RealWorld evaluation methods, can be used on anti-corruption interventions. We recommend using mixed methods evaluation designs, and to pay more attention to mapping and refining the theory of change underpinning the reform. The challenge of measuring (some types of) corruption does not necessarily translate into poor evaluability. When we move from an abstract, country-level measure of corruption to specific outcomes of relevance to an initiative, it becomes easier to identify good indicators. With strong evaluation designs using comparison groups, it is sometimes not even necessary to directly measure corruption. The effects of anti-corruption measures can be assessed by the difference in performance between those with anti-corruption measures, and those without.   

So, let’s not give up before we have even tried!

Jesper will be contributing a new BetterEvaluation theme page on anti-corruption initiatives later this year.