Week 12: Evaluation innovation in transparency and accountability

Innovation is a relative concept. It is about new practice … for the topic and person or group in question.

The context-specific nature of what constitutes ‘an innovation’ became clear during a recent event around global transparency and accountability efforts.

Last week the T/A Initiative (see Box)  hosted the second gathering of its community of practice. Over 70 activists, researchers, international NGOs, and donors converged in Jakarta to share ideas and challenges about how to evaluate and learn about ‘Transparency and Accountability’.

T/A initiatives leverage information to empower citizens to hold the state or private sector to account. But they differ greatly in strategies, scale, sector and stakeholders. Examples include working with local communities and citizen groups to promote budget transparency, supporting anti-corruption efforts through research and information, and using evidence on health system performance for social mobilization and advocacy. These initiatives make information available about expected and actual resource use and service performance and support its active use by citizens.

The inherently political nature of T/A work means that evaluation challenges are similar to those of other complex change processes (Rogers 2008). Many different players are involved in the many transitions from access to information, to how people perceive it, their ability to use it, the willingness of receivers to change and finally better performing services, such as schools or hospitals. These lengthy causal pathways are embedded in strong power inequalities. Changing such relations of power is difficult, dangerous and has no guarantees. Formal and informal channels need to be used to build trust with those who want change and those who need to be changed. Change is slow and subject to possible backsliding (Fölscher 2012).

Over the past ten years, the surge of donor interest in the aims of the T/A sector has been accompanied by a growing interest to see ‘proof’ of effectiveness. Efforts were ramped up, for example by the International Budget Partnership to document a series of impact studies and share insights on planning for impact.  Case studies have documented the positive changes in budget institutions, policies, allocations and the quality of expenditure mainly in middle-income countries, with encouraging signs in low- income and challenging environments.

During the Jakarta event, I was struck by three themes in relation to evaluation of the T/A sector.

First, many participants expressed the need for more investment in and capacity for refining their ‘theories of change’. Participants shared several cases of disappointment about unmet expectations and initial strategies that proved inadequate. The consensus seemed to be that prior – or as part of – setting out how to monitor and evaluate T/A efforts, it is critical to be clear about the assumptions between more transparency and more accountability. For example, many discussions recognized that transparency of information is just one strategy to improve social accountability. Participants queried whether perhaps it was better to start with social accountability and then figure out which strategy was best for that context, which might – or might not –information transparency. The T/A sector can learn much from ToC work elsewhere, such as by Hivos, from which to draw inspiration.

Second, more refined theories of change links directly to the challenge of defining success. Is it only when legislation has changed or when hospital patients have better quality care that success can be claimed? Or do the intermediate steps also count, such as encouraging youth to consider pushing their local councilors to publish budgets? Participants were clear about the need to document intermediate steps as progress towards success (and to understand the way change was evolving) while also recognizing the need to be humble. Particularly important was the observation that success depends on where you stand. One example was given of an international NGO who claimed success by gaining press headlines for local action and being kicked out of the region, while staff members of the local organisation were left with empty hands, socially ostracized and receiving threats. 

Third, the definition of success is shaped by often-difficult tensions between what participants called the ‘projectising’ of political change and the long term nature of T/A efforts. Some presenters described how their work was at risk of being viewed as bounded, technocratic projects, when activities had many different facets and scales and were aiming to shift power relations. Donor requirements play an oftentimes unhelpful and unhealthy pressure on T/A efforts to bound the work in terms of time and result guarantees, with limited flexibility for grabbing opportunities as they arise.  

These three themes were discussed intensely by participants at the Jakarta event. While new for the T/A sector, they echo questions many have pondered for the past decades in working with local organisations to develop their M&E systems and learning approaches. How do we improve our theories of change and then use this to develop our M&E systems? What best methods are there for understanding impact? How can more narrative approaches, such as outcome mapping, process tracing and citizen ethnography help us? How do we learn to learn? What is the link between M&E, research and learning? What cases of strong organisations can we learn from?

My takeaway from the TALearn event was that the challenge of innovation for the T/A sector is one of efficient learning about monitoring, evaluation, impact and learning that builds on advances in different sectors, adapting them to the specific questions and processes of social accountability.