Better ways of choosing and using metrics in evaluation of impact investing

We invited Mishkah Jakoet shares some thoughts on how metrics can be more useful for impact investing. Mishkah brings considerable experience in evaluation for impact investing, including contributing to the revision of the IRIS+ indicators.

The IRIS+ metrics are currently under review, and being expanded to include a new catalogue of metrics, a much-anticipated update that promises much benefit within the global impact investment ecosystem. The GIIN is to be commended for undertaking this ambitious project, and for continuing to steer the global conversation relating to impact measurement.

While there’s some great work going on as part of this process, the more progress is made, the more questions arise. In particular, the conversation around these new metrics in impact investing is still very much about ‘what can we measure’, which brings up questions around  what each metric might actually mean in a particular context versus another context, and therefore how metrics might be used effectively or not.

Let’s look at the example of an investment fund that wants to reduce poverty, and is looking at ways of creating decent jobs as a means to achieving this. In this instance, the impact investment doesn’t just aim to create new jobs – the aim is to create decent jobs. ‘Decent jobs’ can be dependent on factors like job stability, decent labour conditions, gender equality within wage structures.

A common indicator used for job stability is how many people are in permanent, full-time employment versus temporary or part-time. However, the usefulness of this indicator varies greatly in different contexts. While it may be really useful in a city like Johannesburg, it’s not as relevant across Africa as a whole where the majority of jobs are in the agricultural sector, which is largely made up of temporary employment due to the business cycles of planting, harvesting, and other seasonal tasks. In this agricultural context, people often work a number of different jobs to earn a living, following the harvest from one farm to the next, and returning back to the same farms the next year. So, in order to measure the stability of this workforce, looking at permanent, full-time jobs is going to give an inaccurate picture of the actual stability. Instead, it would be better to use an indicator that is more appropriate to this context – such as the number of workers that return each year for harvest at the same farm, or the number of months per year an individual is employed.

And it’s not just the risk of not gaining an accurate picture of workforce stability that’s an issue here – there’s a real risk of inappropriate metrics causing harm.

Focusing on an indicator that favours full-time, permanent positions could drive investment out of innovations designed for the temporary and part-time workforce, like different gig economy platforms and other alternatives that are nascent and need investment. Worse still, if the focus of an impact investment fund was to create more full-time, permanent positions in agriculture, and funding and programmes incentivised farms to create these sorts of jobs, you would potentially decrease the number of workers that a farmer could employ. This would be disastrous to the overall goal of poverty reduction because you would have more people shifting into unemployment rather than having some form of income – but the overall metric of more full-time, permanent jobs would be achieved, and therefore the investment fund can report positively on its outcomes.

This is not a new insight.  This conversation has been going on between evaluation and impact investing for almost a decade and a half, but it remains to be seen how the use of metrics in impact investing can be reimagined. We have literature from all over the world with case studies from many different sectors and it shows the same thing: if you set up these ideal, simple metrics for what a service or programme should look like and tie them to funding then you're going to have a situation where people adjust their behaviour to meet these metrics – essentially they try to game the system, not from nefarious motives but  because they want to keep the money coming in. And this can have serious, unintended results.

When we think critically about the role of an impact investor we need to ask, are we really providing additional developmental support to economies, or are we just paying lip service to ‘profit plus good intentions’?

How might we improve the ways metrics are chosen and used in impact investing?

Improving metrics use in impact investing

1. Talk about why it’s important to get the metrics right

We need everyone to understand the serious problems that can arise when the wrong metrics are chosen – especially in terms of skewing investment or implementation in undesirable ways – and to share concrete examples that can be used as a reminder of why this is important.

2. Involve people with evaluation expertise in the choice and development of metrics

Metrics are often decided early on when choosing investments, before evaluators are involved and often without advice from people with expertise in measurement, data collection or data use.  This can lead to metrics that can be unfeasible, misleading or lead to undesirable consequences.

3. Use theories of change and scenarios to identify what information about impacts is needed and relevant

A good theory of change can identify what results are likely to be evident and at what stage, especially identifying those that will be evident by the time of reporting. A good theory of change can also show where there is good evidence that later impacts are likely, given the achievement of earlier outcomes.

4. Use a systematic process to develop metrics that will be feasible and useful in the ways intended

Each metric needs a feasible plan to gather, analyse and report metric data, as well as a test to determine whether it can be correctly interpreted under different scenarios.  The theory of change might show additional data that needs to be reported in order to correctly interpret metrics, such as contributing factors or changes in the context.

5. Know that you don’t always have to boil everything down to a single number or ratio

A single number can leave out important considerations, especially when there are multiple aspects of an impact that should be included.  Consider the use of rubrics to synthesise diverse evidence into a global scale which is transparent and empirical.

6. Learn from others’ experiences about actual use of metrics to inform investment and implementation

There is a global trend amongst investors to put together ‘use cases’ for their different metrics: how they actually use them and what they actually measure.  This is a good start, but it’s not the complete picture. In addition, evaluators would benefit from use cases that explain the link between findings and the decisions made using these metrics. This additional context would help evaluators understand why these metrics matter and how they could be made more useful.

As a field we need to collaboratively build knowledge about how metrics have been used, including learning from cases that haven’t worked, and share this knowledge.  This will be a big challenge as much of the information in the field is proprietary and not readily shared or opened to wider scrutiny. We need to change the incentives to encourage people to share more about what’s worked and what hasn’t. It has been a privilege to participate in the IRIS+ Decent Jobs Working Group and it is encouraging to see the GIIN mobilise a wide range of organisations and individuals to share openly and honestly during these discussions.

This is the first in a series of guest blogs on impact investing.  Do you have other suggestions for how to improve the way evaluation supports impact investing, or comments on the suggestions outlined here?