The balancing act of research impact

Over recent decades, governments everywhere have increased their scrutiny of public spending, and public universities have not escaped this scrutiny.

Evaluation of academic research plays a significant role in government efforts to steer public universities. The scope of such evaluation is now being extended to include the ‘relevance’ or ‘impact’ of academic research outside academia.

In the context of the Australian Government’s National Innovation and Science Agenda, Senator Simon Birmingham (Minister for Education and Training) has just announced a 2017 pilot initiative to test the impact of academic research. It will include both quantitative and qualitative information, and will involve universities, industry and other end-users of research.

Non-academic research impact is of course tricky to measure, but ANZSOG EMPA Director Prof Michael Mintrom (with co-author Dr Andrew Gunn) has recently published some design principles to help inform these kinds of efforts.

The key question being asked is: How is this research relevant to the world outside the academy and does it deliver benefits beyond the research community? Such impact could be technological, environmental, economic or social. It could affect the policies, strategies, and actions of businesses, governments, NGOs, and community groups. This is a very different matter from how impact has traditionally been assessed in the academic community, which mainly measures the influence of published books and articles through their citation counts.

Gunn and Mintrom address how evaluation of non-academic research impact can measure impact effectively without undermining academic freedom and research excellence. They consider five questions on evaluation design:

1. What should be the object of measurement?
2. What should be the timeframe?
3. How should non-academic users of research inform evaluation processes
4. How should controversial impacts be managed?
5. When in funding cycles should impact evaluation occur?

What should be the object of measurement?

Accurately tracing the line of influence from one specific research project to a well-defined non-academic impact is challenging work. A tight definition of project-to-impact may also exclude impacts that can’t be attributed to particular academic outputs but flow from academic work more generally. However, impacts not flowing from a specific output are more difficult to directly attribute to academic research, therefore creating further methodological challenges.

What should be the timeframe?

Some research impacts might come fast – especially when technologies with well-known properties can be matched to well-specified problems. Other research impacts, particularly those from basic research, could take years. This suggests the importance of informing research impact assessment exercises with some historical perspective.

How should non-academic users of research inform evaluation processes?

Having academics alone assessing and rating non-academic impact may lead to a lack of confidence in the evaluation process. This is because academics, using the values from within the academic community, could privilege impacts perceived and valued by academics as of importance. This means there is a case for bringing research users into the evaluation process – but the total population of research users is as diverse as civil society. Only a sample of ‘non-academic voices’ could ever inform evaluations of research impact.

The inclusion of non-academic assessors in this new generation of research evaluations could challenge the perceived ‘regulatory capture’ of academics assessing their own work, and diminish academic control. But this raises the issue of ‘balance’ in the evaluation of academic work – too much control by outside entities could undermine long-held values of academic freedom and research excellence.

How should controversial impacts be managed?

Economic impact can be measured in dollar terms or in improved efficiency or productivity. Societal impact can be more complicated to measure, as efforts to define what is good for society quickly run into controversy. Impact can also be associated with activities that are politically divisive, such as mining with hydraulic fracturing, the use of stem cells, or GMOs.

When in funding cycles should impact evaluation occur?

Distinctly different incentives can be established for researcher depending on where impact evaluations occur in research funding cycles.

Gunn and Mintrom conclude that non-academic impact should be selectively promoted and evaluated. This is how greater gains from research will be best captured without imposing misguided and onerous reporting requirements on individuals and institutions.

This blog post was originally posted on the Australia and New Zealand School of Government (ANZSOG) blog. It has been republished with ANZSOG's permission. You can find the BetterEvaluation resource page for the article discussed here.

Related Resource

Evaluating the non-academic impact of academic research: Design considerations

In this journal article, Andrew Gunn (University of Leeds) and Michael Mintrom (ANZSOG/Monash University ) explore how evaluation of non-academic research impact can measure impact effectively, without undermining academic freedom and research excellence.