Evaluation practice in Aboriginal and Torres Strait Islander settings – Evaluating the evaluations

A new project is underway to make the BetterEvaluation website more useful for those conducting or managing evaluations involving Aboriginal and Torres Strait Islander people.

As part of this, BetterEvaluation is working with Aboriginal and Torres Strait Islander people to share and promote their evaluation methods and processes, and to facilitate their feedback and reviews on evaluations that have been conducted in their communities or regions. You can view these outputs on our new set of theme pages: Evaluation in Aboriginal and Torres Strait Islander Settings. You’ll also be able to download PDF versions of some of the key materials for easy sharing.

Throughout this project, there has been a real focus on doing things ethically – this includes the guidance for and examples of evaluative practice, but also includes how the BetterEvaluation team and the project’s Working Group. Outlined below are some of the main outputs from this project so far, as well how the project came about and the processes that guided the work.

The process behind the project

When the project was started with a grant from the Department of Prime Minister and Cabinet (DPM&C), Indigenous Affairs, in July 2018, the need for cultural sensitivity and input from Aboriginal and Torres Strait Islander people was emphasised.

A Steering Committee and Advisory Group consisting of Aboriginal and Torres Strait Islander people and non-Indigenous people working in the evaluation sector was established. A number of people on the Advisory Group highlighted the need for a way to ensure appropriate conduct when managing valuable Indigenous Cultural Intellectual Property (ICIP) on the BetterEvaluation website as curated or co-created resources (see below for information on the Contributor Agreement).

With a second grant from DPM&C, a working group of six consultants with expertise in evaluation in Aboriginal and Torres Strait Islander settings was established. This working group included Belinda Gibb & Sharon Babyack (Indigenous Community Volunteers), Donna Stephens (Menzies School of Health Research), Carol Vale & Debbie Hoger (Murawin Consulting), Kate Kelleher (Kate Kelleher Consulting), supported by Greet Peersman (BetterEvaluation). The group agreed that it was important to put time and effort into communicating the approach they would be taking and to ensure systems were in place to protect ICIP. This was seen as an important step to building trust with contributors before engaging with them.

Key outputs

Code of conduct

To guide the BetterEvaluation team’s engagement with evaluators and communities* and how examples of good practice are shared on the website, a formal code of conduct was developed. It addresses 10 key principles and how the team puts these into practice and is in line with the Contributor Agreement. You can find our Code of Conduct explained here and downloadable here.

Ethical Protocol

The working group identified the need for guidance for evaluators on how to apply ethical principles to better protect the ICIP, rights and protocols of Aboriginal and Torres Strait Islander people and communities*. The group shared stories of where ethics had failed, the damage this has caused and the injustice of how data can be misused, misrepresented through a Western lens and stolen from the people it should serve to benefit.

The group recognised that there were many existing ethical frameworks and guides that could be draw on, including the work already done with Terri Janke and Co for the project’s Code of Conduct. After undertaking a literature review, the working group identified the need to tailor existing work to the evaluation sector in. The group categorised the principles and protocols from other ethical frameworks and guides and discovered the same themes emerging, differing in the specific language used. They then consolidated these themes and, drawing on their yarns, put them into their own words.

The protocol is currently based on desk-based research and yarning among a group of Aboriginal and non-Indigenous evaluators. Some of these evaluators are independent consultants, some are internal evaluators, others work in research and evaluation institutes. The experience and backgrounds of the working group are diverse, and a number of different Aboriginal groups are represented. The working group acknowledges the need to test the practice guide further with other Aboriginal and Torres Strait Islander people and communities, and view the guide as a working document.

You can find the ethical protocol explained further and downloadable here.

Examples of good practice

Rather than asking for contributors to share their IP and ICIP (such as existing guidelines and tools related to evaluation), the working group  agreed that it would be more valuable to, instead, collect  good practice evaluations and provide community members* the opportunity to evaluate these evaluations that involved them. The working group members were passionate about ensuring that the voice of Aboriginal and Torres Strait Islander People was at the heart of the project.

It was agreed that only examples of evaluations that received endorsement as a good practice from the community involved would be presented on the website. 

We currently have two examples presented on the website and are in the process of identifying more.

You can find these examples via the interactive map. This includes details about how the examples were identified, how they were assessed by the community members and how the evaluation put the ethical protocol into practice.

The Contributor Agreement

The BetterEvaluation core team was referred to Terri Janke and Co, an Indigenous legal firm specialising in ICIP, who developed a contributor agreement that brings together Western and Aboriginal and Torres Strait Islander protocols to protect the Intellectual Property (IP) and ICIP of contributors to the project. The working group adapted the initial contributor agreement to reflect the shift in direction towards highlighting the voice of communities about evaluations affecting them. The document was also revised based on the approach the Australian Institute of Aboriginal and Torres Strait Islander Studies (AIATSIS) takes with their contributor agreement. The group wanted to ensure it was user-friendly and appropriate for the contexts in which they would be using it. The contributor agreement has become an important discussion tool before any interview with the evaluators and community members*. It opens up opportunities for contributors to ask questions and feel confident and informed before they agree to participate. 

Data Sovereignty

The Indigenous Data Sovereignty Collective relates to a movement that aims to give ownership and control of Indigenous data back to Indigenous peoples. BetterEvaluation has signed on to the data sovereignty network and has adopted data sovereignty principles in the ethical protocol for the BetterEvaluation project. We encourage you to read more about Data Sovereignty here.

Read more

Evaluation in Aboriginal and Torres Strait Islander settings

The aim of these new pages is to help those conducting or managing evaluations in Aboriginal and Torres Strait Islander settings to learn from examples of good practice to improve the benefit of evaluation for Aboriginal and Torres Strait Islander people. Visit the new set of theme pages to find examples of good practice evaluations and guidance on applying the ethical protocol developed as part of this project.

A special thanks to those who contributed to this project

Project Working Group: Belinda Gibb & Sharon Babyack (Indigenous Community Volunteers), Donna Stephens (Menzies School of Health Research), Kate Kelleher (Kate Kelleher Consulting), Carol Vale & Debbie Hoger (Murawin Consulting), and Greet Peersman (BetterEvaluation).

We acknowledge the input and review provided by Elizabeth Mason, Solicitor at Terri Janke and Company and thank the members of the Advisory Group and the Steering Committee for their guidance and feedback.

We would like to acknowledge and thank Maria Stephens, an Arrabi/Binning woman who speaks the Iwaidja language. She generously provided her artwork for this project, which has been used as the feature image for this blog.

* The term ‘community’ is used to refer to a geographically based community defined by specific boundaries, a community organisation, or a community defined by a shared interest. Within the context of this project, an evaluation may not include interactions with all members of a community but could also be with one or more representatives of a community.