Reflection on the review of the IEG@50 writing competition on culturally responsive evaluation

A recent essay competition, jointly hosted by the Independent Evaluation Group (IEG), the Global Evaluation Initiative (GEI), and EvalYouth, invited emerging evaluators to propose solutions for closing the gap between global and local knowledge. The submitted essays were assessed by four judges. They have been reflecting on the ideas presented in the essays. This blog presents reflections by Dr Mercy Fanadzo.

In reading through the 29 essays submitted for the young and emerging evaluators competition, a number of important themes and insights emerged.

The majority of the essay authors acknowledged that culturally responsive evaluations (CREs) are collaborative and promote learning. Therefore, CRE initiatives should be intentional, prioritized, and intensified to foster a sense of ownership and facilitate progress in learning, promote advocacy for its integration, and improve impact. Delayed collaboration may result in CRE remaining unfruitful and of niche relevance, as it is perceived.

Many essays discussed how culture influences the success of a program’s design and its implementation, the types of impacts of that program, the effectiveness of the evaluations, and uptake of evidence. They emphasized the impact of evaluators’ cultural knowledge in producing meaningful results aligned with cultural contexts and considered the positionality of evaluators, as well as social, political, and cultural values of specific contexts. The essays argued that accounting for positionality and cultural competency is vital in determining the meaningfulness of evaluation processes and results.

Some essays explored ways of combining intersectionality theory with CRE and some highlighted the need to develop intersectionality methodologies further. This approach acknowledges that groups’ vulnerabilities to shocks such as floods are not equal. Thus the intersectionality of people’s identities and the influence of these on people’s experiences of these shocks should be considered. Combining CRE with intersectionality can encourage evaluators to be more intentional about generating quality data.

Three essays discussed data science methods, specifically AI-driven methods, that could enhance the integration of CRE as they can assist with the systematic analysis of data sets and augment the traditional scope. However, AI methodologies require further development to improve accuracy and minimize risks and biases. The discussion of how AI can enhance evaluation experiences served as a stern reminder that one needs to keep up with the pace of change diligently, intentionally, and urgently.

Insights from the four themes of the competition

Case studies on the integration of Culturally Responsive Evaluation into evaluation practice

A few essays identified case studies in which CRE approaches were utilised to produce effective evaluations and promote the uptake of evidence. In some examples, conflicting results emerged at the onset, but CRE integration helped achieve the evaluation goals. A case study offered guidance for conducting culturally responsive data analysis for evaluations in contexts of fragility, conflict, and violence, showcasing how CRE is effective in successfully and objectively designing, implementing, and assessing interventions and improving their impact.

Some case studies focused on using mixed and participatory approaches for CRE integration. For example, they argued that linguistic knowledge should help identify biases to bring an evaluation closer to the local reality. The essays proposed that tools and concepts should be translated into local languages to add value.

Some case studies noted the importance of using sampling frames that ensure diversity and inclusion of multiple perspectives to increase validity, for example, by considering a range of dimensions when making sampling decisions such as focus group composition. They also mentioned culturally relevant indicators that can help recognize the impact of programs, particularly when they are designed using participatory approaches.

A case study for CRE integration highlighted institutions that are fostering the use of CRE, including the African Evaluation Association, among many others. Such institutions and groups provide platforms for sharing information on approaches and methodologies for integrating culture into evaluations.

Challenges for integration of culturally responsive evaluation into practice

The essays mentioned several challenges to integrating CRE into practice as follows:

• Some stakeholders may withhold sensitive information, for example financial statements, resulting in incorrect impact documentation.
• Although country-led evaluations can facilitate CRE integration, local evaluators can be limited in championing its practice.
• The absence of adequate funding poses a risk of compromising methodologies.
• Development initiatives that are conceptualized, designed, and developed outside their context of implementation, including the formulation of theories of change create a disconnect among development initiatives and social realities.
• Published guidelines and tool kits on CRE practice are not always available and there is a lack of proper coordination of relevant interventions.

Communicating the validity and importance of culturally responsive evaluation to decision makers

The winning essay of this competition showed that, because CRE is often perceived as being of niche relevance, it faces resistance and scepticism. Authors argued that a wide range of stakeholders should be engaged to build a global community of practice to address complexities of cultural responsiveness. As CRE lacks consensus on definitions, working with such a community to clarify definitions may be helpful in eliminating misconceptions and building support.

It’s important to demonstrate the value of collecting data that is culturally responsive and show that evaluations developing evaluator’s capacity increase the chances of evidence uptake and foster collaboration. Possible innovations include providing incentives for publicizing evaluations integrating CRE, embedding CRE in curriculum development, developing tool kits, and intentionally developing terms of reference promoting the adoption of CRE approaches in evaluations.

The role of emerging evaluators in the future of culturally responsive evaluation

Emerging evaluators are often well placed to act as bridge builders to connect global evaluation standards with local intricacies, particularly in cases where they have established connections and communication with local communities and are familiar with both the culture and the language. This can be instrumental in providing contextual information and operationalizing CRE. In practice, this means adapting methodologies that are involved in evaluation planning, designing, and implementation, and advocating for representation in evaluation teams.

However, acquiring cultural competency is a process, and using it in evaluations requires wisdom and diligence. Whether emerging or experienced, evaluators located in a community have the potential to make or break an evaluation, and cultural competence, which goes beyond simply belonging to or having knowledge about a community, is critical to ensuring the success of this role. Tailormade interventions such as scholarships, internships, and mentorships could allow young and emerging evaluators to acquire foundational knowledge and practical skills earlier in their careers and empower this cohort to operationalize CRE effectively.

IEG, GEI, and EvalYouth are thankful to everyone who participated in the essay competition and made this initiative possible.

If you’d like to stay up to date with GEI and suggest content on culturally responsive evaluation you can join its BetterEvaluation platform here.

For more information on IEG@50 events that you can participate in, visit the IEG@50 page.