Determine what constitutes high quality evaluation

Methods

Criteria relating to products

• Accessibility

  Accessibility of evaluation products includes consideration of the format and access options for reports, including plain language, inclusive print design, material in multiple languages, and material in alternative formats (such as online, audio, or braille).

• Accuracy

  Accuracy refers to the correctness of the evidence and conclusions in an evaluation. It may have an implication of precision.

• Credibility

  Credibility refers to the trustworthiness of the evaluation findings, achieved through high-quality evaluation processes, especially rigour, integrity, competence, inclusion of diverse perspectives, and stakeholder engagement.

• Transferability

  Transferability involves presenting findings in a way that they can be applied in other contexts or settings, considering the local culture and context to enhance the utility and reach of evaluation insights.

Criteria relating to processes

• Bias reduction

  Bias reduction involves identifying possible sources of bias and taking steps to reduce it. This is one way of improving the validity of an evaluation.

  Types of bias include,

• Common good and equity

  Consideration of common good and equity involves an evaluation going beyond using only the values of evaluation stakeholders to develop an evaluative framework to also consider common good and equity more broadly.

• Competence

  Competence refers to ensuring that the evaluation team has or can draw on the skills, knowledge and experience needed to undertake the evaluation.

  Competence is one of the guiding principles of the American Evaluation Association:

• Cultural competency

  Cultural competency involves ensuring that evaluators have the skills, knowledge, and experience necessary to work respectfully and safely in cultural contexts different from their own.

• Ethical practice

  Ethical practice in evaluation can be understood in terms of designing and conducting an evaluation to minimise any potential for harm and to maximise the value of the evaluation.

• Evaluation accountability

  Evaluation accountability relates to processes in place to ensure the evaluation is carried out transparently and to a high-quality standard.

• Feasibility

  Feasibility refers to ensuring that an evaluation can be realistically and effectively implemented, considering factors such as practicality, resource use, and responsiveness to the programme's context, including factors such as culture and politics.

• Human rights and gender equality

  Human rights and gender equality refer to the extent to which an evaluation adequately addresses human rights and gender in its design, conduct, and reporting.

• Impartiality

  Impartiality in evaluation refers to conducting an evaluation without bias or favouritism, treating all aspects and stakeholders fairly.

  Key aspects of impartiality in evaluation can include:

• Inclusion of diverse perspectives

  Inclusion of diverse perspectives requires attention to ensure that marginalised people and communities are adequately engaged in the evaluation.

• Independence

  Independence can include organisational independence, where an evaluator or evaluation team can independently set a work plan and finalise reports without undue interference, and behavioural independence, where evaluators can conduct and report evaluati

• Integrity

  Integrity refers to ensuring honesty, transparency, and adherence to ethical behaviour by all those involved in the evaluation process.

  Integrity is one of the guiding principles of the American Evaluation Association:

• Professionalism

  Professionalism within evaluation is largely understood in terms of high levels of competence and ethical practice.

• Propriety

  Propriety refers to ensuring that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in it and those affected by its results.

• Respect for people

  Respect for people during an evaluation requires those engaged in an evaluation to respect the security, dignity, and self-worth of respondents, program participants, clients, and other evaluation stakeholders.

• Rigour

  Rigour involves using systematic, transparent processes to produce valid findings and conclusions. There are significant differences in what this is understood to mean in evaluation.

• Strengthening national evaluation capacities

  Strengthening national evaluation capacities refers to the ways in which an evaluation can have broader value beyond a single evaluation report by increasing national capacities.

• Systematic inquiry

  Systematic inquiry involves thorough, methodical, contextually relevant and empirical inquiry into evaluation questions.

  Systematic inquiry is one of the guiding principles of the American Evaluation Association:

  Systematic Inquiry

• Transparency

  Transparency refers to the evaluation processes and conclusions being able to be scrutinised.

• Utility

  Utility standards are intended to increase the extent to which program stakeholders find evaluation processes and products valuable in meeting their needs.

• Validity

  Validity refers to the extent to which evaluation findings are correct.

  In the Evaluation Standards produced by the Joint Committee on Educational Evaluation, standards relating to this were labelled as Accuracy Standards:

Resource

• What counts as good evidence?

  This paper, written by Sandra Nutley, Alison Powell and Huw Davies for the Alliance for Useful Evidence, discusses the risks of using a hierarchy of evidence and suggests an alternative in which more complex matrix approaches for identifying evidence qu