Determine What 'Success' Looks Like

Evaluation is essentially about values, asking questions such as : What is good, better, best?  Have things improved or got worse? How can they be improved? Therefore, it is important for evaluations to be systematic and transparent in the values that are used to decide criteria and standards.

Criteria

Criteria refer to the aspects of an intervention that are important to consider when deciding whether or not, and in what ways, it has been a success or a failure, or when producing an overall judgement of performance. There are different types of criteria:

Positive outcomes and impacts: for example, should childcare be judged in terms of its success in supporting early childhood development or in supporting parents to engage in education or work? If it is both, how should they be weighted?

Negative outcomes and impacts: for example, an infrastructure development might produce negative unintended effects (e.g. soil erosion caused by a new road) as well as positive intended effects)

Distribution of costs and benefits: for example, is it important for everyone to receive some benefit or the same benefit or for the intervention to be targeted so that the most disadvantaged receive more benefit?

Resources and timing: for example, is there a need for results to be achieved within a certain timeframe?

Processes: for example, use of recyclable materials; providing access to groups with restricted mobility

Standards

Standards refer to the levels of performance required for each of the criteria. For example, if a project aims to reduce maternal mortality, what level of performance is needed for it to be considered successful? Any reduction?  A reduction of at least xx%?  A reduction of at least xx in absolute terms? A reduction to a rate of x.x that matches other similar regions, or matches official targets?

Criteria and standards need to be agreed on in order to identify the data that need to be gathered for an evaluation.

In addition, these data need to be combined to form an  overall judgement of success or failure, or to rank alternatives against each other.  For example, if a road project achieves its economic objectives but produces environmental damage, should it be considered a success overall?  How much damage, and at whose cost, would be enough to outweigh the positive impacts?  These issues are addressed under the task Synthesise data from a single evaluation

Options

Some options are used to identify possible criteria and standards that could be used in an evaluation, drawing on formal and informal sources, and some options are used to negotiate which should be used and how they should be weighed.

Formal statements of values

  • OECD-DAC criteria: setting out high level evaluation criteria for evaluations which must be operationalized for each evaluation (OECD's Development Assistance Committee).
  • Millenium Development Goals (MDGs):  a set of time bound and quantified goals and targets developed to help track progress in eradicating poverty
  • Standards, evaluative criteria and benchmarks: developing explicit standards, evaluative criteria or benchmarks or using existing relevant standards, criteria or benchmarks to define values.
  • Stated goals and objectives (including legislative review and policy statements):  stating the program's  objectives and goals so they can be used to assess program success. 

Articulate and document tacit values

  • Hierarchical Card Sorting (HCS): a participatory card sorting option designed to provide insight into how people categorize and rank different phenomena.
  • Open space technology: facilitating a group of 5 - 500 people in which a central purpose, issue, or task is addressed without a formal initial agenda.
  • Photovoice: using cameras to allow participants (often intended beneficiaries) to take and share photos in order to describe how they relate to important issues for them.
  • Rich Pictures: exploring, acknowledging and defining a situation through diagrams in order to create a preliminary mental model.
  • Stories of change: showing what is valued through the use of specific narratives of events.
  • Values Clarification Interviews: interviewing key informants and intended beneficiaries to identify what they value.
  • Values clarification public opinion questionnaires: seeking feedback from feedback from large numbers of people about their priorities through the use of questionnaires.

Negotiate between different values

  • Concept Mapping: negotiating values in order to frame the evaluation.
  • Delphi Study: generating a consensus without face to face contact by soliciting opinions from individuals in an iterative process of answering questions.
  • Dotmocracy: recording participants opinions by using sticky dots to either record agreement or disagreement with written statements.
  • Open Space Technology: facilitating a group of 5 - 500 people in which a central purpose, issue, or task is addressed without a formal initial agenda.
  • Public Consultations: conducting public meetings to provide an opportunity for the community to raise issues of concern and respond to options.

Approaches

  • Critical System Heuristics: an approach used to surface, elaborate, and critically consider boundary judgments, that is, the ways in which people/groups decide what is relevant to the system of interest (any situation of concern). 
  • Participatory evaluation: involving key stakeholders in the evaluation process.

 

 

Comments

James Maraga's picture
James Nyanchoka Maraga

This site is great! Very useful evaluation resources. Many thanks.

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.