Having an agreed set of Key Evaluation Questions (KEQs) makes it easier to decide what data to collect, how to analyze it, and how to report it.
Sometimes the KEQs are already prescribed by an evaluation system or a previously developed evaluation framework. Usually they need to be developed and agreed on at the beginning of evaluation planning.
Key Evaluation Questions are derived from the purpose(s) of the evaluation.
Try not to have too many Key Evaluation Questions - a maximum of 5-7 main questions will be sufficient. It might also be useful to have some more specific questions under the KEQs.
The following information has been taken from the New South Wales Government, Department of Premier and Cabinet Evaluation Toolkit, which BetterEvaluation helped to develop.
A program evaluation should focus on only a small set of key questions. These are not questions that are asked in an interview or questionnaire but high level research questions that will be answered by combining data from several sources.
Here are some typical key evaluation questions for the 3 main types of evaluation:
Key evaluation questions for the main types of evaluation
|Type||Typical key evaluation questions|
|Process evaluation||How is the program being implemented? How appropriate are the processes compared with quality standards? Is the program being implemented correctly? Are participants being reached as intended? How satisfied are program clients? For which clients? What has been done in an innovative way?|
|Outcome evaluation (or impact evaluation)||How well did the program work? Did the program produce the intended outcomes in the short, medium and long term? For whom, in what ways and in what circumstances? What unintended outcomes (positive and negative) were produced?
To what extent can changes be attributed to the program? What were the particular features of the program and context that made a difference? What was the influence of other factors?
|Economic evaluation (cost-effectiveness analysis and cost-benefit analysis)||What is the most cost-effective option? Has the intervention been cost-effective (compared to alternatives)? Is the program the best use of resources?
What has been the ratio of costs to benefits?
Appropriateness, effectiveness and efficiency
Three broad categories of key evaluation questions to assess whether the program is appropriate, effective and efficient are often used.
Organising key evaluation questions under these categories, allows an assessment of the degree to which a particular program in particular circumstances is appropriate, effective and efficient. Suitable questions under these categories will vary with the different types of evaluation (process, outcome or economic).
|Typical key evaluation questions|
|Appropriateness||To what extent does the program address an identified need?How well does the program align with government and agency priorities?
Does the program represent a legitimate role for government?
|Effectiveness||To what extent is the program achieving the intended outcomes, in the short, medium and long term?To what extent is the program producing worthwhile results (outputs, outcomes) and/or meeting each of its objectives.|
|Efficiency||Do the outcomes of the program represent value for money?To what extent is the relationship between inputs and outputs timely, cost-effective and to expected standards?|
While you can use different processes to develop evaluation questions, these should emerge as you consider the different activities associated with this step (the purpose of the evaluation, the type of evaluation, stakeholder interests, and preliminary assessments). There may be formal and at times general evaluation questions that need to be addressed where evaluations are mandated in legislation or arrangements such as National Partnership Agreements.
To clarify the purpose and objectives of an evaluation, there should be a limited number of higher order evaluation questions (roughly 3 to 5 questions) with sub-questions underneath each higher-order question. The higher-order questions can be grouped under the categories of appropriateness, effectiveness and efficiency.
A way to test the validity and scope of evaluation questions is to ask: when the evaluation has answered these questions, have we met the full purpose of the evaluation?
- A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions: provides a detailed, step by step guide to engaging stakeholders in the development of evaluation questions. (Robert Wood Johnson Foundation)
- Looking Back, Moving Forward: SIDA Evaluation Manual: provides a step-by-step guide to formulating evaluation questions. (p 70 - 72, SIDA)
- Evaluation questions: provides a comprehensive guide to the use of evaluation questions. (EuropeAid)
- Stakeholders’ Interest in Potential Evaluation Questions: provides a template for developing evaluation questions which engage stakeholders' interest in the process. (National Science Foundation).
- Prioritize and Eliminate Questions: provides a template which allows the organisation and selection of possible evaluation questions. (National Science Foundation).
- Evaluation at country level: provides a range of example evaluation questions for country level evaluations. (EuropeAid)
- Evaluation at regional level: provides a range of example evaluation questions for regional level evaluations. (EuropeAid)
- Sector or thematic global evaluation: provides a range of example evaluation questions for sector or thematic global evaluations. (EuropeAid)
Cite this page
'Develop the Evaluation brief' (2016) New South Wales Government Department of Premier & Cabinet Evaluation Toolkit. © State of New South Wales Department of Premier and Cabinet 2015. Retrieved from: http://www.dpc.nsw.gov.au/programs_and_services/policy_makers_toolkit/st...