Specify the Key Evaluation Questions

Having an agreed set of Key Evaluation Questions (KEQs) makes it easier to decide what data to collect, how to analyze it, and how to report it.

Sometimes the KEQs are already prescribed by an evaluation system or a previously developed evaluation framework.  Usually they need to be developed and agreed on at the beginning of evaluation planning.

Key Evaluation Questions are derived from the purpose(s) of the evaluation.

Try not to have too many Key Evaluation Questions - a maximum of 5-7 main questions will be sufficient. It might also be useful to have some more specific questions under the KEQs.

The following information has been taken from the New South Wales Government, Department of Premier and Cabinet Evaluation Toolkit, which BetterEvaluation helped to develop.

A program evaluation should focus on only a small set of key questions.  These are not questions that are asked in an interview or questionnaire but high level research questions that will be answered by combining data from several sources. 

Here are some typical key evaluation questions for the 3 main types of evaluation:

Key evaluation questions for the main types of evaluation 

Type Typical key evaluation questions
Process evaluation How is the program being implemented? How appropriate are the processes compared with quality standards? Is the program being implemented correctly? Are participants being reached as intended? How satisfied are program clients? For which clients? What has been done in an innovative way?
Outcome evaluation (or impact evaluation)  How well did the program work? Did the program produce the intended outcomes in the short, medium and long term? For whom, in what ways and in what circumstances? What unintended outcomes (positive and negative) were produced?

To what extent can changes be attributed to the program? What were the particular features of the program and context that made a difference? What was the influence of other factors?

Economic evaluation (cost-effectiveness analysis and cost-benefit analysis) What is the most cost-effective option? Has the intervention been cost-effective (compared to alternatives)? Is the program the best use of resources?

What has been the ratio of costs to benefits?

Appropriateness, effectiveness and efficiency

Three broad categories of key evaluation questions to assess whether the program is appropriate, effective and efficient are often used.

Organising key evaluation questions under these categories, allows an assessment of the degree to which a particular program in particular circumstances is appropriate, effective and efficient. Suitable questions under these categories will vary with the different types of evaluation (process, outcome or economic). 

  Typical key evaluation questions
Appropriateness  To what extent does the program address an identified need?How well does the program align with government and agency priorities?

Does the program represent a legitimate role for government?

 Effectiveness To what extent is the program achieving the intended outcomes, in the short, medium and long term?To what extent is the program producing worthwhile results (outputs, outcomes) and/or meeting each of its objectives.
Efficiency Do the outcomes of the program represent value for money?To what extent is the relationship between inputs and outputs timely, cost-effective and to expected standards?

While you can use different processes to develop evaluation questions, these should emerge as you consider the different activities associated with this step (the purpose of the evaluation, the type of evaluation, stakeholder interests, and preliminary assessments).  There may be formal and at times general evaluation questions that need to be addressed where evaluations are mandated in legislation or arrangements such as National Partnership Agreements.

To clarify the purpose and objectives of an evaluation, there should be a limited number of higher order evaluation questions (roughly 3 to 5 questions) with sub-questions underneath each higher-order question. The higher-order questions can be grouped under the categories of appropriateness, effectiveness and efficiency.

A way to test the validity and scope of evaluation questions is to ask: when the evaluation has answered these questions, have we met the full purpose of the evaluation?







Cite this page

'Develop the Evaluation brief' (2016) New South Wales Government Department of Premier & Cabinet Evaluation Toolkit. © State of New South Wales Department of Premier and Cabinet 2015. Retrieved from: http://www.dpc.nsw.gov.au/programs_and_services/policy_makers_toolkit/st...


There are currently no comments. Be the first to comment on this page!

Add new comment

Login Login and comment as BetterEvaluation member or simply fill out the fields below.