Evaluation design

An evaluation design describes how data will be collected and analysed to answer the Key Evaluation Questions. 

There are different pathways for you as manager depending on who will develop the evaluation design. In most cases your evaluator will develop the evaluation design.  In some cases you will – if you have evaluation expertise and/or the evaluation design has already been developed (for example, in an evaluation that is intended to match an earlier evaluation).

Take into account the following important factors when developing an evaluation design

1. The nature of what is being evaluated
   In particular whether there are complicated or complex aspects that need to be addressed in the evaluation, and other particular challenges such as delays before impacts are evident or barriers to collecting accurate data.
   Manager's guide to evaluation : 

   Consider important elements of what is being evaluated

    

2. The nature of the evaluation
   In particular the types of Key Evaluation Questions that are being asked, when the answers are needed. 
   Manager's guide to evaluation : 

   Consider important aspects of the evaluation

    

3. Available resources and constraints
   Resources including money, existing data, expertise, technical equipment. Constraints including requirements to use certain common indicators, limits to availability of key informants or barriers to accessing existing data. 
   Manager's guide to evaluation : 

   Consider the implications of the resources available and specific constraints

If an EVALUATOR will develop the evaluation design

• Engage a competent evaluation expert - internal, external or a combination. (See 'Select an evaluator / evaluation team' for advice).
• Work with the expert(s) to ensure they understand important factors that should be taken into account in the evaluation design (see section above)
• The design should provide details of how data will be collected analysed. It is often useful to do this in the form of an Evaluation Matrix which shows how each Key Evaluation Question will be answered.

If  YOU (as manager) will develop the evaluation design

• Understand important factors that should be taken into account in the evaluation design (see section above)
• Develop an evaluation design that addresses these important factors.
• Summarise the design in the form of an Evaluation Matrix which shows how each Key Evaluation Question will be answered.

Subsequently, arrange for a technical review of the evaluation design and arrange for a review of the design by the evaluation management structure (e.g., steering committee). Ideally this will include representation from primary intended users.

Arranging technical review of the evaluation design

Before finalizing the design, it can be helpful to have a technical review of it by one or more independent evaluators.  It might be necessary to involve more than one reviewer in order to provide expert advice on the specific methods proposed, including specific indicators and measures to be used.  Ensure that the reviewer is experienced in using a range of methods and designs, and well briefed on the context, to ensure they can provide situation specific advice.

Arranging review of the design by the evaluation management structure

In addition to being considered technically sound by experts, it is essential for the evaluation design to be seen as credible by those who are expected to use it.

Get formal organisational review and endorsement of the design by an evaluation steering committee (see 'Identify who will be involved in decisions and what their roles will be' for possible structures, processes and terms of reference for a steering committee)

Where possible do data rehearsal of possible findings with primary intended. This is a powerful strategy for checking the appropriateness of the design by presenting mock-ups of tables, graphs and quotes that the design might produce. It is best to produce at least 2 different versions – one that would show the program working well and one that would show it not working.  Ideally the primary intended users of the evaluation will review these and either confirm the suitability of the design or request amendments to make the potential findings more relevant and credible. (For more information see Patton, MQ (2011) Essentials of Utilization-Focused Evaluation. pp. 309-321).