How do we use advisory groups effectively in evaluation?

This guest blog by Marlène Läubli Loud aims to start a discussion about what advisory group practices work well in what situations.

Marlène looks back on her experiences and outlines some of the conditions that she believes have contributed to securing the “best value” from advisory groups, and asks for other ideas and examples for engaging and utilising advisory groups to their full advantage. 

In my (long) experience in evaluation, I’ve been involved in a number of situations where advisory groups have worked very well, and a number where they haven’t. While these days engaging an advisory group is becoming usual practice, there’s still work to be done to find ways to get the most out of advisory group arrangements.

This blog aims to start a discussion about what advisory group practices work well in what situations. I look back on my experiences and outline some of the conditions that I believe have contributed to securing the “best value” from advisory groups. I’m hoping you will join in and share your own thoughts, experiences and opinions so that we can share ways of engaging and utilising advisory groups to their full advantage. 

What is an advisory group for evaluation?

An advisory group provides advice to those who are making decisions about an evaluation, an evaluation system or the evaluation function - which might be an evaluation steering committee, or an identified manager or management group.  (for more detail, please see the updated page on Advisory Groups which I have contributed revisions to).

Advisory groups are set up to engage the multiple perspectives, values and subject-specific knowledge of key stakeholders and primary intended users. This expert advice can include technical “know how” or contextual interpretation, perspectives, and thoughts.  It can be drawn on throughout the life of the evaluation, from design, through data collection and analysis, to reporting and use.

This can develop a sense of ownership of an evaluation amongst key stakeholders, and also give the evaluation team access to a valuable source of knowledge and perspectives that they wouldn’t otherwise have had.

What constitutes good practice from my perspective

• Include representation of the key, external stakeholders.

• Allow the advisory group to play an equal role with commissioners in advising on the evaluation design and process as well as the final results.

• Ensure that the advisory group’s purpose, organisation and tasks are transparent and acceptable to all members from the onset (with perhaps reminders throughout to be sure!)

• Establish a collaborative working atmosphere.

For instance, wherever advisory groups have been set up by commissioners who are not fully convinced of working in partnership with the stakeholder, the purpose, modus operandi and tasks of the advisory group are likely to be unclear. This can lead to misunderstandings, mistrust, tensions and unresolvable conflicts. At worst, the advisory group is just a means of ticking the box rather than being used as a valuable resource!

• Acknowledge the extra workload and time input involved

An advisory group inevitably means an additional workload for the evaluators and those managing the evaluation. So it is vital to check that the needed support is secured both by the relevant commissioner and advisory group organisations (for the evaluation manager) and within the evaluation budget (for the evaluator).

• Present and respect stakeholders’ multiple values perspectives equally – make sure no one or two stakeholders are allowed to sabotage the event.

• Ensure all advisory group members work together in agreeing on a communication strategy for dealing with, and acting on, the evaluation results.

• Develop an agreed process for resolving conflicting views and recommendations.

For example, unresolved disagreements can be either written up as a preface to the final evaluation report or into the Executive Response.

Under such conditions, with the inclusion of stakeholders, the evaluation enjoys invaluable expertise and input. The validity/credibility of the evaluation and its use is better assured and - equally important - the status of evaluation is likely enhanced.  One more thing! Recognised or not, with real engagement and commitment, the whole process is in itself a learning process too! For example, determining the theory of change together is a real learning event.

Particular challenges

Should intended beneficiaries be included in the advisory group?

Beneficiary representation would be a plus where this is feasible and would make sense, but this can be difficult to do in practice. In my experience, I have only seen this happen on two occasions.  Who you include in terms of beneficiary representation, and how you do this, can be a challenge. For example, to include the public, consumer associations could be used as representatives, but there are often more than one.

Moreover, for a country like Switzerland, for instance, there are four official languages, each with its own consumer association. One alone would not be representative. Issues like these need to be carefully thought through before putting together the advisory group.

Should external evaluation experts be included in an advisory group or a separate technical advisory group?

 I have found that mixing evaluation methodology expertise together with stakeholders in an advisory group doesn’t work! Discussing the “nitty gritty” of evaluation theory and practice is best kept separate. But, of the few cases I know of where a separate methodology advisory group was set up to provide peer support/advice to the evaluator(s), only in one case would I say it was a success. Understandably, once an evaluator is hired, mutual trust needs to be secured between the evaluator and the commissioner and its internal evaluation manager. Additional methodological “outsiders” are often not appreciated!

However, in the one exception that I mentioned, each of the key stakeholder organisations assigned a manager from their own internal evaluation unit to meet together with the evaluation team. From the onset, there was a collaborative spirit established, with everyone pulling together to find solutions to the somewhat difficult challenges of dealing with an intervention with widespread, global coverage. And it worked! (Probably the legitimacy of the group helped considerably).

Your experiences

So how much of the above rings true for you? What are your ideas, thoughts, experiences on these or other issues relative to the use of Advisory Groups? What's worked well for you in the past? Do you have comments on the good practices I have suggested – or answers to the particular challenges? 

Marlène Läubli Loud is an evaluation consultant whose co-edited book (with John Mayne) Enhancing Evaluation Use: Insights from Internal Evaluation Units includes discussion of evaluation advisory groups. Marlène has also  contributed to BetterEvaluation by revising the option page on Advisory Groups.