An advisory group can be established to provide advice on an individual evaluation, a series of evaluations, or the evaluation function within an organization. Decisions (such as approving a terms of reference, selection of an external evaluator, or acceptance of a final report) are made by a different group, which might be a formal steering committee or an identified manager or managers.
An advisory group is typically made up of one or more of three potential groups: stakeholders (especially intended users), evaluation experts, and intended beneficiaries. It can consist entirely of external members (external to the organization or to the particular unit where the evaluation is being undertaken), or it can include internal and external members.
Having an advisory group accompany the evaluation process means that stakeholders are encouraged to bring their perspectives and concerns to the table in a collaborative spirit. Differences can be articulated and discussed so that a fuller understanding of the issues can be made clearer to the evaluation.
Advisory group participation and engagement in the evaluation process can lead to better quality, utility and use of the evaluative work. This is particularly important in cases where reaching consensus and support is highly-valued, such as experimental policy or when suggesting important but controversial changes to existing policies,
An advisory group is often chosen by the evaluation’s steering group, who typically invite participation from key stakeholder groups, such as intended users and beneficiaries. Evaluators with specific knowledge of the evaluation’s design methodology (e.g. economic analysis) are sometimes also included in the advisory group, either in the same advisory group as the key stakeholders or as a separate advisory group to advise on methodological aspects.
Advisory groups are not always needed for every evaluation. If an evaluation design is already very participatory-oriented and will engage with various stakeholder groups and even beneficiaries on occasion, then the creation of an additional advisory group may not create enough added value to justify the effort.
The question of resources is also important when deciding whether or not to set up an advisory group. The benefits drawn need to sufficiently outweigh the investment in time and effort required to establish and manage an advisory group. There are extra costs to be taken into account, but also there is the question of time. The work involved in preparing, organising and participating in meetings means that more time will need to be built into the evaluation timeframe. This is not always feasible where results are needed within a very limited timescale.
The culture of the commissioning body or organisation will also have a bearing on the success of appointing an advisory group. If there is a mature evaluation in the organisation, there is likely to also be an appreciation of the value that an advisory group can bring. If there is a culture of democratic deliberation, collaboration, transparency, and a will to embrace pluralism and stakeholder involvement, it is more likely there will be effective processes for appointing, and adequately engaging with, an advisory group. However, if an evaluation is tightly controlled either by the commissioning bureaucrats or by the external evaluator, it is less likely that an advisory committee will be effectively used. (See Democratic Evaluation for a discussion about democratic, bureaucratic and autocratic evaluation).
The scale and scope of advisory group participation in the different phases of an evaluation will depend on many factors, which include members’ available time as well as the steering committee’s wants and expectations.
Questions worth carefully considering when setting up an advisory group include:
- What is the main purpose for setting up this group?
- What are the likely benefits? How important are these?
- How important is it to take account of multiple perspectives and values?
- How important is collaboration and transparency to the commissioners?
Depending on the answers to these questions, the degree of advisory group participation may be limited to only being called on for advice at key stages of the evaluation (e.g., commenting on the Terms of Reference and initial design, the interpretation of the intermediate findings, providing feedback on the draft report, especially the conclusions and recommendations, and discussing and planning for how best to use the results).
When there is a higher commitment to collaboration and the involvement of different perspectives, more frequent and deeper engagement may be called for. When this is the case, advisory groups may be invited to contribute to or comment on methodological issues such as the logical framework, the theory of change, and the comprehensiveness and validity/credibility of the data collected.
Advisory groups can also in certain situations be asked to provide data for secondary analysis or invited to make a joint statement with the commissioners on what has been learned and what will happen next.
Whatever the scale of involvement and participation adopted, it’s important to set and agree upon clear and realistic expectations about this.
- Terms of Reference: This example from the Victorian Department of Health in Australia provides a Terms of Reference for an Evaluation Advisory Group and clearly demonstrates the functions and role of an Advisory Group.
- Developing and Using an Evaluation Consultation Group: This guide from the Centers for Disease Control and Prevention (CDC) provides a detailed outline of the need for and the process of developing an Evaluation Consulting Group (ECG) or Advisory Group.
- Evaluation Advisory Groups: This journal from the American Evaluation Association (AEA)provides a detailed overview of Evaluation Advisory Groups with chapters devoted to the roles of groups in a variety of evaluation scenarios.
- Enhancing Evaluation Use: Insights from Internal Evaluation Units: This book, co-edited by Marlène Läubli Loud and John Mayne, offers invaluable insights from real evaluators who share strategies they have adopted through their own experiences in evaluation.