Week 13: Producing engaging and accessible evaluation reports

This week we start the first in an ongoing series of Real-Time Evaluation Queries, where BetterEvaluation members ask for advice and assistance with something they are working on, together we suggest some strategies and useful resources - and then we find out what was actually useful (or not) and why.  

Recently BetterEvaluation member Rituu B. Nanda  asked us for advice on producing interesting evaluation reports:

This is a very common challenge for evaluators and evaluation commissioners, who often don't have resources available to engage specialist graphic designers and editors to make evaluation reports more interesting and engaging.  

Fortunately, there are lots of great resources available to address this issue.  Here are some possible strategies:

1. Ask the primary intended users how to present the findings

This suggestion might not be suitable in this instance - but could be a useful strategy in the future.

Ideally, this issue should be raised early in the evaluation, and mocked up reports shared to help people have an informed discussion about the sort of reports that would be useful.  

Michael Quinn Patton calls this "data rehearsal" where  the primary intended users are presented with hypothetical data answering the planned evaluation questions - early enough in the process so that the evaluation plan can be changed if need be to gather different or additional data.  This process can also be used to develop some shared expectations about the format of evaluation reporting. 

Jane Davidson suggests negotiating agreement on an evaluation report 'skeleton' which sets out the overall structure and an estimated page length, before you start writing the report. 

2. Be clear about the main messages you want to communicate

​Stephanie Evergreen refers to this as the "six word presentation story", building on the idea of the "Six Word Obituary" 

Check out the free 45 minute webinar on "Messaging" developed as part of the American Evaluation Association's Potent Presentations Initiative .  While it refers to presentations, this webinar, which focuses on identifying your key messages, is also useful for preparing reports.

3. Use headings that summarise key messages

​Use report headings and chart and table titles to provide a summary of these key messages.

For example, the Office for Development Effectiveness report "From Seed to Scale-Up: Lessons from Australia's rural development assistance" identified a number of lessons in the executive summary (for example "Start with a considered understanding of how the poor will benefit from the intervention​") and then used these as headings to discuss the findings.

Jane Davidson provides suggestions for structuring the evaluation report around key evaluation questions in her demonstration session at the American Evaluation Association conference, which can be accessed through the AEA free elibrary. 

4. Prepare good visualisations

There is a lot that can be done to present visualisations of numeric, visual and textual data.

For textual data, you can create a visual "map" of key ideas and add boxed quotes which illustrate summary findings. (Yes, Wordle can produce visualisations of words, but, as Humprhey Costello so eloquently explains, unless data have been collected with this type of analysis in mind wordclouds can be meaningless).

If you have numeric data (and this can include qualitative data that has been grouped into categories and then counted), check out the guides from Ann Emery and Stephanie Evergreen which show how you can make Excel produce lovely looking, and very effective, graphs if you change the default settings.

• Showcasing key findings through excel charts: This webpage by Ann K Emery provides a number of short video tutorials for creating Excel charts that can be used for highlighting evaluation findings in reports.  It includes simple lessons on inserting charts and changing chart type through to more detailed ones for changing chart axes.
• Evaluation Executive Summaries & Reports: This blog by Stephanie Evergreen provides six examples of executive summaries and evaluation reports that incorporate the use graphics to present findings and engage the reader. She also provides analyses of each of the examples to highlight the effective and innovative ways that graphics can be used in reports.
• Using Visual Communication to Increase Evaluation Utilization: This blog post from Nate Wilairat for Evergreen Data provides a range of tips for increasing evaluation utilisation through the use of visual communication.  Based on the lessons learned from producing effective executive summaries the post particularly highlights the use of annotated dashboards and matrices. These examples draw on the assistance of a graphic designer.
• For an overview of data visualisation tools, check out the interactive periodic table - for example, the knowledge map to communicate key ideas and how they are related.

5. Provide vivid descriptions of some key incidents or examples

As well as providing summaries from your qualitative data, create some vignettes or examples that provide a rich vicarious experience for the reader about a particular person, site or incident.  Try to write this as  a story with a beginning , a middle and an end and be clear about how this is intended to explain, illustrate or elaborate the overall findings.

For some examples of using vignettes (short account) in reports see the description of using vignettes for reporting by Roberts Evaluation.

6. Prepare an engaging presentation to complement a written report

Don't rely on the report to do all the work. Ensure you create some event where you can present the findings, or some particular aspect of them. You can use different types of events:

• Conference: discussing a set topic or theme in a large group of people at a set venue.
• Feedback workshops:
• Teleconference: facilitating discussion of evaluation findings via telephone.
• Verbal briefings: providing specific information to an audience of interested participants allowing for a structured question and answer format based on that information. 
• Videoconference: gathering data, communicating information about an evaluation, reporting findings, receiving feedback, and planning for utilization.
• Web-conference: bringing people together from around the world using the internet.
• Summit Workshop - you can plan an event to present draft findings and engage stakeholders in checking and extending the data interpretation 

You can use different types of materials in these presentations:

• Displays and exhibits: drawing attention to particular issues and assisting in community engagement. 
• Flip Charts: providing a useful way of interacting with your audience and therefore allowing you to present your own ideas and results and also to immediately record input, feedback and ideas from your audience.
• Posters: presenting your evaluation findings in the form of a poster provides a good opportunity to get your message across in a clear way while also providing opportunities for feedback.
• PowerPoint: organizing and communicate information coming from evaluations in the form of a slide show which can be used at a meeting or conference.
• Video: highly flexible and immediate medium which allows you to make an emotional meaningful connection with the audience.

Some additional useful resources:

• Before & After Slides: Stay on the Side of Simplicity: This blog post from Stephanie Evergreen highlights the steps that were taken to revise the some presentation slides showing data. Stephanie describes the process she used to create a slide that served as a simple visual support for the speaker while still clearly representing the data of the original slide.
• Check out the Potent Presentation guides on Design and Delivery of effective presentations 

7. Prepare alternative or complementary reporting products

Don't just stick with a final report.  Think about some creative options:

• Cartoons: allowing readers to see a point differently, add humour, and break up large sections of prose.
• Photographic reporting: making your report more appealing to readers and also making the key messages more memorable by including photographs.
• Poetry: communicating the experience of participants can be achieved by presenting some of the findings in the form of a poem.
• Reporting in pictures: presenting information in an alternative way and therefore increasing understanding of your results.
• Theatre: communicating evaluation findings and engaging intended users in responding to them.

Here are two recent examples of being innovative in presenting evaluation findings:

Evaluation fortune cookies by Stephanie Evergreen 

Evaluation chocolates by Susan Kistler