BetterEvaluation 2022 Holiday Reading List
Happy Holidays! As we head into 2022 we thought we'd share a list of resources for you to peruse in the new year.
The selection includes some picks from the BetterEvaluation team, as well as some contributions from the BetterEvaluation community over recent years.
This isn't a definitive list of resources by any means, it's purely meant to share a few items we think will be of interest or fun to read as we get into gear for the year to come.
We hope you find something in this list that inspires you to try something new or think about something in a different light in 2022, or just makes you smile.
Enjoy!
Contents
What is evaluation? Patricia Rogers' picks for understanding what evaluation is and why it's important
- Evaluation Building Blocks - A Guide
- Eva the Evaluator
- Leaving a footprint. Stories of evaluations that made a difference
Stories and listening. Alice Macfarlan's picks for valuing the qualitative in evaluation
- Week 44: Anecdote as Epithet - Rumination #1 from Qualitative Research and Evaluation Methods
- Dr Richard Kreuger on Qualitative Listening
- Deep Listening – Impact beyond words (podcast)
Community contributions
- Designing Quality Survey Questions - Recommended by Kylie Hutchinson
- OPM’s approach to assessing Value for Money - Recommended by Anand Maraj
- A Short Primer on Innovative Evaluation Reporting - Recommended by Sheila B. Robinson
- Intentional Practice for Museums: A Guide for Maximizing Impact - Recommended by Erin Wilcox
- Delivering and Demonstrating Mediation Results - Recommended by Ian Wadley
What is evaluation? Patricia Rogers' picks for understanding what evaluation is and why it's important
"So what do you actually do?" It's a common question at end of year gatherings and family events - on top of having spent much of the working year explaining to managers and staff that there is more to evaluation, and evaluators, than their previous negative experiences suggest. Here are some of my favourite resources to explain evaluation to friends, family, colleagues and strangers. Patricia.
Evaluation building blocks - A guide
View Evaluation building blocks - A guide
Recommended by: Patricia Rogers
Authors: Kinnect Group - Kate McKegg, Judy Oakden, Nan Wehipeihana, Julian King
Year: 2018
Key features
This free downloadable guide provides a simple 9 step process for planning and undertaking an evaluation.
It follows a similar logic to BetterEvaluation’s Manager’s Guide to Evaluation, with emphasis up-front on identifying the purpose and intent of the evaluation, and developing a theory of change to inform the evaluation.
How might someone use this resource?
It is particularly strong in terms of Step 4 – develop a framework to assess performance, which has detailed examples of using global scales (rubrics) to synthesise both quantitative and qualitative data to avoid the common problems caused by replying only on Key Performance Indicators and targets.
Why I'd recommend this resource
This could be a useful resource for both those who are new to evaluation (as it provides a step by step process) and those who are aiming to improve their evaluation practice. I’d especially recommend it in terms of developing a framework for evaluating performance.
Eva the evaluator
Recommended by: Patricia Rogers
Authors: Roger Miranda (Author), Birgit Stadler (Illustrator)
Year: 2009
Key features
The story revolves around Eva and her father as he explains to her what he does for a living (evaluator). The father answers Eva's questions as she imagines herself engaged in the scenarios being described. Some mischievous characters appear highlighting that evaluation is not without pitfalls.
How I've used this
I’ve used this as a gift for evaluators (who appreciate the many visual jokes referencing famous evaluation writers) and also for non-evaluator friends and family as a gentle way into understanding the world in which I work. (Yes, I really did buy it for my mother for a Christmas present – and she keeps it on the coffee table to show visitors). I would like to use it one day in a professional development course, perhaps as a follow-up to Michael Patton’s metaphor exercise (“Evaluation is like a …. because …….), where we might explore the various roles an evaluator can play.
Why I'd recommend this resource
For my evaluator friends, it helps to create a sense of being part of a community of practice, with common reference points and in-jokes. For non-evaluators, it is a gentle introduction to a longer discussion we might have. It doesn’t present a definition of evaluation or of evaluators, but that’s where it could help the conversation to go.
Leaving a footprint: Stories of evaluation that made a difference
Recommended by: Patricia Rogers
Authors: Pablo Rodríguez Bilella and Esteban Tapella
Year: 2018
Key features
These accounts of evaluations that have made a difference can be used to inspire and inform the work of those who manage evaluations and those who do them. Available in English and Spanish
How I've used this
I haven’t yet used this but will be drawing on it in future evaluation courses. It’s particularly helpful to have good examples of evaluations to analyse and discuss. I would use it initially to give participants practice in applying evaluation terminology (for example - Is this formative or summative evaluation? Who made the decisions about the evaluation? What were the key evaluation questions)
Why I'd recommend this resource
We need good examples of evaluations that tell the story of how and why it was conducted – stories that are rarely evident from published evaluation reports or journal articles. These accounts are particularly useful as they provide non-Spanish speakers with insights from practice in Latin America. Seeing quite different approaches to evaluation can enlarge our repertoire, or at the very least make us more aware of the choices and assumptions we make.
Stories and evaluation
I found myself down a rabbit hole recently exploring qualitative data, stories, and listening in evaluation and wanted to share some of the resources that I came across. Alice.
Week 44: Anecdote as epithet - Rumination #1 from qualitative research and evaluation methods
Recommended by: Alice Macfarlan
Authors: Michael Quinn Patton
Year: 2014
Key features
This blog from the BetterEvaluation archive is part of a series of ruminations from Michael Quinn Patton, originally published in the 4th edition of Qualitative Research and Evaluation Methods (Michael Quinn Patton, 2014). In these ruminations, Patton reflects on issues that he explains have “persistently engaged, sometimes annoyed, occasionally haunted, and often amused me over more than 40 years of qualitative research and evaluation practice. In these ruminations I state my case on the issue and make my peace with it.” In this rumination, Patton discusses the place of anecdotes in evaluation.
How I’ve used this resource
Qualitative data still gets a bad rap, and many a time I’ve been in the situation where I’ve come up against arguments that qualitative data just isn’t robust enough. Try as I might to bring people round to the merits of qualitative data, I’m often not quite as articulate as I’d like to be (which is a real shame considering I’m trying to argue for the value of words). This is when I like to come back to this rumination, which does this much better – unpacking and rebutting the common (mis)quote: “the plural of anecdote is not data”.
Why I'd recommend this resource
It’s a lovely read on the value of people’s stories and makes a valid case for how anecdotes can be used as data – but also offers an important caveat to this by suggesting that it’s not simply the anecdotes alone that make meaningful data, but the ability of the researcher to “listen, systematically collect, and rigorously analyse anecdotes.”
Dr Richard Kreuger on qualitative listening
View Dr Richard Kreuger on qualitative listening
Recommended by: Alice Macfarlan
Authors: Susan Eliot (interviewer) and Richard Kreuger (interviewee)
Year: 2014
Key features
In this interview in The Listening Resource blog (August 29th, 2013), Susan Eliot talks to Dr. Richard Kreuger, professor emeritus at the University of Minnesota specializing in program evaluation, applied research methodology and qualitative analysis, about qualitative interview skills and how new researchers can apply these.
How I've used this resource
To understand more about the complexities of listening in the context of evaluation, and how this differs from listening in other contexts. It also has some specific strategies that can be applied to difficult listening situations.
Why I'd recommend this resource
It's a candid and informative discussion between two very knowledgeable practitioners. I imagine even those who have been interviewing and running focus groups for many years will get something out of this conversation.
Deep listening – Impact beyond words (podcast)
Recommended by: Alice Macfarlan
Authors: Oscar Trimbolo
Year: 2017-current
Key features
This podcast by Oscar Trimboli takes a deep dive into the process of listening. Each episode contains an interview with a different 'listener' - including translators, doctors, journalists, marketers, anthropologists, mediators and linguists.
How I've used this resource
It's fascinating to get a glimpse into the ways others listen, and why they do this. I'm only a few episodes in but I'm really enjoying this as a way to reflect on and improve my own practice of listening.
Why I'd recommend this resource
I have a linguistics background so I might be more inclined to find a podcast about the process of listening and communication interesting than other people, but I genuinely think this is a great podcast, especially for evaluators. Listening is such an important part of the evaluation process - from listening to user needs around use and engagement, to data collection through interviews and focus groups, and facilitating and managing relationships - and yet it's one of those 'soft' evaluation skills that can be overlooked and underestimated. I'd recommend putting this podcast on your holiday playlist to reflect and learn about your own listening practice during the break.
Community contributions
Thanks to everyone who has suggested resources for this post!
OPM’s approach to assessing value for money
View OPM’s approach to assessing value for money (PDF, 2.5MB)
Recommended by: Anand Maraj (as well as being in Patricia Rogers' list)
Author: Oxford Policy Management and Julian King & Associates
Year: 2018
Key features
This clear guide discusses different aspects of value for money – economy, efficiency (in terms of production of outputs), effectiveness, cost-effectiveness (in terms of outcomes), and equity. It uses explicit evaluative reasoning and global scales (rubrics) for assessing performance in terms of each of these.
Anand Maraj: Why I'd recommend this:
I would like to recommend OPM’s approach to assessing Value for Money by Julian King, Oxford Policy Management. I am required to provide assessments at varying intervals on the performance of Government Policy and whether programs and/or projects provide value. I found this guide to be well defined and informative.
Patricia Rogers: Why I'd recommend this
Its greatest strength is the explicit use of evaluative thinking in drawing the evaluative conclusions. Many guides to economic evaluation focus only on the cost of achieving intended objectives. This guide examines trade-offs between intended and unintended outcomes, and distributional effects.
Designing quality survey questions
View Designing quality survey questions
Recommended by: Kylie Hutchinson
Author: Sheila B. Robinson and Kimberly Firth Leonard
Year: 2018
Key features
This book covers considerations before drafting a survey, including understanding the purpose and audience of your survey; developing survey questions and response options, including avoiding common problems and designing special purpose and sensitive questions; and knowing when a survey is ready for use, including pretesting strategies and survey instructions. The appendix also includes a survey design checklist, and there are currently two chapters available for free on the Sage website: Chapter 1: WHY Quality Surveys and Quality Questions Matter and Chapter 4: Sourcing and Crafting Questions.
The book includes numerous examples of practice and discussion questions, practice exercises ('design drills') and further reading lists.
Kylie Hutchinson: Why I'd recommend this
So, true story: I had a tight deadline to re-vamp an evaluation client’s questionnaire, but I also wanted to give myself a quick refresher on survey design by reading this book. I only had a couple of hours while my young daughter was at an event so I sat in my car in the parking lot and read Chapters 4 & 5.
Wow. It was well-written, easy to read, to-the-point, and had lots of great examples. It was such a worthwhile use of my time. I'm looking forward to finishing the rest of it.
A short primer on innovative evaluation reporting
View A short primer on innovative evaluation reporting
Recommended by: Sheila B. Robinson
Author: Kylie Hutchinson
Year: 2017
Key features
This book by Kylie Hutchinson presents a number of innovative ways of reporting, including different options for presentations, narrative summaries, presenting findings visually and making use of digital outputs. Kylie also discusses how to make sure your messages get through to your audience - including using a layering strategy to present your key messages in different levels of depth across multiple content types, and advice on making these message 'sticky'.
Sheila Robinson: Why I'd recommend this
I was asked to do a presentation at a statewide conference on evaluation reporting and got a copy of this book just in time for it to inform what I taught my audience, and to recommend it to them as a rich, accessible, and truly innovative resource. What I especially love about this book is that it's not just a list of ideas, but rather starts with a solid foundation of WHY we would put these new ideas about evaluation reporting into place. That sets the mindset for using the ideas.
I truly enjoyed this book and go back to it again and again as I'm thinking about how to report evaluation results to different audiences.
Intentional practice for museums: A guide for maximizing impact
View Intentional practice for museums: A guide for maximizing impact
Recommended by: Erin Wilcox
Author: Randi Korn, Founder of RK&A, Inc.
Year: 2018
Key features
The book describes an impact-driven way of thinking and working that places a museum’s raison d’être—achieving impact—at the center of its work. The book deconstructs the Cycle of Intentional Practice, which embodies four quadrants (plan, evaluate, reflect, and align) situated around the centerpiece—impact. As exemplified by its “cycle” quality, impact-driven work is ongoing. Museums that choose to pursue impact through intentional practice will benefit, as they, as well as their organization, will be on a continuous cycle of learning.
Erin Wilcox: Why I would recommend this
I've used this resource for guidance in developing impact statements and impact frameworks to serve as a guidepost for achieving intended impact.
The book describes the seven principles of intentional practice and provides basic intentional-practice strategies, exercises, and facilitation questions so readers can begin facilitating impact-driven workshops at their museums.
Valuing peace: Delivering and demonstrating mediation results
View Valuing peace: Delivering and demonstrating mediation results
Recommended and authored by: Ian Wadley, Head of strategy and evaluation at the Centre for Humanitarian Dialogue, Geneva.
Year: 2017
Key features
This is a discussion paper which offers some solutions for practitioners and donors seeking to demonstrate value-for-money in complex environments with limited data, no baseline, and high confidentiality requirements. It describes an adaptive M&E system that adjusts the level of evaluation focus depending on the available evidence, and is grounded in the principles of public sector value-for-money audit / performance audit.
Ian Wadley: Why I'd recommend this
The resource is useful for the mediation and peacemaking sector, but is relevant to negotiation, diplomacy, policy influence, humanitarian affairs, and any domain where the quality of professional judgements in situations of uncertainty is a key element for success.
The resource is targetted towards evaluation managers, donors and practitioners facing the challenge of demonstrating value-for-money. The adaptive M&E and peer review methods described are now being piloted in a number of organisations and government ministries. It is freely available online, and the author welcomes enquiries from interested colleagues.
We hope you enjoy these resources and have a happy holidays!