Evaluating networks - some resources and some software

BetterEvaluation is on the ground at the European Evaluation Society 10^th Biannual conference in Helsinki this week. We’ll be presenting BetterEvaluation on Thursday but we're also attending lots of interesting sessions, and can share some highlights through this site.

I’ve just attended a session titled “Network effects on evaluation and organisation”, which is very closely related to the overall theme of the conference: Evaluation in the networked society. My hope was to learn more about the evaluation of networks themselves, an area of interest for me. After a quick summary of the session I wanted to add to this discussion by sharing a few resources on this subject, as well as some useful tools (mainly Social Network Analysis software).

The session comprised of three presentations about particular networks, but from very different perspectives: The first (Martina Rillo Otero) was an analysis of a network of evaluators in Brazil, the second (Wolfgang Meyer) talked about the evaluations of two so called ‘governance networks’ in Germany, and the third (Juha Latikka) talked about how network mapping can help decisions about researching funding in Finland – so quite a mixed bag.

The common ground between the presentations was the use of network visualisation to present relational data. The first used the visualisations alongside network analytics such as degree centrality and betweeness centrality to understand the interconnectedness of the evaluation field in Brazil; the second used visualizations to demonstrate how networks are evolving over time (they found that small networks grew while the bigger networks shrank); and the third used visualisations to show how different types of research proposals are related to each other.

For me though, there is a lot more to network analysis and evaluation of networks than visualisation – although this is perhaps the sexy part of it because you end up with pretty pictures which can bring data alive and tell fantastic stories (if used well).

For a start I’d like to see more discussion about what networks are as it’s a very overused term and anyone who uses it should lay out clear boundaries for what they are treating as network and what they are not (e.g. coalitions, partnerships, communities of practice, associations, unions are all forms of networks). This paper about the overuse of the term 'network'  provides a very clear functional definition of networks rather than structural – it’s more useful to understand a network by what it does than what it looks like.

In BetterEvaluation we are planning to develop a glossary (or perhaps more appropriately, a Rosetta Stone) of terms useful in evaluation, and we will start by looking at how these different network related terms are used.

The other issues that I think are essential in looking at networks (which I don't want to get into here, but I hope we can get to later) are power dynamics and value creation in networks - particularly using these lenses to analyse network strategies in critical ways.

By way of broadening this discussion, I’d like to share some resources that you might find helpful - please add to this list by suggesting your resources in the comments:

• Next Generation Network Evaluation, by iScale (2010): a review of approaches to evaluating networks
• Measuring while you manage: Planning, monitoring and evaluating knowledge networks (2001), by Heather Creech
• Participation, Relationships and Dynamic Change: New Thinking on Evaluating the Work of International Networks (2002), by Madeline Church et al
• Evaluating the Effects of International Advocacy Networks (2007), by Ricardo Wilson-Grau
• Promoting and assessing value creation in communities and networks: a conceptual framework (2011), Etienne Wenger, Beverly Trayner, Maarten de Laat
• Evaluating Inter-Organizational Relations (2006), Charles Lusthaus, Katrina Rojas and Christine Milton
• Evaluating Networks, IDRC (2005) – great literature reviews and examples of evaluations
• Network mapping as a diagnostic tool, Louise Clark
• Social Network Analysis in Program Evaluation (2005), special edition of the New Directions for Evaluation journal
• NetMap Toolbox: a great participatory research method based on network mapping.

Rick Davies also keeps a well maintained list of Social Network Analysis resources on his website (mande.co.uk) as well as some useful insights on the use of network models in M&E.

And here are some links to my favorite free Social Network Analysis software packages.

• UCINET - it was built by academics for academics so it's got all the analytics and fairly easy to use.
• Gephi - has an easier graphical interface than UCINET and is becoming more popular
• NodeXL  - works with MS Excel which makes it very easy to import data, it's still under development and getting better all the time.

Those are my suggestions but here is a complete list: http://en.wikipedia.org/wiki/Social_network_analysis_software

And if you are really interested in this, then you might like to join me in an online course on SNA that has just started, run by University of Michigan: https://www.coursera.org/course/sna.

Image (top-left): 2010 - February - NodeXL - cscw Twitter Network scaled by followers by Marc_Smith, on Flickr

Image (top-right): A network map of evaluators in Brazil - from the EES session by Cristina Sette