“How do we measure for epiphany?” Rob Stein’s question from MCN2011 haunts me.
Measurement is on my mind. As a theorist, I don’t tend naturally towards the quantifiable. Neither do museums in some ways. Much of the value of museums is other-than-economic, and not easily measurable. But we live in a society that demands success in quantifiable packaging. We really need a Wondermeter, but since we don’t have one finding the right questions to ask, the right things to measure becomes critical.
It occurs to me, then, that changing the conversation on online collections or museum websites (or anything that we know is important) will demand that we have the right metrics; metrics that funders or board members can make sense of in order to benchmark new ways of thinking about digital against other priorities; metrics that alter the way we think about what we do, and why.
When I think about how I conceive of the online museum collection and its potential role in the broader information ecosystem, it doesn’t make sense to me if success is measured simply by numbers of object records online or visits to the website. That doesn’t tell us anything about whether new knowledge is being created using the collection, or how the collection records are meaningful beyond the museum – and I think those are much more interesting questions. As Jasper Visser put it:
I recently realised that we, cultural institutions, are using the wrong metrics to measure our online success, because we’re measuring just that: generic success. We’re using statistics and software that is perfectly fine when you’re selling Cokes, but might not be ideal for culture, heritage and the arts.
There are some important general questions that should be asked when considering digital success and metrics (Clairey Ross’ post following Seb’s MW2011 web metrics course is a great starting place here). But I’m going to imagine some new metrics, ones that measure the things I think are *really* important, beyond the basics. You might not agree with me that these are the right things to measure. That’s great. What is?
1. What new innovations, knowledge, and digital inventions have come from people using your collection/information?
Patrick Hussey recently considered how crowdfunding is changing culture, and asked “Is it possible that crowdfunding is telling us something rather profound – that the most important and popular form of creativity at this point in history is not ‘useless’ art, but digital invention?” It’s a question that recalls to me one of the conclusions that Koven drew from his unconference session on online collections at MW2012, that “The after-market for collections data may be the most important one.” What if making possible digital invention and new collection-driven discovery was the point of museum collections? How would we measure that? Would the right metric be measuring how many new inventions/innovations have come from people utilising collection data, or what new knowledge had come of it? And if that was a metric, would that change the way your museum acted? Would you run more hack days to encourage innovation around the collection? Would you make sure your data was available as an API? Would you change your image licensing allow image downloads for non-commercial and academic use as the UK National Portrait Gallery have just done?
2. How does your collection link to the broader information ecology?
If we were to start thinking of the online museum collection as a living historical document rather than a mere catalogue, one in which we can discover things about our collections that we’ve never known before, what would indicate success? Would we measure how many Wikipedia entries lead back to the collection? How many external links lead out from the collection to authoritative sources? As Nate Solas has reminded us previously, authority online is conferred by linking to the right sources and places, by making the information that people need and want findable and available. You only have to look at the Walker’s website to see how they are making use of this kind of thinking. What could we measure to encourage more of this?
3. How is the world different because your collection, your museum is online?
Ok, this one is getting more towards the esoteric end of things, but go with me. Maxwell Anderson, in You Get What You Measure writes of answering the question “how is the world different because your museum exists?” It would be interesting to try to find measurements that answer the question “how is the world different because your collection is online?” If we cannot find ways to answer that, maybe we aren’t really having an impact at all? In which case, why bother? Of course, this isn’t something that is easily quantifiable (I told you, I’m a theorist not an evaluator), but I’d love to find new ways to measure the real importance of a museum’s online presence, of its impact as an educational institution, and the impact of the online collection. As Jasper puts it, our significance, not just our success.
I want museums and collections to be meaningful, online and offline. I think they are; or should be; or can be. But maybe they aren’t yet as meaningful as they will be. It’s not quite measuring for epiphany, but maybe it’s not that far from it.
What do you think? Am I completely wrong with my imagined new metrics? What would be better? What crazy things would you want to measure, and how would you do it? Feel free to talk about your own area of fascination, right across the institution. Don’t just limit yourself to mine.
BTW – there are some good general posts about museum metrics that have recently surfaced. Worth reading too.
museum geek 
Great post! I know metrics and measuring data is vital (although I personally think over-analyzing and making numbers fit is in issue…) but I often wonder if right ‘targets’ are being set. Regardless of how they are being collecting, what is being collected needs to almost be customized for different venues depending on targets and outcomes.
Hi Suse – been following the blog for quite a while now and really enjoy it. This morning seemed like a particularly good time to chime in.
Your comment about measuring the success of online collections really struck me in light of (what I think is) a really neat new project here at the Grand Rapids Public Museum.
First some background – I have been managing the online collections (www.grmuseum.org/collections) here at GRPM for over 4 years now. During that time we have relied exclusively on the metrics you mention (number of records online and number of page visits) to measure the “success” of this initiative. The data provide a very mixed picture though. With just over 160,000 records online we have basically the whole collection online – YAY! With page visits in double digits each week, no one cares – BOO!
Just a couple of days ago, a new project was launched by a group of volunteers (people not formally affiliated with the museum) to document the collections. Their digital project, called Artifact (www.artifactgr.org) already contains entries for 8 objects. Each one includes beautiful photographs of the object, as well as personal recollections, videos, poems, or fictional stories, and the ability for others to add their own comments.
Thankfully I don’t have to choose between the “official” GRPM database, and the Artifact project. But if I was forced to, I think I would rather have my museum represented by 8 objects with meaningful stories contributed by people in the community, than with 160,000 standard catalog entries.
In the words of the Artifact project – What if you could easily access artifacts of the past, learn from them, and allow them to influence the future? I don’t know, but I am really excited to find out.
Hi Alex! Welcome to the conversation – and what a great comment to enter with. I love your frank admission that the GRPM online collection is both hugely successful and not at all a success, depending on what metrics count. And I’m pretty confident you aren’t the only one with a similar story. We really get what we ask for. If the metric of success is “let’s make sure our collection is accessible to anyone who wants it” – a good aim in itself, and one I’ve argued for in the past – then getting it online in any format is the first priority. But then what?
It might be telling that of the two links you provided, I went first to Artifact before looking at the GRPM collection itself. And I confess, I might have a little crush on Artifact now. There are two entries in particular that just tickle me.
The first is the Robert E. Lee Sword Sash by Dale A. Robertson. First of all, never having been to your museum, I find it completely awesome and bizarre that someone’s favourite object is a sword sash. But beyond that, I love the way Dale has written about the object. He starts with the museum’s label, but then includes “the provenance provided in a public writing on the Civil War, “Men of the 3rd Michigan Infantry””, which he prefers. Dale then leads into a discussion about truth and meaning. Amazing.
I love this! Just as good is Phil Wilson’s write-up of a Brown Jug. He starts:
With this project on its radar, maybe your museum could measure something like “how has the collection been used by the people of GR to tell their stories?”
An addendum to my original post: I just found a great piece by Jonathan Stray on the Nieman Journalism blog, addressing similar questions to those I ask above, but with journalism in mind. Metrics, metrics everywhere: How do we measure the impact of journalism? Really worth a read for those interested in these issues, or to see how these same questions are being translated into other sectors.
Very challenging post.
Douglas Worts’ has used a Critical Assessment Framework to get at this very issue.
The point I want to add to this discussion is the need for a long-term approach to any sort of evaluation. In the same way that if one tags a blog with “measuring success” one will get a ton of hits, and if a museum has a blockbuster exhibit or a gimmick they can get short-term good numbers, it is over the long haul that sustainability is built.
I think there is something in the excerpt from the Neimann Journalism blog – its about the effect caused on the viewer/particpator that we need to measure. Numbers prove very little at the end of the day, yet as you point out Suse museums rely on them a lot.
Within the constructivist framework I want to know whether or how a person has ‘made meaning’ (or not) by using the online collection. And as Robert points out this is done through longditudinal evaluation. I’d like to know whether anyone has done such a study on users of online collections?
We’ve been thinking about some of the same ideas with the collections we help museums publish online on eHive. As well as tracking basic page views (popular objects) we now have a page for interesting objects. We look at a few factors, including how detailed the metadata is, what images are present, how many page views the object has had, how many comments have been added and how many folksonomy tags have been added. It’s far from perfect, but it has helped surface some items which would have been off the radar based on just page views.
We may be able to extend this to include statistics on referrals from other sites to particular objects based on the data we have in Google Analytics. This would give better information on the use of the data outside of the home site.
I think publishing entire collections online is a good start as it at least gives museums access to range of new metrics. Having a web page for each collection item gives the organisation a place to log page views, track comments and tie referrals to. It’s also the starting point for sharing structured data that can be reused – in your words, making digital reinvention possible.
Thanks Paul, it’s interesting to hear and see how museums could use the answers to some of these kinds of questions to direct their work, and how they deal with their online collections. Have you seen Seb’s post about the problems of museum data? http://www.freshandnew.org/2012/08/museum-datasets-un-comprehensive-ness-data-mining/ I’m sure you have, but if not it’s worth looking. Matt Popke’s Q’s in the comments raise some interesting issues that you might have some thoughts for addressing.