Readers of this blog are not new to my ramblings on soft peer review, social metrics and post-publication impact measures:

• can we measure the impact of scientific research based on usage data from collaborative annotation systems, social bookmarking services and social media?
• should we expect major discrepancies between citation-based and readership-based impact measures?
• are online reference management systems more robust a data source to measure scholarly readership than traditional usage factors (e.g. downloads, clickthrough rates etc.)?

These are some of the questions addressed in my COOP ’08 paper. Jason Priem also discusses the prospects of what he calls “scientometrics 2.0″ in a recent First Monday article and it is really exciting to see a growing interest in these ideas from both the scientific and the STM publishing community.

We now need to think of ways of putting these ideas into practice. Science Online London 2010 earlier this month offered a great chance to test a real-world application of these ideas in front of a tech-friendly audience and this post is meant as its official announcement.

ReaderMeter is a proof-of-concept application showcasing the potential of readership data obtained from reference management tools. Following the announcement of the Mendeley API, I decided to see what could be built on top of the data exposed by Mendeley and the first idea was to write a mashup aggregating author-level readership statistics based on the number of bookmarks scored by each of one’s publications. ReaderMeter queries the data provider’s API for articles matching a given author string. It parses the response and generates a report with several metrics that attempt to quantify the relative impact of an author’s scientific production based on its consumption by a population of readers (in this case the 500K-strong Mendeley user base):

The figure above shows a screenshot of ReaderMeter’s results for social scientist Duncan J Watts, displaying global bookmark statistics, the breakdown of readers by publication as well as two indices (the H_R index and the G_R index) which I compute using bookmarks as a variable by analogy to the two popular citation-based metrics. Clicking on a reference allows you to drill down to display readership statistics for a given publication, including the scientific discipline, academic status and geographic location of readers of an individual document:

A handy permanent URL is generated to link to ReaderMeter’s author reports (using the scheme: [SURNAME].[FORENAME+INITIALS]), e.g.:

http://readermeter.org/Watts.Duncan_J

I also included a JSON interface to render statistics in a machine-readable format, e.g.:

http://readermeter.org/Watts.Duncan_J/json

Below is a sample of the JSON output:

{
	"author": "Duncan J Watts",
	"author_metrics":
	{
		"hr_index": "15",
		"gr_index": "26",
		"single_most_read": "140",
		"publication_count": "57",
		"bookmark_count": "760",
		"data_source": "mendeley"
	},
	"source": "http://readermeter.org/Watts.Duncan_J",
	"timestamp": "2010-09-02T15:41:08+01:00"
}

Despite being just a proof of concept (it was hacked in a couple of nights!), ReaderMeter attracted a number of early testers who gave a try to its first release. Its goal is not to redefine the concept of research impact as we know it, but to complement this notion with usage data from new sources and help identify aspects of impact that may go unnoticed when we only focus on traditional, citation-based metrics. Before a mature version of ReaderMeter is available for public consumption and for integration with other services, though, several issues will need to be addressed.

1. Author name normalisation

The first issue to be tackled is the fact the same individual author may be mentioned in a bibliographic record under a variety of spelling alternates: Rod Page was among the first to spot and extensively discuss this issue, which will hopefully be addressed in the next major upgrade (unless a provision to fix this problem is directly offered by Mendeley in a future upgrade of their API).

2. Article deduplication

A similar issue affects individual bibliographic entries, as noted by Egon Willighagen among others. Given that publication metadata in reference management services can be extracted by a variety of sources, the uniqueness of a bibliographic record is far from given. As a matter of fact, several instances of the same publication can show up as distinct items, with the result of generating flawed statistics when individual publications and their relative impact need to be considered (as is the case when calculating the H- and G-index). To what extent crowdsourced bibliographic databases (such as those of Mendeley, CiteULike, Zotero, Connotea, and similar distributed reference management tools) can tackle the problem of article duplication as effectively as manually curated bibliographic databases, is an interesting issue that sparked a heated debate (see this post by Duncan Hull and the ensuing discussion).

3. Author disambiguation

A way more challenging problem consists in disambiguating real homonyms. At the moment, ReaderMeter is unable to tell the difference between two authors with an identical name. Considering that surnames like Wang appear to be shared by about 100M people on the planet, the problem of how to disambiguate authors with a common surname is not something that can be easily sorted out by a consumer service such as ReaderMeter. Global initiatives with a broad institutional support such as the ORCID project are trying to fix this problem for good by introducing a unique author identifier system, but precisely because of their scale and ambitious goal they are unlikely to provide a viable solution in the short run.

4. Reader segmentation and selection biases

You may wonder: how genuine is data extracted from Mendeley as an indicator of an author’s actual readership? Calculating author impact metrics based on the user population of a specific service will always by definition result in skewed results due to different adoption rates by different scientific communities or demographic segments (e.g. by academic status, language, gender) within the same community. And how about readers who just don’t use any reference management tools? Björn Brembs posted some thoughtful considerations on why any such attempt at measuring impact based on the specific user population of a given platform/service is doomed to fail. His proposed solution, however – a universal outlet where all scientific content consumption should happen–sounds not only like an unlikely scenario, but also in many ways an undesirable one. Diversity is one of the key features of the open source ecosystem, for one, and as long as interoperability is achieved (witness the example of the OAI protocol and its multiple software implementation), there is certainly no need for a single service to monopolise the research community’s attention for projects such as ReaderMeter to be realistically implemented. The next step on ReaderMeter’s roadmap will be to integrate data from a variety of content providers (such as CiteULike or Bibsonomy) that provide free access to article readership information: although not the ultimate solution to the enormous problem of user segmentation, data integration from multiple sources should hopefully help reduce biases introduced by the population of a specific service.

What’s next

I will be working in the coming days on an upgrade to address some of the most urgent issues, in the meantime feel free to test ReaderMeter, send me your feedback and feature requests, follow the latest news on the project via Twitter or just help spread the word!

A glimpse at the hottest topics in scholarly literature according to PhdComics.

[via FlowingData]

One of the themes we’ve been discussing here is the idea that prestige and attention are the main currencies of academia. So it only makes sense that you want your online presence to be an accessible and positive reflection of your work and, at the very least, you want to be distinguishable from all of the other John Smiths in the world.

MIT has recently put together a tool called Personas which attempts to figure out this question of online identity. I say attempts because to be honest, it’s a bit hit and miss. The design looks pretty good but the results seem to change each time you run it, you can’t review the underlying data and it doesn’t even have a roll-over to quantify each chunk of your profile (e.g. percent of total, source documents etc).

[click for bigger]

It’s a noble effort though and it got me thinking that there are two sides to the question of online identity.

• User perspective: It’s a pain having a different user name and password for every website so initiatives like OpenID and Shibboleth should make things easier by providing common log-on standards. Similarly APML (Attention Profile Markup Language) is an emerging technology for customizing content based on your interests and habits. Both technologies are valuable for improving your online experience.
• Search perspective: The second issue is being able to search for someone and know that the results pertain to a specific person. The user perspective technologies discussed above can help in this regard, as they establish a common link between all of your online activities. And although I’ve misplaced the link somewhere, I’ve heard of some researchers using generic online data to reverse-engineer a unique identify (e.g. a Joe from California on Facebook here, a Joe who works at UCLA in biology there, etc.).

Clearly there are a lot of privacy issues involved as marketers (and governments) would love to have this sort of detailed record of who went where doing what. But let’s come back to the question of academic online identity. For a user perspective, I think we’re getting there. I can’t speak for everyone obviously but at Imperial, my single sign-on gives access to my publications database, the online journals, administrative data and many other facilities both on the local network and the wider web.

From a search perspective, there’s a way to go before we can amalgamate our various online activities into a consistent verifiable public identity. Yet the academic environment is the perfect place to start building and testing these identity systems. There’s a wealth of metadata available in journals (citations, institutions etc) and one could establish fairly well-defined problem boundaries for example by using the .edu, .ac.uk or journal publisher domains. Google Scholar probably already does this to some extent but when searching for an author, it doesn’t suggest different unique authors. Instead I would love to have one portal, accessed by a single identity which is verified by some official higher education authority, that could crawl the web and aggregate publications, blogs, newspaper articles, conference appearances, etc and combine this with social meta-data from citations or other sources (e.g. LinkedIn). Users could create public profiles and the private data could be useful for determining discipline rankings and influence (e.g. the IDEAS ranking of economists) and so on.

There’s a lot going on in this area and I’ve probably only skimmed the surface. But I wanted to raise the issue and see if anyone had any thoughts about how online identity issues for academics could be handled. At the very least, have a play with the Personas thingy and see if you, like me, are 5% illegal.

Following up on Jose’s musings on good and bad keywords for a productivity blog, I came across an interesting tool to visualize the evolution over time of aggregated social bookmarking tags for popular websites. It is actually a pretty old project called Cloudalicious created a few years ago by Terrell Russell (of ClaimID fame).

If you are a web metrics maniac like yours truly, you won’t resist plugging this tool into your favourite websites, so here’s the graph I generated for AcademicProductivity.com:


(source)

The first nice fact this graph suggests is that the more bookmarks a blog accumulates over time, the more stable the overall tag distribution tends to be, i.e. popular tags tend to get stronger and less popular tags (like tags used by a single user) to be pushed down the list. Obviously, only tags for the top node are displayed in the graph, not the sum of tags for each blog post (which might produce significantly different results).

Even so, the graph reveals some interesting facts about this blog’s core business. According to our del.icio.us readers we are mostly a blog about productivity (103 tags), academic (56 tags) or academia (50 tags), research (50 tags) and gtd (47 tags).

Possibly the most striking feature of this graph is the dramatic drop of bookmarks marked with tag gtd, that moved from a respected 2nd position about one year ago to the current 6th position.

Is there anything we can conclude from this? Is our focus drifting away from GTD? Do our readers find this blog less GTD-centred than one year ago?

I’m not sure I have a good explanation why this happened but I’d love to hear your thoughts. And if you’re intrigued by Cloudalicious, you can give it a try as well.

According to a study by research firm Basex recently covered by the New York Times, information overload will be the Problem of the Year in 2008, costing US companies up to $650 billion a year. The figure is supposed to be an estimate of the cost of unnecessary interruptions in terms of “decreased productivity and stifled innovation”. Recipes to fight email overload, in particular, have become a thriving business over the last few years: how to cope with the stress and lack of productivity caused by an ever-growing volume of email in your inbox?

While self-proclaimed gurus are selling on the Web their own ultimate solutions against email overload, Carolin Horn from DMI Boston has designed a clever visualization tool to represent unresponded email in your inbox. I find this idea way more effective than a million GTD techniques and I think Carolin and her coder collaborator Florian Jenett are onto something.

Carolin says:

Anymails is a visualization of my received emails.
I have investigated how I can use natural metaphors to visualize my inbox, its structure and attributes. The metaphor of microbes is used.

Different categories of email (family, work, university, spam) are represented by different species of microbes. The more recent and urgent an email is, the hairier and faster the corresponding microbe.

With Anymails, your inbox becomes the playground for a swarm of squirming creatures, which you can filter, arrange and group at your convenience. You can even travel back to a time when your mailbox was a nice place inhabited by tame, hairless and well-behaved email messages.

The Anymails prototype is built in Flash and Processing and retrieves email from the user local Apple Mail database. Anymails is available for download (with source included!) from Carolin’s website. I haven’t tested it myself on my local email database (I probably don’t dare see what creatures lurk inside) and a few disclaimers warn that this should not be considered more than a prototype, but Carolin’s videos illustrate the functionality of this great experiment in information visualization. Good job, Carolin and Florian!