No one can read everything. We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped. However, the growth of new, online scholarly tools allows us to make new filters; these alt-metrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem. We call for more tools and research based on alt-metrics.

As the volume of academic literature explodes, scholars rely on filters to select the most relevant and significant sources from the rest.

Unfortunately, scholarship’s three main filters for importance are failing:

• Peer-review has served scholarship well, but is beginning to show its age. It is slow, encourages conventionality, and fails to hold reviewers accountable. Moreover, given that most papers are eventually published somewhere, peer-review fails to limit the volume of research.
• Citation counting measures are useful, but not sufficient. Metrics like the h-index are even slower than peer-review: a work’s first citation can take years. Citation measures are narrow; influential work may remain uncited. These metrics are narrow; they neglect impact outside the academy, and also ignore the context and reasons for citation.
• The JIF, which measures journals’ average citations per article, is often incorrectly used to assess the impact of individual articles. It’s troubling that the exact details of the JIF are a trade secret, and that significant gaming is relatively easy.

Tomorrow’s filters: alt-metrics

In growing numbers, scholars are moving their everyday work to the web. Online reference managers Zotero and Mendeley each claim to store over 40 million articles (making them substantially larger than PubMed); as many as a third of scholars are on Twitter, and a growing number tend scholarly blogs.

These new forms reflect and transmit scholarly impact: that dog-eared (but uncited) article that used to live on a shelf now lives in Mendeley, CiteULike, or Zotero–where we can see and count it. That hallway conversation about a recent finding has moved to blogs and social networks–now, we can listen in. The local genomics dataset has moved to an online repository–now, we can track it. This diverse group of activities forms a composite trace of impact far richer than any available before. We call the elements of this trace alt-metrics.

Alt-metrics expand our view of what impact looks like, but also of what’s making the impact. This matters because expressions of scholarship are becoming more diverse. Articles are increasingly joined by:

• The sharing of “raw science” like datasets, code, and experimental designs
• Semantic publishing or “nanopublication,” where the citeable unit is an argument or passage rather than entire article.
• Widespread self-publishing via blogging, microblogging, and comments or annotations on existing work.

Because alt-metrics are themselves diverse, they’re great for measuring impact in this diverse scholarly ecosystem. In fact, alt-metrics will be essential to sift these new forms, since they’re outside the scope of traditional filters. This diversity can also help in measuring the aggregate impact of the research enterprise itself.

Alt-metrics are fast, using public APIs to gather data in days or weeks. They’re open–not just the data, but the scripts and algorithms that collect and interpret it. Alt-metrics look beyond counting and emphasize semantic content like usernames, timestamps, and tags. Alt-metrics aren’t citations, nor are they webometrics; although these latter approaches are related to alt-metrics, they are relatively slow, unstructured, and closed.

How can alt-metrics improve existing filters?

With alt-metrics, we can crowdsource peer-review. Instead of waiting months for two opinions, an article’s impact might be assessed by thousands of conversations and bookmarks in a week. In the short term, this is likely to supplement traditional peer-review, perhaps augmenting rapid review in journals like PLoS ONE, BMC Research Notes, or BMJ Open. In the future, greater participation and better systems for identifying expert contributors may allow peer review to be performed entirely from alt-metrics. Unlike the JIF, alt-metrics reflect the impact of the article itself, not its venue. Unlike citation metrics, alt-metrics will track impact outside the academy, impact of influential but uncited work, and impact from sources that aren’t peer-reviewed. Some have suggested alt-metrics would be too easy to game; we argue the opposite. The JIF is appallingly open to manipulation; mature alt-metrics systems could be more robust, leveraging the diversity of of alt-metrics and statistical power of big data to algorithmically detect and correct for fraudulent activity. This approach already works for online advertisers, social news sites, Wikipedia, and search engines.

The speed of alt-metrics presents the opportunity to create real-time recommendation and collaborative filtering systems: instead of subscribing to dozens of tables-of-contents, a researcher could get a feed of this week’s most significant work in her field. This becomes especially powerful when combined with quick “alt-publications” like blogs or preprint servers, shrinking the communication cycle from years to weeks or days. Faster, broader impact metrics could also play a role in funding and promotion decisions.

Road map for alt-metrics

Speculation regarding alt-metrics (Taraborelli, 2008; Neylon and Wu, 2009; Priem and Hemminger, 2010) is beginning to yield to empirical investigation and working tools. Priem and Costello (2010) and Groth and Gurney (2010) find citation on Twitter and blogs respectively. ReaderMeter computes impact indicators from readership in reference management systems. Datacite promotes metrics for datasets. Future work must continue along these lines.

Researchers must ask if alt-metrics really reflect impact, or just empty buzz. Work should correlate between alt-metrics and existing measures, predict citations from alt-metrics, and compare alt-metrics with expert evaluation. Application designers should continue to build systems to display alt-metrics, develop methods to detect and repair gaming, and create metrics for use and reuse of data. Ultimately, our tools should use the rich semantic data from alt-metrics to ask “how and why?” as well as “how many?”

Alt-metrics are in their early stages; many questions are unanswered. But given the crisis facing existing filters and the rapid evolution of scholarly communication, the speed, richness, and breadth of alt-metrics make them worth investing in.

Jason Priem (University of North Carolina-Chapel Hill)
Dario Taraborelli (University of Surrey)
Paul Groth (VU University Amsterdam)
Cameron Neylon (Science and Technology Facilities Council)

Source: http://altmetrics.org/manifesto 

Readers of this blog are not new to my ramblings on soft peer review, social metrics and post-publication impact measures:

• can we measure the impact of scientific research based on usage data from collaborative annotation systems, social bookmarking services and social media?
• should we expect major discrepancies between citation-based and readership-based impact measures?
• are online reference management systems more robust a data source to measure scholarly readership than traditional usage factors (e.g. downloads, clickthrough rates etc.)?

These are some of the questions addressed in my COOP ’08 paper. Jason Priem also discusses the prospects of what he calls “scientometrics 2.0″ in a recent First Monday article and it is really exciting to see a growing interest in these ideas from both the scientific and the STM publishing community.

We now need to think of ways of putting these ideas into practice. Science Online London 2010 earlier this month offered a great chance to test a real-world application of these ideas in front of a tech-friendly audience and this post is meant as its official announcement.

ReaderMeter is a proof-of-concept application showcasing the potential of readership data obtained from reference management tools. Following the announcement of the Mendeley API, I decided to see what could be built on top of the data exposed by Mendeley and the first idea was to write a mashup aggregating author-level readership statistics based on the number of bookmarks scored by each of one’s publications. ReaderMeter queries the data provider’s API for articles matching a given author string. It parses the response and generates a report with several metrics that attempt to quantify the relative impact of an author’s scientific production based on its consumption by a population of readers (in this case the 500K-strong Mendeley user base):

The figure above shows a screenshot of ReaderMeter’s results for social scientist Duncan J Watts, displaying global bookmark statistics, the breakdown of readers by publication as well as two indices (the H_R index and the G_R index) which I compute using bookmarks as a variable by analogy to the two popular citation-based metrics. Clicking on a reference allows you to drill down to display readership statistics for a given publication, including the scientific discipline, academic status and geographic location of readers of an individual document:

A handy permanent URL is generated to link to ReaderMeter’s author reports (using the scheme: [SURNAME].[FORENAME+INITIALS]), e.g.:

http://readermeter.org/Watts.Duncan_J

I also included a JSON interface to render statistics in a machine-readable format, e.g.:

http://readermeter.org/Watts.Duncan_J/json

Below is a sample of the JSON output:

{
	"author": "Duncan J Watts",
	"author_metrics":
	{
		"hr_index": "15",
		"gr_index": "26",
		"single_most_read": "140",
		"publication_count": "57",
		"bookmark_count": "760",
		"data_source": "mendeley"
	},
	"source": "http://readermeter.org/Watts.Duncan_J",
	"timestamp": "2010-09-02T15:41:08+01:00"
}

Despite being just a proof of concept (it was hacked in a couple of nights!), ReaderMeter attracted a number of early testers who gave a try to its first release. Its goal is not to redefine the concept of research impact as we know it, but to complement this notion with usage data from new sources and help identify aspects of impact that may go unnoticed when we only focus on traditional, citation-based metrics. Before a mature version of ReaderMeter is available for public consumption and for integration with other services, though, several issues will need to be addressed.

1. Author name normalisation

The first issue to be tackled is the fact the same individual author may be mentioned in a bibliographic record under a variety of spelling alternates: Rod Page was among the first to spot and extensively discuss this issue, which will hopefully be addressed in the next major upgrade (unless a provision to fix this problem is directly offered by Mendeley in a future upgrade of their API).

2. Article deduplication

A similar issue affects individual bibliographic entries, as noted by Egon Willighagen among others. Given that publication metadata in reference management services can be extracted by a variety of sources, the uniqueness of a bibliographic record is far from given. As a matter of fact, several instances of the same publication can show up as distinct items, with the result of generating flawed statistics when individual publications and their relative impact need to be considered (as is the case when calculating the H- and G-index). To what extent crowdsourced bibliographic databases (such as those of Mendeley, CiteULike, Zotero, Connotea, and similar distributed reference management tools) can tackle the problem of article duplication as effectively as manually curated bibliographic databases, is an interesting issue that sparked a heated debate (see this post by Duncan Hull and the ensuing discussion).

3. Author disambiguation

A way more challenging problem consists in disambiguating real homonyms. At the moment, ReaderMeter is unable to tell the difference between two authors with an identical name. Considering that surnames like Wang appear to be shared by about 100M people on the planet, the problem of how to disambiguate authors with a common surname is not something that can be easily sorted out by a consumer service such as ReaderMeter. Global initiatives with a broad institutional support such as the ORCID project are trying to fix this problem for good by introducing a unique author identifier system, but precisely because of their scale and ambitious goal they are unlikely to provide a viable solution in the short run.

4. Reader segmentation and selection biases

You may wonder: how genuine is data extracted from Mendeley as an indicator of an author’s actual readership? Calculating author impact metrics based on the user population of a specific service will always by definition result in skewed results due to different adoption rates by different scientific communities or demographic segments (e.g. by academic status, language, gender) within the same community. And how about readers who just don’t use any reference management tools? Björn Brembs posted some thoughtful considerations on why any such attempt at measuring impact based on the specific user population of a given platform/service is doomed to fail. His proposed solution, however – a universal outlet where all scientific content consumption should happen–sounds not only like an unlikely scenario, but also in many ways an undesirable one. Diversity is one of the key features of the open source ecosystem, for one, and as long as interoperability is achieved (witness the example of the OAI protocol and its multiple software implementation), there is certainly no need for a single service to monopolise the research community’s attention for projects such as ReaderMeter to be realistically implemented. The next step on ReaderMeter’s roadmap will be to integrate data from a variety of content providers (such as CiteULike or Bibsonomy) that provide free access to article readership information: although not the ultimate solution to the enormous problem of user segmentation, data integration from multiple sources should hopefully help reduce biases introduced by the population of a specific service.

What’s next

I will be working in the coming days on an upgrade to address some of the most urgent issues, in the meantime feel free to test ReaderMeter, send me your feedback and feature requests, follow the latest news on the project via Twitter or just help spread the word!

I want to try something a bit different in this post. Here at AP.com, we’ve talked a lot about tools, theory, trends and the general ephemera of academic productivity. But writing as academics, we should probably be trying to take this experience and build it into a cohesive model of productivity. So my goal here is to suggest a general model, one that we might use to understand what we’ve learned from previous posts and hopefully apply to our own work.

My starting point for this post was simple; I wanted to know how my productivity has changed (hopefully improved) since I first started my DPhil. From keeping a research journal, I know that some days are more productive than others and it would very helpful if I could understand when those fits and starts occur, to spot co-occuring events and thereby learn when to say “Forget work, I’m going for a run.”

In other words, I wanted to plot my productivity cycle over time. It might look something like this:

But the obvious problem with this exercise is how to measure productivity. It’s a subject that’s been tackled indirectly on this site before but going through the old posts, I haven’t yet find any attempts at a general theory – and related measures – of productivity. So drawing on the collected wisdom of previous AP.com posts, here’s a rough sketch of such a theory.

What is productivity?

Most simply, productivity is a question of efficiency: what outputs can be produced for a given amount of inputs? If you were working in a factory, measuring productivity is therefore fairly straight-forward: widgets per hour might be a nice personal productivity measure. But in an academic context, these inputs and outputs are not so easily defined.

This dilemma is briefly introduced in a discussion of productivity for programmers, where Gil Gordon suggests that when we say productivity, we really mean effectiveness. In other words, unlike a factory worker, our outputs can be multi-faceted and might be judged by their:

• Quantity (how much gets done)
• Quality (how well it gets done)
• Timeliness (when it gets done)
• Multiple priorities (how many things can be done at once)

Nevertheless the basic efficiency model is a good template for our model. So if P = productivity, O = output and I = input, we can write the basic definition:

Defining outputs

There are many different kinds of academic output. Papers, citations, funding received, teaching feedback, and promotions are just some of the ways in which we can measure our success, either directly or indirectly.

But as Jose pointed out previously, these outputs are all trying to attract a scarce resource, namely attention, and success in attracting attention results in prestige. So we might say that, for a given class of task t, output could be measured as:

where p_t is the prestige associated with task t and n_t is the number of those tasks completed in a given time period.

Measuring prestige varies with the task. For journal papers, citations seems like a sensible measure but for other tasks, this may involve a lot of guesswork. If your university has guidelines on promotion, they can be useful in identify how much of your professional success is expected to come from teaching, research and so on. But most likely you will need to use a technique like Analytic Hierarchy Process to unify your prestige measures.

Defining inputs

If you follow the game of cricket, you are probably familiar with the Duckworth-Lewis method. In an English summer, it often happens that one team has finished batting and their opponents have just started trying to catch the target score when it begins to rain. Rather than call off the whole game, the DL method is used to adjust the target score to account for a reduced amount of playing time. To do this, Messrs. Duckworth and Lewis developed their model using the concept of resources.

With productivity inputs, we can do something similar. The amount and quality of work that we can achieve depends on the resources available to us. But instead of wickets and overs in cricket, academic input resources might include time, money, lab access, the attention and effort we can devote to a task and so on.

Again, some of these inputs are more easily measured than others but if we want to generalize our model, we need some sort of conceptual common currency like we had with prestige on the output side. An economist might attempt to convert everything to money: how much would I have to spend to acquire this piece of data? But to link with our earlier discussion of prestige, I think a more useful framework is to convert everything to a common attention unit: let’s call it the Atnu for short (AU is already taken).

1 Atnu can be defined as the amount of attention necessary for a reference task, such as reading a journal article. It’s a rather arbitrary unit, but it’s intended to acknowledge that an hour of hard concentration is not the same as an hour spent doing miscellaneous administrative tasks like sorting through emails. It also has the advantage that you can define the Atnu as it makes sense to you and your work; if you spend time in a lab, performing an assay might be the base unit. My only suggestion would be that it is the defined as the most attention-consuming task. That way, the most difficult part of your day will correspond with actual hours.

So for task t, we can say that the total input is the total amount of attention hours spent on the job.

where a_t is the Atnu value for task t and h_t is the actual number of hours spent working at that level.

Final notes

Putting it all together then, productivity is the amount of prestige we earn for each attention-hour we invest.

It is a very simple model but from the form of this equation, we can already draw a few practical conclusions (even if they just confirm what we intuitively knew already):

• Productivity is maximized by concentrating on those activities that earn you the most prestige for the least effort. Conveniently the form of the equation is linear so that, assuming not all of the variables are unknowns, you could apply linear programming techniques to come with fancy “optimal” productivity strategies.
• Since the number of hours in a day is fixed, and we can arguably only give our full attention to a fraction of these hours, we should try to improve productivity by reducing either the number of hours or the amount of attention that a task requires. Some strategies let you do both things at once. For example, co-authoring a paper means you can delegate some of the work to someone else; you only need to invest a reduced amount of attention-hours to manage the project but you’ll receive similar amounts of prestige. Case in point: the world’s best connected mathematician.
• For those tasks that you have to do yourself, use your time wisely. This means balancing the levels of attention required by different tasks so you don’t burn out and focusing on those that earn the most prestige. This is why so much of what we write about here is concerned with time management, especially those tools that help us finish necessary but unrewarding administrata. See this post in particular.

What about actually implementing the model?

• Fitting the model to data and standardizing the coefficients for a large population will be difficult. As covered in this interview with Dan Navarro, there are “some big individual differences” in how people work; we shouldn’t put too much hope on one model holding for everyone.
• The interview also touches on another related issue: uncertainty. A strict optimization strategy is probably impossible because you don’t know the prestige associated with a task a priori. As Dan says, “it’s striking the right balance between exploration [of new ideas] and exploitation [of existing work]“.
• A related implementation issue is the problem of timing when collecting productivity data. This is two-fold: 1) when does the prestige arrive relative to the task being completed and 2) what is the temporal resolution of the feedback? One task may yield many different types of “prestige”: a small immediate personal satisfaction for completing a paper, a medium term recognition as it is published in a journal and cited, and maybe 50 years later receiving the Nobel Prize for your work. More from the archives on this here.
• I’ve said nothing about the psychology of productivity. It seems to me there should be feedback and “versus expectation” terms in all of this. Our earlier posts on learning theory and managing large projects (1,2,3,4,5) discuss the need to set clear goals, evaluate your performance against targets, and to learn from these experiences, changing your habits for next time. Such evaluations are key to maintaining your motivation and productivity.
• I’ve also said nothing about the physiology of productivity. What is the role of diet, daylight hours and half a dozen other factors?

So there you go. My two cents on what all of this productivity stuff really boils down to. I’m curious to see what you the readers think. Is this a crazy idea? Should we be trying to model productivity in such a formal way? Does anyone have the appetite for a community effort to gather some data and test the theory out?

Edit: There’s a follow-up post available with data testing this model.

I always wondered how people see the academic world from outside. How do we gauge the interest of the general public on what academics have to say (on average)? One easy way to look at this question is to see the how often people will read an article that has the word ‘academic’ on it.

A proxy on what people read nowadays is digg.com. And the tool to see how often people digg academic posts is now available in Dan Zarella’s blog. Given a keyword, the tool will return data on the average number of links accumulated by stories popular on Digg that mentioned that keyword. This is done with 2007 data.

Well, behold what happens when you enter “academic”:

And compare it to what you get when you type “productivity”:

Why is this important? Well, on average, a single digg increases traffic by 0.10%. So a story that gets 3,000 diggs results in an increase in total traffic to the referring site by 300%.

So, from now on we are a^H^H^H^H^H^H^H^H productivity blog

MINDLESS SIGNIFICANCE TESTING

Decision science news has a post on hypothesis testing that I find relevant.

Some well-made points grow old while no one pays attention to them. One of the most embarrassing for social science is its categorical perception of p-values.

Tender of kindred Web site Andrew Gelman and Hal Stern have an article whose name says it all: The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant.

Link to The Difference Between Significant and Not Significant is Not Statistically Significant