This excellent post covers why academic publishing is obsolete. TL;DR:

1. The time lag is huge; it’s measured in months, or even years.

2. Most academic publications are inaccessible outside universities.

3. Virtually no one reads most academic publications.

4. It’s very unusual to make successful philosophical arguments in paper form.

5. Papers don’t have prestige outside a narrow subset of society.

6. Getting people to read papers is difficult.

7. Academia selects for conformity.

8. Papers have a tradition of violating the bottom line rule.

9. Academic moderation is both very strict and badly run.

Of course, one could argue that Academic papers today are not meant to be discussion forums. Still, blogging, continuous deployment, etc. make academic publishing feel archaic.

I’m writing something on why github could be a good model, I’ll publish it soon.

Most academics really want to improve their writing, because good writing increases your chances of getting your manuscript accepted. Still, I think the academic system doesn´t reward good writing as much as it should.

The best advice I´ve seen recently on how to become a better writer is this quora post by Venkatesh Rao. He pins any quality improvements to rewriting:

The HUGE difference between everyday writing that everybody does and serious writing is the proportion that is re-writing. I’d estimate that for non-writers, rewriting accounts for maybe 10-20% of their writing. For serious writers, it accounts for anywhere between 50-90% depending on how critical the particular piece is.

I´m not sure how much rewriting goes into your average paper, but those numbers ring true for me. This makes things simpler: if you want to improve as a writer, focus on your rewriting skill. This is solid advice. Now I understand why doing revisions of a paper feels so tiresome: it’s not necessarily boredom, it’s the fact that rewriting is more effortful than actual writing.

The explanation below may only make sense to you if you read this from an RSS reader. If you don’t please skip it.

Have you ever sent an email to the wrong person? Did wish you could pull it back? I just did this, but for our blog (!).

I was feeding a WP install of what will be our company blog, and then I did something nasty…

I was about to post from my desktop tool (word 2010, used to be live writer), and I hit ‘post’ on the wrong doc, and ‘about’ page, ‘google-translated’ from German; prose as horrible as one can get. The blog selected was academicproductivity.com, not our own. I immediately went to the admin page and removed it, so ap.com’s readers won’t see it. But the RSS feed… is another story. For a blog that was dead for a year, we still have >4000 followers. This was the first post to break a long silent stretch. I’d hate if you, the reader, think we resurrected, just to find out a nonsensical blog post.

Since we use feedburner, removing the post locally didn’t help. I had to log in at feedburner, and try to remove it from there. They have a ‘nuke’ option that should force a refresh. But it just didn’t work. I tried a few times, the nonsensical blog post was still there. The only option I could think of was to delete the feed from feedburner, in the hope that they do not broadcast it. But it was too late; all people who subscribe to ap.com’s RSS feed have received the post.

I apologize for kidnapping your attention without a good reason. I’m sorry you got involved in this, but silly mistakes do occur. I will be more careful in the future.

Since the feedburner feed is now gone, if you want to continue receiving updates from ap.com you’d need to resubscribe. Simply click again on the RSS icon on the address bar, and follow the steps there. In any case, we are not dead, and will continue writing for ap.com whenever we find something worth writing about.

Scott Aaronson uses an analogy to the game industry to describe the predicament academics are in:

I have an ingenious idea for a company. My company will be in the business of selling computer games.

But, unlike other computer game companies, mine will never have to hire a single programmer, game designer, or graphic artist. Instead I’ll simply find people who know how to make games, and ask them to donate their games to me. Naturally, anyone generous enough to donate a game will immediately relinquish all further rights to it. From then on, I alone will be the copyright-holder, distributor, and collector of royalties.

This is not to say, however, that I’ll provide no “value-added.” My company will be the one that packages the games in 25-cent cardboard boxes, then resells the boxes for up to $300 apiece.

But why would developers donate their games to me? Because they’ll need my seal of approval. I’ll convince developers that, if a game isn’t distributed by my company, then the game doesn’t “count”—indeed, barely even exists—and all their labor on it has been in vain.

As crazy as it sounds, this is exactly the situation with academic publishers. The ‘status quo’ is such that young researchers must publish on established journals (to gain the “seal of approval”). For older researchers, switching to open access publishing doesn’t pay off either: it’d show they don’t believe in the value the journals bring, and they are often editors of those (!).

And this is how the current academic publishing industry survives without adding much value. Survival is not the right word, because the leading firms still carry themselves around with arrogance. At the 2010 Semantic Web conference in Shanghai Jay Katzen, a keynote speaker from Elsevier, announced a big project on using the data on papers to create widgets. The API would allow people to do mashups with scientific data, that could be displayed on the publisher’s page. It was sold as “a new paradigm in the way research information is discovered, used, shared and re-used to accelerate science.” The reaction from the audience was instantaneous: “are you telling us that, not happy with monetizing the data and content we freely give you, you want us to build applications using that content for you to sell?”. The answer was honest: “… huh… yes.”

Today, many journal articles are online. In fact, the papers are often on the author’s homepage, and a simple query on google scholar or MS research search will find them. It is hard to imagine what value a publisher adds here.

However, the alternative is not clear. Open access publishing finds it difficult to obtain sustainable sources of financing. PLoS, the Public Library of Science, is financially sustainable, but ArXiv is struggling.

“Now it’s up to the rest of us to supply the anger.” Says Scott. Now more than 800 researchers have declared a boycott against Elsevier, up from 500 yesterday afternoon. Looks like the anger is there.

(An apology for the lack of posting. Dario has moved on to a position as senior researcher at Wikimedia, and I will be working on my startup full-time in a month. Often, I’ve seen blogpost-worthy issues, but I just didn’t have the mental bandwidth to follow up).

I read a good quote the other day:

“Bollocks to waiting 10 years for progress. I want people to know about it now, and then do something about it” – Dr Paul Fisher

So why do we wait? Why isn’t there immediate publication, analysis and dissemination of data? Publication of Scientific data as it stands is a broken business model…for the most part. The advent of journals like PLoS and their subsequent success shows that the scientific community is taking note of what steps need to be taken. In my short life as a scientist, there has always been one thing that really annoys me. The inefficiency of scientific publishing and subsequent global sharing of knowledge. In terms of making significant advances available to wide audiences as peer reviewed publications, PLoS has it covered. But what about the rest of your research?

What percentage of the figures that went into your undergrad, masters or doctorate thesis were ever published? The ones that you didnt publish were probably good basic science, or figures that didnt tell a complete story. As a PhD student, I became very aware of the fact that a large amount of my data, although good, would never be published as it did not show significant differences. I then began wondering how many times experiments had been repeated globally unnecessarily. And so FigShare started life as an idea for researchers to publish all of their data that would otherwise never leave their lab books. By categorising and tagging the research, it becomes very searchable and other scientists should not reproduce experiments and waste money when they have been conducted several times by other labs. Following the alpha release, FigShare received a lot of attention and a lot of feedback. This caused the site to develop and it now allows the upload of Figures, Datasets and most recently media (eg. videos).

This is not a new idea, and big data and data sharing projects have won several big JISC grants, but your average researcher needs this to be simple in order to adopt. The CKAN repository is a fantastic project which allows you to upload data from any field, such as government finances, weather forecasts and traffic reports. Where I feel this becomes inaccessible for scientists is the ease of uploads. A choice from 50 licenses for your data is intimidating enough to make most postdocs turn and run. For a project like FigShare, the more research that is uploaded, the more useful the site becomes. In order for this to happen, uploading research needs to be simple. This is what FigShare gives you. Give your bit of research, be it a figure, dataset or some other media format a name, hit upload, add details like your name, some tags and you have a nicely presented, citable, published figure.

FigShare now also serves as a repository for preprints figures. ie. Figures that will one day be published but feedback is requested on the prelimonary data. FigShare can be used as a platform to collaborate where users can contact one another and request to use figures in their publications etc. This means that previous unused figures, maybe from unfinished postdocs or PhD projects can be published, gaining the author more publications.

There is also the ability to easily share your figures, datasets and videos via a host of social media platforms through ‘share buttons’ on every page. This is a new way of bringing scientific research online and to a new audience. An example of how this can benefit science is already producing examples such as this one – A lot of scientists hear how social media can benefit research and yet there has been little evidence of how these tools can be exploited to make science more efficient. Imagine real time discussions about science you did yesterday, not last year when you first submitted your paper.

FigShare is a permanent database of your research. To further ensure this, FigShare is supported by Systems Institute. Systems Institute is a not for profit which is providing ongoing support for the hosting of FigShare as it expands. This also allows FigShare to make backups of all of your data each and every day.

So please, upload your data now and do your bit to help science progress in an efficient manner. It’ll probably do wonders for your academic career too!

Koblenz (Germany), 14-15 June 2011
An ACM Web Science Conference 2011 Workshop

Call for papers

The increasing quantity and velocity of scientific output is presenting scholars with a deluge of data. There is growing concern that scholarly output may be swamping traditional mechanisms for both pre-publication filtering (e.g peer review) and post-publication impact filtering (e.g. the Journal Impact Factor).

Increasing scholarly use of Web2.0 tools like CiteULike, Mendeley, Twitter, and blog-style article commenting presents an opportunity to create new filters. Metrics based on a diverse set of social sources could yield broader, richer, and more timely assessments of current and potential scholarly impact. Realizing this, many authors have begun to call for investigation of these “altmetrics.” (see altmetrics.org)

Despite the growing speculation and early exploratory investigation into the value of altmetrics, however, there remains little concrete, objective research into the properties of these metrics: their validity, their potential value and flaws, and their relationship to established measures. Nor has there been any large umbrella to bring these multiple perspectives together. The altmetrics11 workshop aims to  encourage both these. Submissions are invited from a variety of areas:

• New metrics based on social media
• Tracking science communication on the Web
• Relation between traditional metrics and altmetrics
• Peer-review and altmetrics
• Tools for gathering, analyzing, disseminating altmetrics

Important Dates

Submissions

Prospective authors should submit 2-page extended abstracts (max. 1000 words, not including references). If necessary, the workshop organizers will select the most relevant, original, and significant abstracts for presentation. Experimental results will be given preference, followed by technical reports on working altmetrics tools and position papers. All selected submissions will be published online for open peer review and discussion. Authors are encouraged to participate in the discussions of their work. Based on the presentations and online discussion, selected authors may be asked to submit full papers for peer-reviewed proceedings.

Location

The workshop is hosted by the ACM Web Science Conference 2011 (Koblenz, Germany). This interdisciplinary conference focuses on advances in studying the full range of social-technical relationships on the Web. Please visit the Web Science site for more information.

Organizers

• Paul Groth – VU University Amsterdam, NL
• Jason Priem –University of North Carolina at Chapel Hill, USA
• Dario Taraborelli – Wikimedia Foundation, USA

The organizers have an interdisciplinary background covering Sociology, Information and Library Science and Computer Science.

An excellent article published last month in the Chronicle celebrates Wikipedia’s 10th anniversary by observing that today the project doesn’t represent “the bottom layer of authority, nor the top, but in fact the highest layer without formal vetting” and, as such, it can serve as “an ideal bridge between the validated and unvalidated Web”. An increasing number of university students use Wikipedia for “pre-research”, as part of their course assignments or research projects. Yet many among academics, scientists and experts turn their noses up at the thought of contributing to Wikipedia, despite a growing number of calls from the scientific community to join the project (see for instance this recent initiative of the Association for Psychological Science or this call for biomedical experts to help contribute rigorous public health information in Wikipedia).

A survey has been launched by the Wikimedia Research Committee to understand why scientists, academics and other experts do (or do not) contribute to Wikipedia, and whether individual motivation aligns with shared perceptions of Wikipedia within different communities of experts. The survey is anonymous and takes about 20 min to complete. Whether you are an active Wikipedia contributor or not, you can take the survey and help Wikipedia think of ways around barriers to expert participation.

No one can read everything. We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped. However, the growth of new, online scholarly tools allows us to make new filters; these alt-metrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem. We call for more tools and research based on alt-metrics.

As the volume of academic literature explodes, scholars rely on filters to select the most relevant and significant sources from the rest.

Unfortunately, scholarship’s three main filters for importance are failing:

• Peer-review has served scholarship well, but is beginning to show its age. It is slow, encourages conventionality, and fails to hold reviewers accountable. Moreover, given that most papers are eventually published somewhere, peer-review fails to limit the volume of research.
• Citation counting measures are useful, but not sufficient. Metrics like the h-index are even slower than peer-review: a work’s first citation can take years. Citation measures are narrow; influential work may remain uncited. These metrics are narrow; they neglect impact outside the academy, and also ignore the context and reasons for citation.
• The JIF, which measures journals’ average citations per article, is often incorrectly used to assess the impact of individual articles. It’s troubling that the exact details of the JIF are a trade secret, and that significant gaming is relatively easy.

Tomorrow’s filters: alt-metrics

In growing numbers, scholars are moving their everyday work to the web. Online reference managers Zotero and Mendeley each claim to store over 40 million articles (making them substantially larger than PubMed); as many as a third of scholars are on Twitter, and a growing number tend scholarly blogs.

These new forms reflect and transmit scholarly impact: that dog-eared (but uncited) article that used to live on a shelf now lives in Mendeley, CiteULike, or Zotero–where we can see and count it. That hallway conversation about a recent finding has moved to blogs and social networks–now, we can listen in. The local genomics dataset has moved to an online repository–now, we can track it. This diverse group of activities forms a composite trace of impact far richer than any available before. We call the elements of this trace alt-metrics.

Alt-metrics expand our view of what impact looks like, but also of what’s making the impact. This matters because expressions of scholarship are becoming more diverse. Articles are increasingly joined by:

• The sharing of “raw science” like datasets, code, and experimental designs
• Semantic publishing or “nanopublication,” where the citeable unit is an argument or passage rather than entire article.
• Widespread self-publishing via blogging, microblogging, and comments or annotations on existing work.

Because alt-metrics are themselves diverse, they’re great for measuring impact in this diverse scholarly ecosystem. In fact, alt-metrics will be essential to sift these new forms, since they’re outside the scope of traditional filters. This diversity can also help in measuring the aggregate impact of the research enterprise itself.

Alt-metrics are fast, using public APIs to gather data in days or weeks. They’re open–not just the data, but the scripts and algorithms that collect and interpret it. Alt-metrics look beyond counting and emphasize semantic content like usernames, timestamps, and tags. Alt-metrics aren’t citations, nor are they webometrics; although these latter approaches are related to alt-metrics, they are relatively slow, unstructured, and closed.

How can alt-metrics improve existing filters?

With alt-metrics, we can crowdsource peer-review. Instead of waiting months for two opinions, an article’s impact might be assessed by thousands of conversations and bookmarks in a week. In the short term, this is likely to supplement traditional peer-review, perhaps augmenting rapid review in journals like PLoS ONE, BMC Research Notes, or BMJ Open. In the future, greater participation and better systems for identifying expert contributors may allow peer review to be performed entirely from alt-metrics. Unlike the JIF, alt-metrics reflect the impact of the article itself, not its venue. Unlike citation metrics, alt-metrics will track impact outside the academy, impact of influential but uncited work, and impact from sources that aren’t peer-reviewed. Some have suggested alt-metrics would be too easy to game; we argue the opposite. The JIF is appallingly open to manipulation; mature alt-metrics systems could be more robust, leveraging the diversity of of alt-metrics and statistical power of big data to algorithmically detect and correct for fraudulent activity. This approach already works for online advertisers, social news sites, Wikipedia, and search engines.

The speed of alt-metrics presents the opportunity to create real-time recommendation and collaborative filtering systems: instead of subscribing to dozens of tables-of-contents, a researcher could get a feed of this week’s most significant work in her field. This becomes especially powerful when combined with quick “alt-publications” like blogs or preprint servers, shrinking the communication cycle from years to weeks or days. Faster, broader impact metrics could also play a role in funding and promotion decisions.

Road map for alt-metrics

Speculation regarding alt-metrics (Taraborelli, 2008; Neylon and Wu, 2009; Priem and Hemminger, 2010) is beginning to yield to empirical investigation and working tools. Priem and Costello (2010) and Groth and Gurney (2010) find citation on Twitter and blogs respectively. ReaderMeter computes impact indicators from readership in reference management systems. Datacite promotes metrics for datasets. Future work must continue along these lines.

Researchers must ask if alt-metrics really reflect impact, or just empty buzz. Work should correlate between alt-metrics and existing measures, predict citations from alt-metrics, and compare alt-metrics with expert evaluation. Application designers should continue to build systems to display alt-metrics, develop methods to detect and repair gaming, and create metrics for use and reuse of data. Ultimately, our tools should use the rich semantic data from alt-metrics to ask “how and why?” as well as “how many?”

Alt-metrics are in their early stages; many questions are unanswered. But given the crisis facing existing filters and the rapid evolution of scholarly communication, the speed, richness, and breadth of alt-metrics make them worth investing in.

Jason Priem (University of North Carolina-Chapel Hill)
Dario Taraborelli (University of Surrey)
Paul Groth (VU University Amsterdam)
Cameron Neylon (Science and Technology Facilities Council)

Source: http://altmetrics.org/manifesto 

Readers of this blog are not new to my ramblings on soft peer review, social metrics and post-publication impact measures:

• can we measure the impact of scientific research based on usage data from collaborative annotation systems, social bookmarking services and social media?
• should we expect major discrepancies between citation-based and readership-based impact measures?
• are online reference management systems more robust a data source to measure scholarly readership than traditional usage factors (e.g. downloads, clickthrough rates etc.)?

These are some of the questions addressed in my COOP ’08 paper. Jason Priem also discusses the prospects of what he calls “scientometrics 2.0″ in a recent First Monday article and it is really exciting to see a growing interest in these ideas from both the scientific and the STM publishing community.

We now need to think of ways of putting these ideas into practice. Science Online London 2010 earlier this month offered a great chance to test a real-world application of these ideas in front of a tech-friendly audience and this post is meant as its official announcement.

ReaderMeter is a proof-of-concept application showcasing the potential of readership data obtained from reference management tools. Following the announcement of the Mendeley API, I decided to see what could be built on top of the data exposed by Mendeley and the first idea was to write a mashup aggregating author-level readership statistics based on the number of bookmarks scored by each of one’s publications. ReaderMeter queries the data provider’s API for articles matching a given author string. It parses the response and generates a report with several metrics that attempt to quantify the relative impact of an author’s scientific production based on its consumption by a population of readers (in this case the 500K-strong Mendeley user base):

The figure above shows a screenshot of ReaderMeter’s results for social scientist Duncan J Watts, displaying global bookmark statistics, the breakdown of readers by publication as well as two indices (the H_R index and the G_R index) which I compute using bookmarks as a variable by analogy to the two popular citation-based metrics. Clicking on a reference allows you to drill down to display readership statistics for a given publication, including the scientific discipline, academic status and geographic location of readers of an individual document:

A handy permanent URL is generated to link to ReaderMeter’s author reports (using the scheme: [SURNAME].[FORENAME+INITIALS]), e.g.:

http://readermeter.org/Watts.Duncan_J

I also included a JSON interface to render statistics in a machine-readable format, e.g.:

http://readermeter.org/Watts.Duncan_J/json

Below is a sample of the JSON output:

{
	"author": "Duncan J Watts",
	"author_metrics":
	{
		"hr_index": "15",
		"gr_index": "26",
		"single_most_read": "140",
		"publication_count": "57",
		"bookmark_count": "760",
		"data_source": "mendeley"
	},
	"source": "http://readermeter.org/Watts.Duncan_J",
	"timestamp": "2010-09-02T15:41:08+01:00"
}

Despite being just a proof of concept (it was hacked in a couple of nights!), ReaderMeter attracted a number of early testers who gave a try to its first release. Its goal is not to redefine the concept of research impact as we know it, but to complement this notion with usage data from new sources and help identify aspects of impact that may go unnoticed when we only focus on traditional, citation-based metrics. Before a mature version of ReaderMeter is available for public consumption and for integration with other services, though, several issues will need to be addressed.

1. Author name normalisation

The first issue to be tackled is the fact the same individual author may be mentioned in a bibliographic record under a variety of spelling alternates: Rod Page was among the first to spot and extensively discuss this issue, which will hopefully be addressed in the next major upgrade (unless a provision to fix this problem is directly offered by Mendeley in a future upgrade of their API).

2. Article deduplication

A similar issue affects individual bibliographic entries, as noted by Egon Willighagen among others. Given that publication metadata in reference management services can be extracted by a variety of sources, the uniqueness of a bibliographic record is far from given. As a matter of fact, several instances of the same publication can show up as distinct items, with the result of generating flawed statistics when individual publications and their relative impact need to be considered (as is the case when calculating the H- and G-index). To what extent crowdsourced bibliographic databases (such as those of Mendeley, CiteULike, Zotero, Connotea, and similar distributed reference management tools) can tackle the problem of article duplication as effectively as manually curated bibliographic databases, is an interesting issue that sparked a heated debate (see this post by Duncan Hull and the ensuing discussion).

3. Author disambiguation

A way more challenging problem consists in disambiguating real homonyms. At the moment, ReaderMeter is unable to tell the difference between two authors with an identical name. Considering that surnames like Wang appear to be shared by about 100M people on the planet, the problem of how to disambiguate authors with a common surname is not something that can be easily sorted out by a consumer service such as ReaderMeter. Global initiatives with a broad institutional support such as the ORCID project are trying to fix this problem for good by introducing a unique author identifier system, but precisely because of their scale and ambitious goal they are unlikely to provide a viable solution in the short run.

4. Reader segmentation and selection biases

You may wonder: how genuine is data extracted from Mendeley as an indicator of an author’s actual readership? Calculating author impact metrics based on the user population of a specific service will always by definition result in skewed results due to different adoption rates by different scientific communities or demographic segments (e.g. by academic status, language, gender) within the same community. And how about readers who just don’t use any reference management tools? Björn Brembs posted some thoughtful considerations on why any such attempt at measuring impact based on the specific user population of a given platform/service is doomed to fail. His proposed solution, however – a universal outlet where all scientific content consumption should happen–sounds not only like an unlikely scenario, but also in many ways an undesirable one. Diversity is one of the key features of the open source ecosystem, for one, and as long as interoperability is achieved (witness the example of the OAI protocol and its multiple software implementation), there is certainly no need for a single service to monopolise the research community’s attention for projects such as ReaderMeter to be realistically implemented. The next step on ReaderMeter’s roadmap will be to integrate data from a variety of content providers (such as CiteULike or Bibsonomy) that provide free access to article readership information: although not the ultimate solution to the enormous problem of user segmentation, data integration from multiple sources should hopefully help reduce biases introduced by the population of a specific service.

What’s next

I will be working in the coming days on an upgrade to address some of the most urgent issues, in the meantime feel free to test ReaderMeter, send me your feedback and feature requests, follow the latest news on the project via Twitter or just help spread the word!

The Undercover Economist, Tim Harford, has a good article in today’s Financial Times about the stages in life when different professions are most productive. For example, I did a quick Google/calculation: the average median age of a Nobel Prize winner in physics or chemistry is 55; in the literature and peace prizes, it’s 64. (Sorry, not going to do the full test for statistical difference today). This distinction makes some sense, as the great discoveries in the two scientific subjects are marked by innovation (something that may become replaced by habit with age) and excellence in literature and statesmanship benefits from vast amounts of experience.

But, in keeping with our recent discussions about reform in academia, perhaps the bigger question is whether or not we should be actively targeting funding to match these periods of productivity? A quote from the FT article:

Two of my favourite writers, Malcolm Gladwell and Jonah Lehrer, are worried about this – but from different perspectives. Gladwell, a Galenson fan, worries that our obsession with youthful genius will cause us to reject future late bloomers.

Lehrer has the opposite concern: that funding goes to scientists past their prime. He says the US’s National Institutes of Health (NIH) has been funding ever-older scientists. Thirty years ago, researchers in their early thirties used to receive 10 per cent of NIH grants; by 2006 the figure had fallen to 1 per cent.

From my experience in the UK, I think both groups have good, but different, funding opportunities. Established researchers are well-versed in applying for traditional call-based research grants, whereas young researchers are catered for by a number of fellowship schemes. I haven’t seen much evidence of disciplinary-based bias and to be honest, I think anti-discrimination laws would make it difficult to explicitly exclude a group of talented researchers just because they’ve reached an arbitrary age barrier. Think of Andrew Wiles, who found a proof of Fermat’s Last Theorem but just over the Fields Medal’s age limit of 40.

Ultimately the top performers in these disciplines are so unique that it doesn’t make sense to design generalized development or funding programmes for the rest of us. However we can at least take comfort that our best days may be ahead of us!