An excellent article published last month in the Chronicle celebrates Wikipedia’s 10th anniversary by observing that today the project doesn’t represent “the bottom layer of authority, nor the top, but in fact the highest layer without formal vetting” and, as such, it can serve as “an ideal bridge between the validated and unvalidated Web”. An increasing number of university students use Wikipedia for “pre-research”, as part of their course assignments or research projects. Yet many among academics, scientists and experts turn their noses up at the thought of contributing to Wikipedia, despite a growing number of calls from the scientific community to join the project (see for instance this recent initiative of the Association for Psychological Science or this call for biomedical experts to help contribute rigorous public health information in Wikipedia).

A survey has been launched by the Wikimedia Research Committee to understand why scientists, academics and other experts do (or do not) contribute to Wikipedia, and whether individual motivation aligns with shared perceptions of Wikipedia within different communities of experts. The survey is anonymous and takes about 20 min to complete. Whether you are an active Wikipedia contributor or not, you can take the survey and help Wikipedia think of ways around barriers to expert participation.

Readers of this blog are not new to my ramblings on soft peer review, social metrics and post-publication impact measures:

• can we measure the impact of scientific research based on usage data from collaborative annotation systems, social bookmarking services and social media?
• should we expect major discrepancies between citation-based and readership-based impact measures?
• are online reference management systems more robust a data source to measure scholarly readership than traditional usage factors (e.g. downloads, clickthrough rates etc.)?

These are some of the questions addressed in my COOP ’08 paper. Jason Priem also discusses the prospects of what he calls “scientometrics 2.0″ in a recent First Monday article and it is really exciting to see a growing interest in these ideas from both the scientific and the STM publishing community.

We now need to think of ways of putting these ideas into practice. Science Online London 2010 earlier this month offered a great chance to test a real-world application of these ideas in front of a tech-friendly audience and this post is meant as its official announcement.

ReaderMeter is a proof-of-concept application showcasing the potential of readership data obtained from reference management tools. Following the announcement of the Mendeley API, I decided to see what could be built on top of the data exposed by Mendeley and the first idea was to write a mashup aggregating author-level readership statistics based on the number of bookmarks scored by each of one’s publications. ReaderMeter queries the data provider’s API for articles matching a given author string. It parses the response and generates a report with several metrics that attempt to quantify the relative impact of an author’s scientific production based on its consumption by a population of readers (in this case the 500K-strong Mendeley user base):

The figure above shows a screenshot of ReaderMeter’s results for social scientist Duncan J Watts, displaying global bookmark statistics, the breakdown of readers by publication as well as two indices (the H_R index and the G_R index) which I compute using bookmarks as a variable by analogy to the two popular citation-based metrics. Clicking on a reference allows you to drill down to display readership statistics for a given publication, including the scientific discipline, academic status and geographic location of readers of an individual document:

A handy permanent URL is generated to link to ReaderMeter’s author reports (using the scheme: [SURNAME].[FORENAME+INITIALS]), e.g.:

http://readermeter.org/Watts.Duncan_J

I also included a JSON interface to render statistics in a machine-readable format, e.g.:

http://readermeter.org/Watts.Duncan_J/json

Below is a sample of the JSON output:

{
	"author": "Duncan J Watts",
	"author_metrics":
	{
		"hr_index": "15",
		"gr_index": "26",
		"single_most_read": "140",
		"publication_count": "57",
		"bookmark_count": "760",
		"data_source": "mendeley"
	},
	"source": "http://readermeter.org/Watts.Duncan_J",
	"timestamp": "2010-09-02T15:41:08+01:00"
}

Despite being just a proof of concept (it was hacked in a couple of nights!), ReaderMeter attracted a number of early testers who gave a try to its first release. Its goal is not to redefine the concept of research impact as we know it, but to complement this notion with usage data from new sources and help identify aspects of impact that may go unnoticed when we only focus on traditional, citation-based metrics. Before a mature version of ReaderMeter is available for public consumption and for integration with other services, though, several issues will need to be addressed.

1. Author name normalisation

The first issue to be tackled is the fact the same individual author may be mentioned in a bibliographic record under a variety of spelling alternates: Rod Page was among the first to spot and extensively discuss this issue, which will hopefully be addressed in the next major upgrade (unless a provision to fix this problem is directly offered by Mendeley in a future upgrade of their API).

2. Article deduplication

A similar issue affects individual bibliographic entries, as noted by Egon Willighagen among others. Given that publication metadata in reference management services can be extracted by a variety of sources, the uniqueness of a bibliographic record is far from given. As a matter of fact, several instances of the same publication can show up as distinct items, with the result of generating flawed statistics when individual publications and their relative impact need to be considered (as is the case when calculating the H- and G-index). To what extent crowdsourced bibliographic databases (such as those of Mendeley, CiteULike, Zotero, Connotea, and similar distributed reference management tools) can tackle the problem of article duplication as effectively as manually curated bibliographic databases, is an interesting issue that sparked a heated debate (see this post by Duncan Hull and the ensuing discussion).

3. Author disambiguation

A way more challenging problem consists in disambiguating real homonyms. At the moment, ReaderMeter is unable to tell the difference between two authors with an identical name. Considering that surnames like Wang appear to be shared by about 100M people on the planet, the problem of how to disambiguate authors with a common surname is not something that can be easily sorted out by a consumer service such as ReaderMeter. Global initiatives with a broad institutional support such as the ORCID project are trying to fix this problem for good by introducing a unique author identifier system, but precisely because of their scale and ambitious goal they are unlikely to provide a viable solution in the short run.

4. Reader segmentation and selection biases

You may wonder: how genuine is data extracted from Mendeley as an indicator of an author’s actual readership? Calculating author impact metrics based on the user population of a specific service will always by definition result in skewed results due to different adoption rates by different scientific communities or demographic segments (e.g. by academic status, language, gender) within the same community. And how about readers who just don’t use any reference management tools? Björn Brembs posted some thoughtful considerations on why any such attempt at measuring impact based on the specific user population of a given platform/service is doomed to fail. His proposed solution, however – a universal outlet where all scientific content consumption should happen–sounds not only like an unlikely scenario, but also in many ways an undesirable one. Diversity is one of the key features of the open source ecosystem, for one, and as long as interoperability is achieved (witness the example of the OAI protocol and its multiple software implementation), there is certainly no need for a single service to monopolise the research community’s attention for projects such as ReaderMeter to be realistically implemented. The next step on ReaderMeter’s roadmap will be to integrate data from a variety of content providers (such as CiteULike or Bibsonomy) that provide free access to article readership information: although not the ultimate solution to the enormous problem of user segmentation, data integration from multiple sources should hopefully help reduce biases introduced by the population of a specific service.

What’s next

I will be working in the coming days on an upgrade to address some of the most urgent issues, in the meantime feel free to test ReaderMeter, send me your feedback and feature requests, follow the latest news on the project via Twitter or just help spread the word!

There is only a bunch of tickets left for one of the most exciting annual events in the area of ICT for science. Hosted by Mendeley, Nature and the British Library, the second edition of Science Online London (3-4 September 2010) promises to bring together hackers, academics, publishers and startups in the field of software/services for scientists to discuss “how the Web is changing the way we conduct, communicate, share, and evaluate research”. I will be attending and would love to meet other AcaProd readers there.

Peter Sefton wrote a series of posts on wave. He has published on Scholarly HTML so I read attentively what he has to say. What follows is some highlights of his posts, and my thinking about where things are going. There are at least four things that bother me about wave –as it is today:

1- It’s not really HTML

I thought that waves being XML documents would be a good thing because it’d separate content and formatting. But it seems that they made some strange decisions about how to represent formatting with “very tenuous relationship to HTML”. For example

While there is talk of ‘XML documents’ in the whitepapers etc, a wave document in the current implementation is apparently a series of lines of text. All formatting and what you might think of as structure, such as whether something is a heading or not, is considered an annotation.

This is important right now because the only way to get the resulting doc is to dump the html to a file, or ‘copy-paste’. So, in a way, losing formatting like this would completely incapacitate wave for serious paper writing. I have had some success just copy-pasting and keeping most formatting, but I cannot risk to write a long paper and lose all formatting. Which won’t happen, because…

2- It doesn’t work too well for long, structured documents

Having a large blip for the entire paper with many people editing it seems to perform poorly. And having each person write their own blip-per-paragraph is not very pretty. It’s in fact distracting. I don’t discard wave’s usability to go through the roof once people start making things with it (robots). But it all depends on how well the API is designed. For example, following the mailing list, it seems that there’s no easy way to reorder blips programatically. This sounds like bad design to me.

3- It doesn’t integrate well with citation tools

It may be a matter of time before all the bibliographic tools we like get integrated. For example, Igor does offer some basic integration but this is far from satisfactory.

4-Formatting is simplistic

Wave has no table support. Figures are also not what you would expect, even for a notetaker. Captions are not implemented, nor footnotes. finally, LaTeXy is not the most convenient way to get equations done, I’m afraid, and it doesn’t go both ways.

Clearly, the content/form separation in wave is not designed for academic collaboration, and it shows. The questions is whether we can make this happen by writing robots[1]. Whether wave is the open platform that would make academic writing 2.0 happen.

Wave is just a tool. Why does this matter so much?

You may think that thinking too much about tools is counterproductive. But the way things are, it looks like tools are more and more important. Right now, we are stuck with the paper metaphor. Authors can produce pdfs, and publishers too. A publisher may make a prettier one, but that adds little value. We are equaled in terms of tools. However, publishers such as Elsevier want to get away from the paper metaphor (which is a good thing). As a consequence, authors will not be able to produce HTML as rich as the publisher’s. Here, the difference in tools matter.

In this case, Wave does look like an easy way to craft an interactive experience with little effort. So, even if you discard wave’s usefulness as a collaboration tool, it has quite a lot of value.

But it could very well be that wave doesn’t fulfill its promise. Microsoft Office 2010 offers similar functionality (close to real time edits). And of course, word has unparalleled features such as track changes, integration with endNote, etc. It could be that people adopt this new way of collaborative writing in real time without using wave. What worries me is that openOffice looks seriously left behind now. If it looks like a half-assed implementation of word 2003 features now, imagine when real-time hits mainstream. You need a serious server infrastructure to support that, which is possible for Google or MS, but not for a –smallish- open source foundation. I hope they find a way to jump in the train before it’s too late. Wave has a lot potential, because it is open. If openOffice could support that wave protocol, it could be a big achievement for open source.

If you have had any experience drafting a long doc in wave, please post it in the comments.

[1] For more on this, see my Wave intro for RuPy 2009.

Call for participation for a workshop at CSCW 2010
[submission deadline: November 20, 2009]

The confluence of two major trends in scientific research is leading to an upheaval in standard scientific practice and collaborative technologies. A new generation of scientists, working in large-scale collaborations, is repurposing social software for use in collaborative science. Existing social tools such as chat, IM, and FriendFind are being adopted and modified for use as group problem-solving facilities. At the same time, exponentially greater and more complex datasets are being generated at a rate that is challenging the limits of current hardware, software, and human cognitive capability. A concerted effort to create software that will support new scientific practices and handle this data tsunami is redefining the collaboratory and represents a new frontier for computer supported cooperative work.

This follow-on event to a similarly themed workshop at CHI 2009 is intended to foster community among researchers and practitioners from multiple disciplines interested in the changing dynamics of scientific collaborations.

We encourage papers on the following topics, especially those with a focus on changing practices in these areas:

• Collaborative scientific applications concerning data gathering,
  analysis, sharing, and visualization
• Case studies concerning data gathering, analysis, sharing and visualization
• Socio-technical research on scientific collaborations
• Social networks of scientists
• Repurposing social software for science
• Participatory design and/or rapid prototyping for scientific software
• Distributed data gathering and analysis
• Time-critical scientific applications
• Studies of generational differences in how science is done
• Cross-functional applications and comparisons of a scientific to
  a non-scientific field

For more information, please see the workshop web site:
http://www.sci.utah.edu/cscw2010/

Organizers:

• Cecilia Aragon, Lawrence Berkeley National Lab, CRAragon@lbl.gov
• Jeffrey Heer, Stanford University, jheer@cs.stanford.edu
• Charlotte Lee, University of Washington, cplee@u.washington.edu
• Claudio Silva, University of Utah, csilva@sci.utah.edu

My colleague Alastair is conducting a survey about online academic collaboration, use of tools and attitudes to technology in the Academia as part of the Qlectives project. All participants who supply an email address (and complete the questionnaire by the 14 November) will be entered into a prize draw.

The Qlectives project based at the University of Surrey is conducting a survey to investigate how scientists collaborate and use technology. The questionnaire should take around 20 minutes to complete and is being conducted anonymously and confidentially.

As a way of saying thank you to those who take part, we will hold a prize draw for 5 x £20 Amazon vouchers when we have finished
collecting data (we anticipate that this will be around the 14 November 2009). To be considered in the prize draw, simply leave your
email address on the final page of the questionnaire, and we will contact the lucky winners.

A more detailed description of the Qlectives project and this study can be found on the first page of the questionnaire. Please also feel
free to share the study with other colleagues who you think would be interested in taking part.

CC-licensed image courtesy of Enrique Burgos Garcia

Looks like the Connotea team is on the right track. Instead of trying to bolt something to insert references into word, they are trying to go straight to wave.

We have blogged before about what a good integration between references and writing tools should look like, and quite honestly, Igor looks like it’s really getting it in terms of agility. You can specify a few terms and it disambiguates that into the reference you need. Looks smarter than the approach that endnote/bibTeX/zotero/Mendeley use. It only works for the online reference managers citeUlike and Connotea, though.

I’m not sure the references are portable, i.e., if I copy/paste a chunk of text with references, they come along to wherever I paste it to (it must be another wave, in this case). Endnote/bibTeX get this right, although they depend on a local file that you would have to send along.

As things stand, I think wave has a very good chance of becoming _the_ platform for collaborative scientific writing. You may have to convince your collaborators to try it (and some must have been put off by Google Docs, which is clearly not ready for science), but it could be very motivating to see their writing grow next to yours in real time.

Since wave is a lot more open than Google Docs it would not surprise me to see robots coming up to mend the deficiencies that make Docs unfit for papers: no tables, crossrefs, footnotes, equations, etc. Wave gives you versioning for free, which was another pain point of scientific collaboration.

Igor – a Google Wave robot to manage your references from Stew Fnl on Vimeo.

I’m sure everyone here is familiar with drafting. It’s a very  demanding activity, and my feeling is that there are no proper tools out there. Word is clearly not a good drafting tool, and raw latex is not much better. I particularly use onenote, but it’s not without its problems (I work under linux too, and there the closest solution I could find is to use rightnote under wine).

Surprisingly enough, the Office team has a website to request feedback, and they seem to use it (!). This is a way to talk to the developers directly, something I missed when using microsoft products for a long time. So if you have a pet peeve, go post it there. Here’s mine:

In long docs view only a section at a time

Navigating a long doc is awful for drafting. The toilet paper metaphor doesn’t work, human working memory cannot keep track of location of ideas that way. Onenote shows a much better metaphor, where one idea/section is its own tab. This idea agrees with the programming maxim "a function should use a screen at the most. If it doesn’t fit a screen, it’s too long". writing text is not programming, but it’s close: many ideas organized in a logical way, with dependencies.

So the proposal: In long docs, view only a section at a time. This could be draft mode, or a checkbox for any mode.

Btw, outline mode is not what I mean. Apart from being ugly as hell, it shows all other sections folded. I mean a completely crear screen with just the section you are working on.

It sorta can be done now, by using a master document and making each section a subdocument… but it’s not very agile. I rearrange sections a lot. Having each subdoc as another window separately is confusing.

I’m curious to hear what your tricks are for drafting. I’m surprised that things like onenote don’t get more attention in this community. They do take quite a lot of mental effort out from writing for me.

Just a quick one to highlight The False Prophet. S/he presents a list of reasons why things don’t get done, together with Preventive measures and solutions.

This is quite a finding. Example:

4. Distraction (…)

Preventive measures
When you commit to a project, set a daily/weekly schedule. Consistent time-structure is what gets projects done that last longer than the excitement of inspiration. Working on the project first thing in the morning is one way to head off distraction.
Set up your own status meetings, just as with “dependency”, since, in a way, you are depending on yourself to deliver. Having a regular status meeting with a friend (say, in a phone call) can keep you on track.
See “attention overload”, below.
Solutions (to be used after you’ve gotten distracted)
The 5M method. Note the time. Think of a tiny task that you think you can complete in five minutes. Give yourself an hour to do it. You can do the task right now, and then slack off. You can wait until five minutes are left in the hour and then rush. Odds are, once you do this tiny task, you’ll feel different. You’ll have some momentum. Your brain will now be returning to the work you want to do instead of the distraction. Or perhaps not. You can still slack off the rest of the hour if you prefer. A deal’s a deal.

He also has an interesting post on ‘how to fix grad school’. I like the way he presents arguments, and then tries to shut them down.

False Epiphany» Blog Archive » Incompletion: 15 Causes and Solutions

It should be no surprise that many of us love Zotero, especially since they added support for reference sharing and synchronization.

I am probably the only exception in the AP team. As a longstanding MacTeX user, I keep my references organised with BibDesk, one of the sweetest pieces of (open source) software ever written for TeX users working on Mac OS. When hunting for references, I use CiteULike as a fast and effective solution to bookmark and tag papers. My workflow usually starts with an exploratory phase based on CiteULike. As soon as I have read a paper and need to cite it, I export its reference from CiteULike into BibDesk, filing the PDFs with the help of the autofile functionality in BibDesk. So far I have been quite happy with this workflow even if it involves a little bit of fiddling to correctly import references into my local library.

After some research and discussion in the support forums, I discovered that CiteULike and BibDesk can seamlessly integrate with each other. BibDesk has an external file group option that allows you to read structured references from any online source. CiteULike has a bibtex export feature that allows you to expose your references for consumption by software like BibDesk. This is all you need to get the two to work together. Here’s how my new workflow looks like:

1. Create a new external file group in BibDesk

Click on the cog icon on the bottom left of the BibDesk window and select “Add External File Group”. This will add a new folder in your library pointing to a remote source.

2. Point the new group folder to CiteULike

CiteULike can generate bibtex files on-the-fly from any reference list, e.g. your reference library, the list of papers for a specific author, papers tagged with a specific keyword etc. The bibtex export filter for any of these lists can be obtained by adding the /bibtex prefix to the corresponding URL. For instance, to export in bibtex format my full reference library:

http://citeulike.org/user/dartar/

I simply need to type the following URL:

http://citeulike.org/bibtex/user/dartar/

The following URL parameters are a useful addition:

• fieldmap=<oldval>:<newval> allows you to specify translation rules for custom fields, e.g. record creation date (what CiteULike calls Posted-At must be translated to BibDesk’s Date-Added)
• do_username_prefix=0&key_type=4 toggles an option to create bibtex keys using an AuthorYearTitle scheme when none is available (which I prefer to the default numeric key)

The final URL we will use is the following:

http://citeulike.org/bibtex/user/dartar?fieldmap=posted-at:date-added&do_username_prefix=0&key_type=4

3. Browse your CiteULike library from BibDesk

Once you have pointed BibDesk to the URL of the CiteULike bibtex filter, you can start directly browsing your CiteULike library from BibDesk. You will see an “Import” button on the left of each item, which is greyed out when the reference is already imported in your local BibDesk library.

4. Import selected items

Click on “Import” to download a reference and file it in your local BibDesk library: you are done!

There is definitely room for further improvement (in particular to allow bidirectional sync’ing of references and PDFs and custom bibtex keys in CiteULike), but this solution will save you a lot of time when working with references between a local and a remote library. Kudos to the BibDesk and CiteULike team for being so responsive to feature requests from their user community!

On a related note, if you are a Mendeley user, you will be happy to learn that today their integration with CiteULike is finally seeing the light.