No one can read everything. We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped. However, the growth of new, online scholarly tools allows us to make new filters; these alt-metrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem. We call for more tools and research based on alt-metrics.

As the volume of academic literature explodes, scholars rely on filters to select the most relevant and significant sources from the rest.

Unfortunately, scholarship’s three main filters for importance are failing:

• Peer-review has served scholarship well, but is beginning to show its age. It is slow, encourages conventionality, and fails to hold reviewers accountable. Moreover, given that most papers are eventually published somewhere, peer-review fails to limit the volume of research.
• Citation counting measures are useful, but not sufficient. Metrics like the h-index are even slower than peer-review: a work’s first citation can take years. Citation measures are narrow; influential work may remain uncited. These metrics are narrow; they neglect impact outside the academy, and also ignore the context and reasons for citation.
• The JIF, which measures journals’ average citations per article, is often incorrectly used to assess the impact of individual articles. It’s troubling that the exact details of the JIF are a trade secret, and that significant gaming is relatively easy.

Tomorrow’s filters: alt-metrics

In growing numbers, scholars are moving their everyday work to the web. Online reference managers Zotero and Mendeley each claim to store over 40 million articles (making them substantially larger than PubMed); as many as a third of scholars are on Twitter, and a growing number tend scholarly blogs.

These new forms reflect and transmit scholarly impact: that dog-eared (but uncited) article that used to live on a shelf now lives in Mendeley, CiteULike, or Zotero–where we can see and count it. That hallway conversation about a recent finding has moved to blogs and social networks–now, we can listen in. The local genomics dataset has moved to an online repository–now, we can track it. This diverse group of activities forms a composite trace of impact far richer than any available before. We call the elements of this trace alt-metrics.

Alt-metrics expand our view of what impact looks like, but also of what’s making the impact. This matters because expressions of scholarship are becoming more diverse. Articles are increasingly joined by:

• The sharing of “raw science” like datasets, code, and experimental designs
• Semantic publishing or “nanopublication,” where the citeable unit is an argument or passage rather than entire article.
• Widespread self-publishing via blogging, microblogging, and comments or annotations on existing work.

Because alt-metrics are themselves diverse, they’re great for measuring impact in this diverse scholarly ecosystem. In fact, alt-metrics will be essential to sift these new forms, since they’re outside the scope of traditional filters. This diversity can also help in measuring the aggregate impact of the research enterprise itself.

Alt-metrics are fast, using public APIs to gather data in days or weeks. They’re open–not just the data, but the scripts and algorithms that collect and interpret it. Alt-metrics look beyond counting and emphasize semantic content like usernames, timestamps, and tags. Alt-metrics aren’t citations, nor are they webometrics; although these latter approaches are related to alt-metrics, they are relatively slow, unstructured, and closed.

How can alt-metrics improve existing filters?

With alt-metrics, we can crowdsource peer-review. Instead of waiting months for two opinions, an article’s impact might be assessed by thousands of conversations and bookmarks in a week. In the short term, this is likely to supplement traditional peer-review, perhaps augmenting rapid review in journals like PLoS ONE, BMC Research Notes, or BMJ Open. In the future, greater participation and better systems for identifying expert contributors may allow peer review to be performed entirely from alt-metrics. Unlike the JIF, alt-metrics reflect the impact of the article itself, not its venue. Unlike citation metrics, alt-metrics will track impact outside the academy, impact of influential but uncited work, and impact from sources that aren’t peer-reviewed. Some have suggested alt-metrics would be too easy to game; we argue the opposite. The JIF is appallingly open to manipulation; mature alt-metrics systems could be more robust, leveraging the diversity of of alt-metrics and statistical power of big data to algorithmically detect and correct for fraudulent activity. This approach already works for online advertisers, social news sites, Wikipedia, and search engines.

The speed of alt-metrics presents the opportunity to create real-time recommendation and collaborative filtering systems: instead of subscribing to dozens of tables-of-contents, a researcher could get a feed of this week’s most significant work in her field. This becomes especially powerful when combined with quick “alt-publications” like blogs or preprint servers, shrinking the communication cycle from years to weeks or days. Faster, broader impact metrics could also play a role in funding and promotion decisions.

Road map for alt-metrics

Speculation regarding alt-metrics (Taraborelli, 2008; Neylon and Wu, 2009; Priem and Hemminger, 2010) is beginning to yield to empirical investigation and working tools. Priem and Costello (2010) and Groth and Gurney (2010) find citation on Twitter and blogs respectively. ReaderMeter computes impact indicators from readership in reference management systems. Datacite promotes metrics for datasets. Future work must continue along these lines.

Researchers must ask if alt-metrics really reflect impact, or just empty buzz. Work should correlate between alt-metrics and existing measures, predict citations from alt-metrics, and compare alt-metrics with expert evaluation. Application designers should continue to build systems to display alt-metrics, develop methods to detect and repair gaming, and create metrics for use and reuse of data. Ultimately, our tools should use the rich semantic data from alt-metrics to ask “how and why?” as well as “how many?”

Alt-metrics are in their early stages; many questions are unanswered. But given the crisis facing existing filters and the rapid evolution of scholarly communication, the speed, richness, and breadth of alt-metrics make them worth investing in.

Jason Priem (University of North Carolina-Chapel Hill)
Dario Taraborelli (University of Surrey)
Paul Groth (VU University Amsterdam)
Cameron Neylon (Science and Technology Facilities Council)

Source: http://altmetrics.org/manifesto 

Cambridge zoologist Peter A. Lawrence has published a thoughtful piece on the frustration of scientists (whether young or not so young) facing the ruthlessness of the research granting system (Real Lives and White Lies in the Funding of Scientific Research). He suggests how a “drastic simplification of this grant-writing process would help scientists return to the business of doing science” and quotes a passage from a recent NYT column by Stephen Quake, who asks what sounds to me like a challenging question:

Could we stimulate more discovery and creativity if more scientists had…security of…research support? Would this encourage risk-taking and lead to an overall improvement in the quality of science?

I take this as a genuine question in search of a convincing empirical answer.

• The full article is available in PLoS Biology.
• CC-licensed photo courtesy of A. Kuzminski.

A few years ago I visited the Huntington Library in Los Angeles. We spent most of our time poking around the beautiful gardens, enjoying the Californian sun. But the Library collection is pretty remarkable too and it holds copies of the Gutenberg bible, Audubon’s bird drawings, early Shakespeare editions and – a definite highlight – Benjamin Franklin’s autobiography.

I’m not sure why I suddenly remembered this now, almost four years later, but when he wasn’t experimenting with electricity and founding countries, Franklin was also a bit of a productivity guru. Check out this extract from Chapter 8 of his autobiography (click for bigger):

He was pretty keen on everything having its own allocated time, supporting what he called the virtue of Order. He never seemed to be quite satisfied with the progress he made (bit hard on himself really) but the interesting thing, I think, is that you can see him actively reflecting on his progress and acknowledging its benefit. Check out these extracts:

I enter’d upon the Execution of this Plan for Self Examination, and continu’d it with occasional Intermissions for some time. I was surpris’d to find myself so much fuller of Faults than I had imagined, but I had the Satisfaction of seeing them diminish.

And later:

In truth, I found myself incorrigible with respect to Order; and now I am grown old, and my memory bad, I feel very sensibly the want of it. But, on the whole, tho’ I never arrived at the perfection I had been so ambitious of obtaining, but fell far short of it, yet I was, by the endeavour, a better and a happier man than I otherwise should have been if I had not attempted it

So there you go: Benjamin Franklin, the grandfather of personal productivity c. 1791. As if he hadn’t done enough already!

Looks like the Connotea team is on the right track. Instead of trying to bolt something to insert references into word, they are trying to go straight to wave.

We have blogged before about what a good integration between references and writing tools should look like, and quite honestly, Igor looks like it’s really getting it in terms of agility. You can specify a few terms and it disambiguates that into the reference you need. Looks smarter than the approach that endnote/bibTeX/zotero/Mendeley use. It only works for the online reference managers citeUlike and Connotea, though.

I’m not sure the references are portable, i.e., if I copy/paste a chunk of text with references, they come along to wherever I paste it to (it must be another wave, in this case). Endnote/bibTeX get this right, although they depend on a local file that you would have to send along.

As things stand, I think wave has a very good chance of becoming _the_ platform for collaborative scientific writing. You may have to convince your collaborators to try it (and some must have been put off by Google Docs, which is clearly not ready for science), but it could be very motivating to see their writing grow next to yours in real time.

Since wave is a lot more open than Google Docs it would not surprise me to see robots coming up to mend the deficiencies that make Docs unfit for papers: no tables, crossrefs, footnotes, equations, etc. Wave gives you versioning for free, which was another pain point of scientific collaboration.

Igor – a Google Wave robot to manage your references from Stew Fnl on Vimeo.

Cal Newport says people think that the more hard things they do, the more impressive they’ll be to potential employers. He calls this the diligence hypothesis. This is a leitmotiv in his blogging.

However, this trend of getting (and looking!) as busy as possible is not exclusive to undergrads (his audience). I don’t know any academic that doesn’t look stressed. We mostly hoard more tasks that they can realistically accomplish. But academics love their jobs (or so legend has it), whereas most people don’t. People who have day jobs say their long-term strategy for dealing with no life is to amass enough wealth to have more freedom of time to be able to do  things they love. We try to do the opposite: a job we love that invades every corner of our lives.

So the ideal Cal advocates is that of a hyperspecialist, that does one thing well, and that’s about it. This also fit the description of many successful academics: find a corner on your field where you are the undisputed king, and maybe start a fight with someone else on something obscure only the two of you care about. Penelope Trunk (one truly great blogger) seems to agree on the value of hyperspecialization in the corporate world too (and by the way, she does not recommend anyone to do a PhD ).

But are academic employers really impressed if you do that? I have no clue. It’s hard to guess what’s in the mind of hiring committees. But by looking at some recent hires on top depts in my field, I’d say it pays off to be an specialist, at least for an early career.

Cal got this very insightful comment (as usual):

I came to the conclusion that being stressed out and busy was for some reason a sought after way of being in western society.

Surprisingly, in the academia we manage to look really busy (as in trying new ideas) and hyperspecialized at the same time… Something doesn’t add up. Is being hyperspecialized a consequence of the market, or a way to alleviate stress (i.e., I don’t care about the theories that work two meters away from where I stand, I have plenty on my plate already)? Do you prefer to be perceived as a deep-but-narrow thinker or a risk-taking ‘trying new things’ explorer? Do you act in consequence with what you prefer?

Study Hacks » Blog Archive » Diligence vs. Ability: Rethinking What Impresses Employers

In a previous episode, I suggested that productivity is really just an efficiency measure. Since the working currency for academics is arguably prestige, productive researchers are those that can acquire the most prestige for the least effort and this can be formally written as:

where each task t is assigned a prestige benefit (p_t per activity × n activities) and an effort cost (attention units per hour a_t × h_t number of hours).

The comments on the original post suggested that there was a lot of enthusiasm for implementing and testing the theory and so I’ve spent the past month gathering data and preparing for a bit of an empirical assessment. The results are a work-in-progress but I hope to keep the conversation going and get your feedback. Here then is a step-by-step guide to how I’ve analysed my productivity over the last month using the general model.

1. Data collection

   I started by logging all of my work activities into a comma-separated file with three columns: the date, a description of the task, and the amount of time spent on that task. At first this was a pain but after a while, I got in the habit of opening the log file each morning and adding the data. I didn’t worry too much about data normalization at the time and recorded information at 15 minute intervals.

2. Data preparation

   Once the data was collected, I needed to tidy it up a bit. A quick inspection of my data file showed that I had used 83 distinct activity types but this was impractical for Step 3 below. Therefore I reduced these activities to 16 categories at first, then 8, and finally 6 categories. These included:

   • Administrata (email, filing expenses, preparing for conferences, sorting out IT problems etc.)
   • Internal meetings (with students, supervisors, project members)
   • Internal writing (conference summaries, minutes, project reports etc)
   • External writing (conference and journal papers, preparing external presentations)
   • Networking (participating at conferences and external events)
   • Research (reading, programming, data analysis)

   Non-work activities and work-related travel were excluded from the analysis.

3. Calculate productivity costs and benefits

   The next step was to calculate i) the amount of prestige gained (p_t) and ii) effort require to perform (a_t) each activity. In other words, I needed to develop quantitative measures that could distinguish between those activities that take a lot of hard work and yield big rewards and those trivial tasks that need to be done on a daily basis.

   To do this, I used the analytic hierarchy process. First, I coded an R function that performs an AHP analysis for a set of input factors (the code’s at the end of this post). AHP is usually done as a “hierarchy”, i.e. comparing options against different criteria and then successively aggregating the results into an overall score. However I did two separate analyses as a way of developing normalized scores for the prestige benefit and attention cost of each activity.

   For example, let’s consider the question of prestige. Each of the 6 categories defined above has different prestige measures that could be used such as citations for a journal publication. However comparing or aggregating these “native” measures for different activities is difficult and contentious; AHP instead rephrases the question and lets you work out in a rough sense what activities are the most prestigious.

   The function I wrote takes the set of categories as input and then asks you to perform pairwise comparisons. Categories are compared on a reciprocal 9-point scale where 1 means options A and B equally preferred and 9 means A is extremely preferred to B (and if A vs. B = 9, then B vs. A = 1/9). As I noted above, I had to reduce the number of categories because this routine requires n*(n-1)/2 comparisons. So 83 categories would mean 3403 comparisons and 6 requires only 15.

   The result is a series of weights for each category and a consistency ratio. The weights indicate the relative importance of each activity category and the consistency ratio indicates the extent to which the judgments were consistent. In other words, if you say that apples are tastier than pears and pears tastier than oranges, but oranges tastier than apples, then your results are inconsistent and the weights can’t be trusted. Ideally the consistency ratio should be less than about 10%. In my first attempt at this, I had a consistency ratio of 0.15 which is why I further reduced the number of categories to 6.

   The table below shows the results of the analyses. I’ve used the prestige weights directly from the AHP analysis; that is, the weights add up to one but don’t reflect any real units. However I’ve normalised the effort weights so that the largest weight represents one hour of maximum concentration. By taking the ratio of prestige to effort, we can work out which activity is most productive on a per-hour basis. Perhaps not surprisingly, networking is twice as productive as writing a paper or attending internal meetings.

   

   However there is some path dependence hidden in here: research may have a very low productivity ratio but clearly the more productive activities must be supported by good quality research results. One can’t happen without the other. This also explains why it’s easier for professors to accumulate prestige: the hard work is often done by grad students and researchers while the authorship and networking opportunities come more readily once your name is established.

4. Calculating the productivity index

   These weights can now be used to calculate the productivity of each day. Using the equation above, I’ve assumed that n_t is 1 for each entry in the database and then calculated the total productivity index for the day. The figure below shows how the productivity scores varies over time, with a three day moving average. Interestingly the low point during late June and early July coincided with a conference when I was doing a lot of traveling (the spike on 29 June represent the day I was presented a paper.)

   

   The data can also be used to examine work patterns. The first figure shows the average amount of time worked on each day of the week, not including meals, tea breaks etc. Out of interest, I typically commute to work on Fridays (and during this month, Wednesdays). This is nearly 3 hours a day on trains which, unless I’m reading work material, is “wasted” time (I do get through a lot of novels).

   

   The second figure shows how this corresponds to productivity. Again because I only have a couple days per week in the office, these tend to be my “necessary evil” days. Hence Wednesdays and Fridays score lower because I’m often busy with student supervisions and other administrative matters. And clearly, taking the weekends off helps to make Mondays more productive.

   

There are other analyses that I could perform with the data, for example breaking up productivity by task, so please feel free to add your suggestions to the comments. But I wanted to end by flagging up a couple issues that struck me while doing this:

• Simply recording your productivity increases productivity; it’s a question of reflective learning and feedback. Several times I caught myself writing “Surfed web” in the log file and then duly chastened, spent the next few hours knuckling down to do some hard work.
• The key to all of this is data collection so you need to find a system that works for you. The flat text file I’ve used is not a bad way of doing it but the excellent Flowing Data blog suggests that a person might use a private Twitter feed to record such information. Could be an interesting experiment.
• When performing the AHP analysis, it’s extremely difficult to evaluate certain activities. What is the prestige value of networking for example? If it contributes immediately to a grant proposal (e.g. in a funding workshop), then that obviously has a benefit. But if it’s simply meeting people, it may be a long time before any measurable prestige comes out of it. (Which of course is not to say that you should only meet people when you get something immediate out of it!)
• Work-related travel is a big drain on productivity, but a necessary evil. Although it’s obvious, using this time for reading is by far the best use of your time. All those research activities with low productivity ratios have to happen at some point and sometimes the office is too distracting.

I’m going to keep recording my productivity data and perhaps once I’ve got more data, I’ll be able to tease out some larger trends. But for the moment, feel free to try out the method and add your comments below.

The code
Here’s the R code I’ve used to perform the AHP analysis (AHP.r). It can be called using something like:

ahp <- AHP(c("Apples","Oranges","Pears"))

It will ask you compare all of the categories: answer using 1,3,5,7,9 or 1/3,1/5,1/7,1/9. Then to get the consistency ratio, type ahp$cr and to get the weights, ahp$weight

I want to try something a bit different in this post. Here at AP.com, we’ve talked a lot about tools, theory, trends and the general ephemera of academic productivity. But writing as academics, we should probably be trying to take this experience and build it into a cohesive model of productivity. So my goal here is to suggest a general model, one that we might use to understand what we’ve learned from previous posts and hopefully apply to our own work.

My starting point for this post was simple; I wanted to know how my productivity has changed (hopefully improved) since I first started my DPhil. From keeping a research journal, I know that some days are more productive than others and it would very helpful if I could understand when those fits and starts occur, to spot co-occuring events and thereby learn when to say “Forget work, I’m going for a run.”

In other words, I wanted to plot my productivity cycle over time. It might look something like this:

But the obvious problem with this exercise is how to measure productivity. It’s a subject that’s been tackled indirectly on this site before but going through the old posts, I haven’t yet find any attempts at a general theory – and related measures – of productivity. So drawing on the collected wisdom of previous AP.com posts, here’s a rough sketch of such a theory.

What is productivity?

Most simply, productivity is a question of efficiency: what outputs can be produced for a given amount of inputs? If you were working in a factory, measuring productivity is therefore fairly straight-forward: widgets per hour might be a nice personal productivity measure. But in an academic context, these inputs and outputs are not so easily defined.

This dilemma is briefly introduced in a discussion of productivity for programmers, where Gil Gordon suggests that when we say productivity, we really mean effectiveness. In other words, unlike a factory worker, our outputs can be multi-faceted and might be judged by their:

• Quantity (how much gets done)
• Quality (how well it gets done)
• Timeliness (when it gets done)
• Multiple priorities (how many things can be done at once)

Nevertheless the basic efficiency model is a good template for our model. So if P = productivity, O = output and I = input, we can write the basic definition:

Defining outputs

There are many different kinds of academic output. Papers, citations, funding received, teaching feedback, and promotions are just some of the ways in which we can measure our success, either directly or indirectly.

But as Jose pointed out previously, these outputs are all trying to attract a scarce resource, namely attention, and success in attracting attention results in prestige. So we might say that, for a given class of task t, output could be measured as:

where p_t is the prestige associated with task t and n_t is the number of those tasks completed in a given time period.

Measuring prestige varies with the task. For journal papers, citations seems like a sensible measure but for other tasks, this may involve a lot of guesswork. If your university has guidelines on promotion, they can be useful in identify how much of your professional success is expected to come from teaching, research and so on. But most likely you will need to use a technique like Analytic Hierarchy Process to unify your prestige measures.

Defining inputs

If you follow the game of cricket, you are probably familiar with the Duckworth-Lewis method. In an English summer, it often happens that one team has finished batting and their opponents have just started trying to catch the target score when it begins to rain. Rather than call off the whole game, the DL method is used to adjust the target score to account for a reduced amount of playing time. To do this, Messrs. Duckworth and Lewis developed their model using the concept of resources.

With productivity inputs, we can do something similar. The amount and quality of work that we can achieve depends on the resources available to us. But instead of wickets and overs in cricket, academic input resources might include time, money, lab access, the attention and effort we can devote to a task and so on.

Again, some of these inputs are more easily measured than others but if we want to generalize our model, we need some sort of conceptual common currency like we had with prestige on the output side. An economist might attempt to convert everything to money: how much would I have to spend to acquire this piece of data? But to link with our earlier discussion of prestige, I think a more useful framework is to convert everything to a common attention unit: let’s call it the Atnu for short (AU is already taken).

1 Atnu can be defined as the amount of attention necessary for a reference task, such as reading a journal article. It’s a rather arbitrary unit, but it’s intended to acknowledge that an hour of hard concentration is not the same as an hour spent doing miscellaneous administrative tasks like sorting through emails. It also has the advantage that you can define the Atnu as it makes sense to you and your work; if you spend time in a lab, performing an assay might be the base unit. My only suggestion would be that it is the defined as the most attention-consuming task. That way, the most difficult part of your day will correspond with actual hours.

So for task t, we can say that the total input is the total amount of attention hours spent on the job.

where a_t is the Atnu value for task t and h_t is the actual number of hours spent working at that level.

Final notes

Putting it all together then, productivity is the amount of prestige we earn for each attention-hour we invest.

It is a very simple model but from the form of this equation, we can already draw a few practical conclusions (even if they just confirm what we intuitively knew already):

• Productivity is maximized by concentrating on those activities that earn you the most prestige for the least effort. Conveniently the form of the equation is linear so that, assuming not all of the variables are unknowns, you could apply linear programming techniques to come with fancy “optimal” productivity strategies.
• Since the number of hours in a day is fixed, and we can arguably only give our full attention to a fraction of these hours, we should try to improve productivity by reducing either the number of hours or the amount of attention that a task requires. Some strategies let you do both things at once. For example, co-authoring a paper means you can delegate some of the work to someone else; you only need to invest a reduced amount of attention-hours to manage the project but you’ll receive similar amounts of prestige. Case in point: the world’s best connected mathematician.
• For those tasks that you have to do yourself, use your time wisely. This means balancing the levels of attention required by different tasks so you don’t burn out and focusing on those that earn the most prestige. This is why so much of what we write about here is concerned with time management, especially those tools that help us finish necessary but unrewarding administrata. See this post in particular.

What about actually implementing the model?

• Fitting the model to data and standardizing the coefficients for a large population will be difficult. As covered in this interview with Dan Navarro, there are “some big individual differences” in how people work; we shouldn’t put too much hope on one model holding for everyone.
• The interview also touches on another related issue: uncertainty. A strict optimization strategy is probably impossible because you don’t know the prestige associated with a task a priori. As Dan says, “it’s striking the right balance between exploration [of new ideas] and exploitation [of existing work]“.
• A related implementation issue is the problem of timing when collecting productivity data. This is two-fold: 1) when does the prestige arrive relative to the task being completed and 2) what is the temporal resolution of the feedback? One task may yield many different types of “prestige”: a small immediate personal satisfaction for completing a paper, a medium term recognition as it is published in a journal and cited, and maybe 50 years later receiving the Nobel Prize for your work. More from the archives on this here.
• I’ve said nothing about the psychology of productivity. It seems to me there should be feedback and “versus expectation” terms in all of this. Our earlier posts on learning theory and managing large projects (1,2,3,4,5) discuss the need to set clear goals, evaluate your performance against targets, and to learn from these experiences, changing your habits for next time. Such evaluations are key to maintaining your motivation and productivity.
• I’ve also said nothing about the physiology of productivity. What is the role of diet, daylight hours and half a dozen other factors?

So there you go. My two cents on what all of this productivity stuff really boils down to. I’m curious to see what you the readers think. Is this a crazy idea? Should we be trying to model productivity in such a formal way? Does anyone have the appetite for a community effort to gather some data and test the theory out?

Edit: There’s a follow-up post available with data testing this model.

Sciplore has produced an interesting hybrid between a mind mapping and reference management tool. Freemind Scholar adds two basic features over the excellent Freemind: you can have references (at this time, only bibTeX) inserted, and you can drag and drop hig hlights on pdfs (the pdf is linked).

This looks like the perfect IDE for sketching notes while reading papers. I use oneNote for this, but I’ve tried mind maps before and could easily revert to it.

As soon as they implement zotero/endnote references, I can imagine many people finding this tool very useful.

Freemind Scholar is in alpha right now. Feel free to try it out and send them your comments, chances are they will implement your feature requests since they are just starting.

According to Wikipedia, “the semantic web is expected to revolutionize scientific publishing, such as real-time publishing and sharing of experimental data on the Internet.” The W3C HCLS group’s Scientific Publishing Task Force is going to explore how this could happen.

Currently, one describes experiments in a more or less ad-hoc way. The mapping between experiments, papers, and titles is… well, not the most consistent ever.

Do you want to know if the experiment you have in mind is done already? Good luck mining the literature. Although mostly everyone is well-versed on building queries in scientific search engines, the task is far from accurate.

Maybe the problem is in the way we write the literature. If we could write a description of every experiment in some kind of agreed format that both humans and machines understand, searches would be trivial.

An alternative would be to use an ontology to describe experiments. The ontology should not be too complicated to use. If a user feels overwhelmed by the large number of parameters required to describe an experiment, this user may hesitate to do it. Of course, every field would need to built its own ontology. The effort to integrate ontologies across fields may be titanic.

There is some progress in the direction of using named entity extraction as metadata already. For example, the pubmed interface gopubmed is above and beyond anything I have seen. It uses facets (left sidebar) to show metadata. I do not know the details on how it works, but going back to say Web of Science after gopubmed feels like going 5 years back in time. Is there any hope to have a similar interface for all scientific databases? I sure hope so.

This is an exciting release.

In a single stroke, Zotero may have added the most important feature of online apps such as citeUlike (collaboration) and the best feature of Mendeley (metadata extraction). I have no idea how well these work, as I have just moved to zotero recently and don’t want to risk trying the beta this soon, but if they work well, this is a quantum leap.

Here’s a  list of the new features:

Automatic synchronization of collections across multiple computers. For example, sync your PC at work with your Mac laptop and your Linux desktop at home.

Free automatic backup of your library data at Zotero.org.

Automatic synchronization of your attachment files to a server of your choice (e.g. iDisk, Jungle Disk, or university-provided web storage).

Recover recently deleted items with Zotero’s trash can.

Rich-text notes

New style manager allowing you to add and delete CSLs and legacy style formats

Automatic detection of PDF metadata (author, title, etc.)

Automatic detection and support for proxy servers

New Website Features

Browse through your library online.

New user profiles tied to Zotero accounts.

Preliminary support for following other Zotero users: In the future this will generate a twitter-like feed of users public research activity.

WYSIWYG CV creator, with the ability to dynamically generate all or part of your CV from Zotero collections.

Search for friends, colleagues, and other users by institutional affiliation, username, email address, or field of interest.

I’ve been using endNote for about 6 years now, so I have experience a hard case of vendor lock-in (I have plenty of text with references in the endNote format. This is giving me a really bad feeling about endNote makers, and the money I have invested in a clearly inferior product doesn’t make me happy either. Suing Zotero was the last drop. I finally moved to Zotero out of principle. What I find funny is that zotero uses less memory with the same database size, even building on top of Firefox. And Word integration is as good, if not better, than endNote’s.