Open Source Science Evolves

I normally don’t cut and paste stories, but the Wall Street Journal has a pay wall, and I thought this was worthy of being accessed for free. It’s a great example of how the open source movement is changing fields left and write… er…. right.

By NICHOLAS ZAMISKA
September 14, 2006; Page B1
Wall Street Journal

Nature, one of the world’s most prestigious scientific research
journals, has embarked on an experiment of its own.

In addition to having articles submitted for publication subjected to
peer reviews by a handful of experts in the field, the 136-year-old
journal is trying out a new system for authors who agree to
participate: posting the paper online and inviting scientists in the
field to submit comments praising — or poking holes — in it.

Lay readers can see the submitted articles as well, but the site says
postings are only for scientists in the discipline, who must list
their names and institutional email addresses. Nature says its editors
screen out those they find irrelevant, intemperate or otherwise
inappropriate.

Meanwhile, the papers also make their way through the journal’s
traditional peer-review gauntlet. Nature says it’s taking both sets of
comments into account when deciding whether to publish.

So far, there have been only 70 posts on the 62 papers that authors
have decided to put on the Web site, according to Linda Miller, U.S.
executive editor of Nature, published by a unit of Macmillan
Publishers Ltd.

The experiment (at http://blogs.nature.com/nature/peerreview/trial/1)
is an early sign of how the scientific publishing establishment is
pushing the limits of its hallowed but opaque peer-review system.
Critics of the traditional process say it lets not only low-quality
papers, but also sometimes fraudulent ones, slip through the gates.
The Nature trial of a Web-based system could usher the spirit of
Wikipedia, the online encyclopedia edited by its readers, from the
margins of scientific publishing to the mainstream.

Dr. Miller says the current peer-review system is a good one but can
always be improved. "Journals are doing a lot of thinking about the
role of peer review and what tools we have available to us to try to
prevent the publication of either bad science or fraud," she says.
"Peer review was designed to filter out badly designed experiments and
poorly drawn conclusions. Can it really do anything other than that?
I’m not sure."

Questionable scientific research has made headlines recently. Late
last year, the claims of a once-revered South Korean cloning pioneer
unraveled and led to a criminal indictment and the retraction by the
journal Science of his fraudulent papers. Just last month, Nature
ballyhooed the work of researchers at Advanced Cell Technology Inc.,
of Alameda, Calif., claiming they had found a way to make embryonic
stem cells without destroying embryos. Now, that research also is
attracting criticism.

It’s far from clear that having those papers vetted on the Internet
before publication would have produced different outcomes. Still, many
eyes scanning a scientific work might catch flaws or inconsistencies
that a couple of officially sanctioned reviewers may miss.

At the same time, the experiment highlights the pressure on elite
science journals to broaden their discourse. So far, they have stood
on the sidelines of certain fields as a growing number of academic
databases and organizations have gained popularity.

One Web site, ArXiv.org2, maintained by Cornell University in Ithaca,
N.Y., has become a repository of papers in fields such as physics,
mathematics and computer science. In 2002 and 2003, the reclusive
Russian mathematician Grigory Perelman circumvented the
academic-publishing industry when he chose ArXiv.org to post his
groundbreaking work on the Poincaré conjecture, a mathematical problem
that has stubbornly remained unsolved for nearly a century. Dr.
Perelman won the Fields Medal, for mathematics, last month.

"Obviously, Nature’s editors have read the writing on the wall [and]
grasped that the locus of scientific discourse is shifting from the
pages of journals to a broader online conversation," wrote Ben
Vershbow, a blogger and researcher at the Institute for the Future of
the Book, a small, Brooklyn, N.Y., nonprofit, in an online commentary.
The institute is part of the University of Southern California’s
Annenberg Center for Communication.

"We’re following the experiment with interest," says Monica Bradford,
executive editor of the journal Science, a competitor of Nature. "Peer
review is central to scientific communication, and it’s important that
we’re open to examining the peer review process to ensure that it
remains a reliable means of vetting research."

Providing public access to scientific research before it hits print
could help address another thorny problem: the reluctance of
scientists in certain fields to share their work with colleagues
before they get credit for it. Public access to unpublished papers
could speed research on emerging diseases by reducing the time between
discoveries in a lab and their public release, allowing scientists to
build on one another’s work more quickly.

But the broader forum poses real risks. Journalists could report
prematurely on research that ended up discredited and not accepted for
publication. Ruth Francis, senior press officer for Nature, says the
journal won’t try to stop reporters from covering papers that are
posted online. (To date, most of the papers authors have chosen to
share online have not contained research that was generally newsworthy
or that could affect a public company’s stock price.)

And for researchers, putting one’s work out in the open is risky on a
personal level. In the fiercely competitive world of science, other
scientists could theoretically steal an author’s ideas and beat him or
her to publication.

One scientist who braved the online forum is Carl Spandler, a
researcher at the Institute of Geological Sciences at Switzerland’s
University of Bern. In early July, his paper was posted on Nature’s
public Web site about magma, molten rock beneath the earth’s surface
that can erupt through volcanoes.

In less than five weeks, five people, including top experts in their
fields, had chimed in with their thoughts. One was Paul Asimow,
associate professor of geology and geochemistry at the California
Institute of Technology, Pasadena, who posted a lengthy critique of
the paper. He attacked the paper on broad grounds — writing "the
abstract, introduction and conclusions all assume a false dichotomy"
— and also seized on a technical point. "I might quibble that the use
of 1-D diffusion fitting to a 250 micron long profile into the face of
a 1 mm 3-D olivine crystal could be pushing one’s luck."

"I thought the context was misstated and the interpretation was
inappropriate," Dr. Asimow said in a telephone interview. He says
Nature has tapped him in the past to participate in traditional peer
reviews, and if he had been asked to review this paper, it wouldn’t
have made the cut.

On Aug. 25, Dr. Spandler replied to Dr. Asimow’s comments with a
1,130-word rebuttal posted on the Nature Web site. In an email, he
wrote that the open system is a good idea in principle but said he
worries that the comments will sway official peer reviewers relying on
the online remarks as a crutch. "If they are lazy," he says, they "may
not dig deeper into the validity of criticisms."

Dr. Spandler also asserts that most of the comments will inevitably be
negative, "posted by those who may be threatened by the new results."

Write to Nicholas Zamiska at nicholas.zamiska@wsj.com3