Friday, December 10, 2010

Peer Review and Scientific Publishing - Time to kill the sacred cow

If the arsenic bug controversy has taught us anything it is that peer-review is an imperfect means of screening out bad science from good science.  The peer reviewers of the recent Science Express article missed, or deliberately overlooked several major flaws in the paper.

Here's one that I have not seen publicized among all the critiques (caught by an insightful post-doc in our lab) - Figure 2 shows the growth of the mystery bacterium under 3 conditions, with phosphate added, but not added arsenate (+P/-As), with arsenate added but no phosphate (-P/+As) and the "control" (-P/-As).  The +P/-As grew best, while the -P/+As grew a little (to about 1/10th the density of the +P/-As), and the "control" did not grow at all.  The authors suggest (and one would normally agree) that this shows that the bacterium could use As in place of P for growth (they grew above the -P/-As control), but not as well.  However when you read the supplemental material, the control is not a good control; both the +P/-As and the -P/+As media had glucose (a simple sugar and a very good food source for most bacteria), while the -P/-As had no glucose added!  Now, if the control had glucose added and had not grown, it would be a reasonable deduction that they had not grown due to lack of P, or in it's absence As.  However, if they had grown and matched the -P/+As, you would concluded rather that the ambient or contaminant P in the medium (there was some and it was not inconsequential) was sufficient to allow growth, and that the As had no effect.  So that particular graph is terribly misleading.

The process of scientific publishing and peer review has grown into at least a minor industry.  A number of large publishing houses have divisions devoted to taking articles from authors, getting them peer-reviewed, and publishing them, and selling the journals (or more recently their digital equivalents) to libraries and scientists, at very high costs (a typical journal might cost a scientist $500 a year for a personal subscription, and research libraries get a real deal, 4-5 times that cost.  The process of peer review does, in fact, screen out many truly bad articles, as any one who has reviewed papers can attest.  However, it take months, sometime even years for a paper to get past peer review and into press.  And the process is expensive, with page charges for authors publishing the papers often costing over a $1000.  There is also a possibility that peer review can be used by scientific partisans to prevent the publication of opposing views.  This was long asserted in the Global Climate Change vs. "The Climate Deniers", and the e-mails leaked from CRU clearly indicate that pro-anthropogenic global warming folk actively sought to suppress articles that did not agree with their work, and then chided the opposition for not having sufficient peer review articles.

My radical solution.  Do away with the entire apparatus of scientific publishing as it exists.  Allow scientists to digitally publish, on the web, whatever they wish.  If it's crap, you'll find that either no one cares (in which case the publication probably would generate no interest any way, there's a lot of that out there now anyway, the "least publishable unit" or LPU, used to pad the CV for tenure decisions), or that somebody who cares will read the article critically, and comment on their own on its strengths and flaws. Let the institution where the scientist works provide the web space, or his own "Slog" or the sponsor, or even if necessary, the government (but I'm pretty sure Google will suck it all up into their archives and make it unnecessary).  Voila, instant publication!  Is is bad?  Maybe, but we see bad stuff getting through now.  Make scientists responsible for the evaluation of the work they rely on instead of some anonymous reviewers who may or may not be looking critically at the paper, and who may or may not have an axe to grind.

Some more interesting articles on the arsenic bug...

Discover blogs

Rosie Redfield

No comments:

Post a Comment