How to Easily Fix Peer-review

(comment originally posted at “The Predatory Gastropod“)

Here’s how to easily-fix peer review. Distribute via the web everything that the authors want to publish, alongside the reviewers’ comments. Clearly mark what has been approved by the reviewers (eg print on paper only what the Editors indicate as worthwhile…but that’s what’s happening anyway, as articles approved by reviewers may still be discarded by Editors).

It’s caveat emptor and all that.

I would keep out of “scientific” distribution only the most egregious nonsense, such as papers arguing about the Moon being made of green cheese. Everything else has to have the opportunity to see the light of the day in a commented manner, and if people start publishing flawed result after flawed result, well, their reputation will crash faster than a PhD graduate’s making fraudulent claims in peer-reviewed articles.

Likewise for reviewers’ reputation. This will also stop the nonsense of reputable scientists wasting time by trying to stab each other in the back in order to prevent “competitors” from being able to claim to have been peer-reviewed.

I do hope we are at a threshold similar to when the “one” Church discovered it didn’t have the “monopoly of the Truth”. Despite what some people thought at the time, having a plurality of Christian denominations hasn’t meant the demise of Christianity, to the contrary, it has helped improve the lot. Likewise, publishing scientific papers that aren’t fully peer-approved won’t mean the demise of Science, it will likely help make findings and theories stronger.

Peer-Review Flaws, circa 2002

People familiar with what happened in climate science during the last year might find Lawrence K. Altman’s NYT article “THE DOCTOR’S WORLD; When Peer Review Produces Unsound Science” of June 11, 2002 more than prescient (emphasis all mine, of course):

[…] Yet for all its acclaim, the system [of peer-review] has long been controversial. Despite its system of checks and balances, a number of errors, plagiarism and even outright fraud have slipped through it.

[…] A particular concern is that because editors and reviewers examine only what authors summarize, not raw data, the system can provide false reassurances that what is published is scientifically sound.

[…] Researchers reported [in the “The Journal of the American Medical Association” in June 2002] considerable evidence that many statistical and methodological errors were common in published papers and that authors often failed to discuss the limitations of their findings. Even the press releases that journals issue to steer journalists to report peer reviewed papers often exaggerate the perceived importance of findings and fail to highlight important caveats and conflicts of interest.

[…] Because the anonymous peers chosen to review manuscripts are often the authors’ scientific competitors, jealousies and competitive advantage can become factors in the reviews.

[…] The peer review system also tends to set a very high barrier for authors to publish truly novel findings.

[…] Yet research on peer review has found that many studies are conducted without the benefit of adequate consultation with statisticians, sometimes because none were available.

[…] Once statistical errors are published, it is hard to stop them from spreading and being cited uncritically by others. […]

Are AGW-prone Editors A Hindrance To Science?

Imagine preparing to submit an article to a scientific journal, in response to something they have just published.

Imagine spending an enormous amount of time getting all relevant references together, all the reasoning properly done, all the computations calculated and reviewed thrice, to be sure about your results.

Imagine clarifying your message with the original authors, arriving at the conclusion that there really is something new in what you’ve found.

Imagine submitting your contribution following each and every rule for publication.

Imagine awaiting the response of the anonymous referees.

Imagine getting positive reviews and recommendation for publication from two referees out of two, a 100% success rate on the peer-review side.

Imagine then receiving an e-mail from the Editor of that “scientific” journal, explaining that they are not going to publish what you’ve referenced, reasoned, computed, clarified, submitted, and even gotten positive reviews about.

Imagine looking at your work, reviewed and approved by peers. Unpublished because of an Editor.

Oh…and of course…your work was not following the AGW consensus. No need to imagine that.

Peer Review (And Fraud) Ain't The Only Things To Fix In Modern Science

Worried about Editors of scientific publications overeager to publish only what confirms and conforms to their prejudices? Of peer reviewers too friendly to their omerta-based ilk and too ideological to accept what may contradict their work?

And now there is something else to worry about, in the realm of scientific publishing: foul play in citation (aka “bibliographic negligence” and “citation amnesia“).  That is, the malpractice to “forget” the citation of a rival’s article, or of previous research that would detract from the allegedly unprecedented, ground-breaking nature (and therefore, importance) of one’s article.

As suggested by Richard Gallagher in the same article, the solution may be straightforward:

We need a code of practice for citation, which journals should adopt explicitly. Gene Garfield called for this many years ago, suggesting that authors sign a pledge or oath that they have done a minimal search of the literature and that to the best of their knowledge there is no other relevant work. This is, in fact, the oath one signs when filing for a US patent.

But can the above be used to poo-poo modern science? Au contraire. Listen to Gallagher again:

Judging by the amount of publicity for fraud and greed in science, standards appear to be in freefall. I am not sure that I buy it. I think that the openness gifted us by the Internet is revealing the lax standards that have been in place all the time. The purifying glare of publicity may actually help us [the scientists] get our house in order—I wish that the editors of research journals would get this.

A "Scientific Ombudsman" To Avoid a Scientific Schism

The Scientist” reports about University of Cambridge biologist Peter Lawrence‘s (and others’) complaint against Cell, “one of the most cited scientific journals” according to Wikipedia.

Improper citation, disregard for antecedent research, and shoddy experimentation – those are just a few of the allegations levied against a recent research paper […]

Lawrence wrote in a letter to Cell that the paper was “seriously flawed both scientifically and ethically […]” Lawrence’s letter was not published in Cell, but he sent it to The Scientist. […]

Editors at Cell did not respond to an email request for comment in this story. However, the journal’s senior scientific editor, Connie Lee, did respond to Lawrence’s letter [but] declined his request to publish a minireview, instead offered Lawrence the opportunity to post his comments on Cell’s website. […]

Lawrence, however, would like to see action taken to address the issue of scientific scoopsmanship on a broader level. “There should be some kind of scientific ombudsman that people could contact when they feel they’ve been wronged,” he said. “At the moment, there’s nothing.”

It is said that scientific peer-review is like Democracy: full of flaws, but there isn’t anything better (I do have indirect experience with asinine comments by ignorant reviewers taken as Truth by editors of scientific journals with a purpose). But Democracy has been evolving and dare I say improving itself with time, whilst peer-review is somehow considered too saintly to be touched.

This has the unfortunate consequence that there are now people explicitly asking for its “overhaul”.

=============

UPDATE NOV 26: The Scientist has today another article praising peer-review as it happens today, and a few comments critical of it.

=============

The risk there is for a major Scientific schism, with some sticking to their little ivory towers of mutual peer-review; and others deliberately abandoning any attempt to publish in peer-reviewed journals, consigning their work to the Internet masses.

Whole areas of research may descend into “scientific wars” full of mutually-incompatible claims about the world we all live in. That will leave everybody unfamiliar with the field at a complete loss on what is, and what is not known.

This may have already happened, in Climatology, leading to Intergovernmental Panels etc etc.

I’d rather prefer a scientific ombudsman, thank you very much.

Scientific American's Mauling to Pieces of Nature's Review Process

This will be no news to those that know of the Hockey stick controversy
Nature editors reject peer review process that reduces gender bias

Following a surprisingly unscientific line of reasoning, the editors at the most renowned and prestigious of science journals have rationalized away the need to fix an ailing peer-review system.

Increasing skepticism about the effectiveness and integrity of single-blind peer review—the process by which most academic papers submitted for publication are accepted or rejected—has prompted empirical evaluation of the system.

Standard practice is: reviewers—selected for their expertise and fluency in the chosen discipline—are aware of all authors’ names and affiliations, while authors are kept in the dark about the identity of their reviewers (although some journals allow them to request specific referees).

The growing argument against this lopsided method is that knowledge of authors’ identity—gender, nationality, research institution, level of experience in the field—can (and does) bias reviewers’ opinions on the merit of the research.

The most vocal critics of the current system are those who believe their submissions do not get fair consideration—women, early-career scientists, people with foreign-sounding names—when matched up against authors who sail through the submission process on the status of their lab or the history of their career. And in an environment in which research funding, hiring, tenure, salary, and academic reputation are massively dependent on publishing record, one can easily imagine the ripple effects such a disadvantage would bring. […]