Two excellent posts on "peer review"

By Shannon Love on Chicago Boyz:

"Scientific Peer-Review is a Lightweight Process".

Peer review isn’t even central to science. Science functioned fine for centuries without peer review and scientists who work in secret or proprietary environments do not use it. Instead, peer review serves economic and social functions related to scientific publishing and does nothing else. Peer review somewhat protects the integrity of scientific media, not the quality of science itself.

Peer review is a very superficial process more akin to a newspaper editor checking the grammar, spelling and punctuation in a letter to the editor before publishing it. . . .

Saying a paper is peer reviewed says nothing about the validity of its conclusions.

Quite the reverse, it is the fate of most scientific papers to be proven completely wrong. For example, at present, there are dozens of different variation of string theory in physics, all of which contradict each other. So, we know for a fact that the vast majority of published peer-reviewed papers on string theory are dead wrong. We just don’t know which ones because we lack the experimental technology to test which string theory papers accurately predict the outcome of experiments. That doesn’t stop physicists from churning out papers on string theory and it doesn’t stop journals from publishing them. They do this to foster the scientific dialog, not because they have any idea which of the papers, if any, will eventually be proven correct.

"No One Peer-Reviews Scientific Software".

Most climatology papers submitted for peer review rely on large, complex and custom-written computer programs to produce their findings. The code for these programs is never provided to peer reviewers and even if it was, the peer climatologists doing the reviewing lack the time, resources and expertise to verify that the software works as its creators claim.

Even if the peer reviewers in climatology are as honest and objective as humanly possible, they cannot honestly say that they have actually preformed a peer review to the standards of other fields like chemistry or physics which use well-understood scientific hardware. (Other fields that rely on heavily on custom-written software have the same problem.)

Comments

[kyle8]

I cannot agree with that article at all. I recently saw something on the Science channel about a theoretical physicist who pulled the wool over everyone for an entire year with fake experiments. But he was caught when someone noticed a problem with one of his experiments in a peer review article he published several years ago.

[Richard]

Without getting into the whole global warming thing, I think that the first article is way off base. To say that peer review is akin to grammar checking is just not true. The peer review process can be very rigorous - just ask any researcher who has toiled for months trying to satisfy a referee.

Furthermore, the statement that "Peer review somewhat protects the integrity of scientific media, not the quality of science itself" is absurd. By definition, the peer reviewers cannot know the "answer", as their job is merely to assess whether the conclusions of the paper are supported by the method. Sure, we have multiple string theories, and in theory they can't all be correct, but we do know that thanks to peer review, all the theories are pretty robust, not merely written with good grammar.

[JorgXMcKie]

Peer review undoubtedly varies depending on the discipline, the journal, the quality of the reviewers and their willingness to spend adequate time in their review, etc.

In poli sci I've learned that if you press authors about shaky stats work enough pretty soon you don't get asked to review any more submissions.
;->=

On the receiving end, I've also noticed that reviewers seem frequently to want the researcher to use their favorite stats method regardless of whether it is useful in the context of the research. just sayin'.