Last week, while I was waiting for my sandwich at the deli downstairs, I was applying about 10% of my consciousness to CNN running on the TV behind the counter.  I saw some woman, presumably in the Obama team, defending some action of the administration as being based on “peer reviewed” science.

This may be a legacy of the climate debate.  One of the rhetorical tools climate alarmists have latched onto is to inflate the meaning of peer review.  Often, folks, like the person I saw on TV yesterday, use “peer review” as a synonym for “proven correct and generally accepted in its findings by all right-thinking people who are not anti-scientific wackos.”  Sort of the scientific equivalent of “USDA certified.”

Here is a great example of that, from the DailyKos via Tom Nelson:

Contact NBC4 and urge them to send weatherman Jym Ganahl to some climate change conferences with peer-reviewed climatologists. Let NBC4 know that they have a responsibility to have expert climatologists on-air to debunk Ganahl’s misinformation and the climate change deniers don’t deserve an opportunity to spread their propaganda:

NBC 4 phone # 614-263-4444

NBC 4 VP/GM Rick Rogala email: rrogala(ATSIGN)wcmh.com

By the way, is this an over-the-top attack on heresy or what?  Let’s all deluge a TV station with complaints because their weatherman has the temerity to have a different scientific opinion than ours?  Seriously guys, its a freaking local TV weatherman in central Ohio, and the fate of mankind depends on burning this guy at the stake?  I sometimes get confused about what leftists really think about free speech, but this sure sounds more like a bunch of good Oklahoma Baptists reacting to finding out their TV minister is pro-abortion.   But it is we skeptics who are anti-science?

Anyway, back to peer review, you can see in this example again the use of “peer review” as some kind of impremateur of correctness and shield against criticism.   The author treats it as if it were a sacrament, like baptism or ordination.   This certification seems to be so strong in their mind that just having been published in a peer-reviewed journal seems to be sufficient to complete the sacrament — the peer review does not necessarily seem to even have to be on the particular topic being discussed.

But in fact peer review has a much narrower function, and certainly is not, either in intent or practice,  any real check or confirmation of the study in question.  The main goals of peer review are:

• Establish that the article is worthy of publication and consistent with the scope of the publication in question.  They are looking to see if the results are non-trivial, if they are new (ie not duplicative of findings already well-understood), and in some way important.  If you think of peer-reviewers as an ad hoc editorial board for the publication, you get closest to intent
• Reviewers will check, to the extent they can, to see if the methodology  and its presentation is logical and clear — not necessarily right, but logical and clear.  Their most frequent comments are for clarification of certain areas of the work or questions that they don’t think the authors answered.  They do not check all the sources, but if they are familiar with one of the sources references, may point out that this source is not referenced correctly, or that some other source with which they are familiar might be referenced as well.  History has proven time and again that gross and seemingly obvious math and statistical errors can easily clear peer review.
• Peer review is not in any way shape or form a proof that a study is correct, or even likely to be correct.  Enormous numbers of incorrect conclusions have been published in peer-reviewed journals over time.  This is demonstrably true.  For example, at any one time in medicine, for every peer-reviewed study I can usually find another peer-reviewed study with opposite or wildly different findings.  The fraud in the “peer reviewed” Lancet on MMR vaccines and autism by Andrew Wakefield is a good example.
• Studies are only accepted as likely correct a over time after the community has tried as hard as it can to poke holes in the findings.  Future studies will try to replicate the findings, or disprove them.  As a result of criticism of the methodology, groups will test the findings in new ways that respond to methodological criticisms.  It is the accretion of this work over time that solidifies confidence  (Ironically, this is exactly the process that climate alarmists want to short-circuit, and even more ironically, they call climate skeptics “anti-scientific” for wanting to follow this typical scientific dispute and replication process).

Further, the quality and sharpness of peer review depends a lot on the reviewers chosen.  For example, a peer review of Rush Limbaugh by the folks at LGF, Free Republic, and Powerline might not be as compelling as a peer review by Kos or Kevin Drum.

But instead of this, peer review is used by folks, particularly in political settings, as a shield against criticism, usually for something they don’t understand and probably haven’t even read themselves.  Here is an example dialog:

Politician or Activist:  “Mann’s hockey stick proves humans are warming the planet”

Critic:  “But what about Mann’s cherry-picking of proxy groups; or the divergence problem  in the data; or the fact that he routinely uses proxy’s as a positive correlation in one period and different, even negative, correlation in another; or the fact that the results are most driven by proxys that have been manually altered; or the fact that trees really make bad proxies, as they seldom actually display the assumed linear positive relationship between growth and temperature?”

Politician or Activist, who 99% of the time has not even read the study in question and understands nothing of what critic is saying:  “This is peer-reviewed science!  You can’t question that.”

Postscript: I am not trying to offend anyone or make a point about religion per se in the comparisons above.  I am not religious, but I don’t have a problem with those that are.  However, alarmists on the left often portray skepticism as part-and-parcel of what they see as anti-scientific ideas tied to the religious right.  I get this criticism all the time, which is funny since I am not religious and not a political conservative.  But I find parallels between climate alarmist and religion to be interesting, and a particularly effective criticism given some of the left’s foaming-at-the-mouth disdain for religion.