A Quick Thought on “Peer Review”

One of the weird aspects of climate science is the over-emphasis on peer review as the ne plus ultra guarantor of believable results.  This is absurd.  At best, peer review is a screen for whether a study is worthy of occupying limited publication space, not for whether it is correct.  Peer review, again at best, focuses on whether a study has some minimum level of rigor and coherence and whether it offers up findings that are new or somehow advance the ball on an important topic. 

In "big boy sciences" like physics, study findings are not considered vetted simply because they are peer-reviewed.  They are vetted only after numerous other scientists have been able to replicate the results, or have at least failed to tear the original results down.  Often, this vetting process is undertaken by people who may even be openly hostile to the original study group.  For some reason, climate scientists cry foul when this occurs in their profession, but mathematicians and physicists accept it, because they know that findings need to be able to survive the scrutiny of enemies, not just of friends.  To this end, an important part of peer review is to make sure the publication of the study includes all the detail on methodology and data that others might need to replicate the results  (which is something climate reviewers are particularly bad at).

In fact, there are good arguments to be made that strong peer review may even be counter-productive to scientific advancement.  The reason is that peer review, by the nature of human beings and the incentives they tend to have, is often inherently conservative.  Studies that produce results the community expects often receive only cursory scrutiny doled out by insiders chummy with the authors.  Studies that show wildly unexpected results sometimes have trouble getting published at all.

Poscscript:  As I read this, it strikes me that one way to describe climate is that it acts like a social science, like sociology or gender studies, rather than like a physical science.  I will ahve to think about this — it would be an interesting hypothesis to expand on in more depth.  Some quick parallels of why I think it is more like a social science:

  • Bad statistical methodology  (a hallmark, unfortunately, of much of social science)
  • Emphasis on peer review over replication
  • Reliance on computer models rather than observation
  • Belief there is a "right" answer for society with subsequent bias to study results towards that answer  (example, and another example)
  • og

    (which is something climate reviewers are particularly bad at).

    That statement is erroneous. Climate reviewers are not particularly bad at replicating those results, they are SPECTACULARLY bad at it. Please be more careful choosing your words.

  • Scientist

    Your beliefs about climate science get ever stranger, and ever further from reality. The mere fact of peer review does not mean a paper is any good, or its results believable, and no climate scientist has ever claimed otherwise. Unless you can prove me wrong by showing us a statement to the contrary, then you are shown again to be a liar.

  • og

    Ooh! He’s made it all the way to page 56 of the AGW playbook. “When all else fails, attack the critic’s credentials”. As if this means anything. Coming from the very definition of lack of credentials.

    I swear, boss, this guy cannot possibly be for real, even Algore isn’t this dense. You sure this isn’t a sockpuppet?

  • Josh S

    The NOAA says that if it’s peer-reviewed, you can have faith that it’s valid:
    http://www.ncdc.noaa.gov/paleo/globalwarming/peerreview.html

    Michael Mann needs to read Thomas Kuhn
    http://www.realclimate.org/index.php/archives/2005/01/peer-review-a-necessary-but-not-sufficient-condition/

    “Not sufficient” sounds tantalizing, but if you read the article, means little more than “things that don’t support the consensus occasionally slip through the cracks.”

  • dreamin

    My spouse is a professor in the humanities and has had to face peer review. After reading the reviews, it seems to me that the reviews are more the reflective of the political leanings of the reviewer than the actual merit of the publication.

    In my experience, the vast majority of people are incapable of assessing arguments and evidence objectively if they have their own viewpoint.

    I don’t know much about peer review in science, but I have a strong feeling that the same sort of political correctness is invading climatology. Thus I am skeptical of this worship of peer review which pervades the CAGW debate.

  • mbabbitt

    Thank you for that discussion. It’s really needed. Most people who are not in the sciences believe that peer reviewed means peer approved, that the reviewers found the arguments credible and supported by the evidence presented and thus in some important way true. Like you say, peer review really just “focuses on whether a study has some minimum level of rigor and coherence and whether it offers up findings that are new or somehow advance the ball on an important topic.” Then the study is staged to be replicated and critiqued. But only people in the sciences know this and so when they hear “peer reviewed” they believe it has been found to be true. And this is where the weakness of politicizing science does the most damage.

  • mbabbitt

    correction: “so when they hear “peer reviewed” ” in the last sentence of my commment above refers to the non-scientist.

  • Stevo

    Here’s a physics professor talking on the same subject.

    “The millions of man-hours that went into all that worldwide research have proved worthless in the pursuit of human progress. The main reason for this failure was the formulation of an establishment orthodoxy that prevented the publication of alternatives.”

    http://www.numberwatch.co.uk/then_and_now.htm

    I’m afraid it’s just human nature.

  • Luis Dias

    I think it should be inflated to a kind of IPCCgate. That would infuriate the status quo. And that is always in need.

  • stan

    Climate “scientists” do a tremendous amount of their “work” using the data from ground weather monitoring stations. Yet, no one ever bothered to see if these stations meet the most basic level of scientific competence. It took amateurs to go look and discover that the vast majority fail to meet standards.

    Climate “scientists” accepted Mann’s fraudulent, error-ridden study and overturned everything the scientific world had known about the little ice age and med. warming period. Yet, no one checked his data, his math, or his statistical modeling (all of which turned out to be bad when finally checked by an outsider).

    The sea surface temperature “study” relied upon by climate scientists turned out to have a major error because the “scientists” were too lazy to make a phone call to the navy. Their wild ass guess wasn’t discovered for years because no one in their field ever checks anyone else’s “work”.

    Note the testimony before Congress.

  • Josh S

    The same thing happens in CFD, too. This “eddy viscosity” business won’t go away because, well, it’s orthodox.

  • “Peer review” is important to collectivists and those who believe consensus is a validating factor in science, since they equate peer-review with validation and thus rightness – thesis/antithesis/synthesis/ is the road they march upon. It’s how they grade themselves.

  • One way to clean up the peer review vagaries would be to require journals to publish the peer review comments, with full disclosure of who the reviewers are, along with the original article. I believe M. Crichton has pushed for this. By publishing the reviewers’ comments, one could better determine why the paper was accepted for publication. It would also bring more scrutiny to the reasoning used by the reviewers, if the paper was accepted because the reviewer just happened to be associated with the author, some heads might turn.

    Right now, in some fields, peer review has become a joke. As Warren pointed out, in physics and mathematics, peer review simply means that a paper is sufficiently robust for publication and debate. In climate science, peer review has come to mean “I’m right, my peers said so, so get lost.” One question that haunts me is how journals decide who the peers are. If Lord Moncton wrote a paper and submitted it for “peer review” then the peers should be non-PhD individuals with education and track record similar to Moncton. A peer is a member of a group of people of the same age, status, ability.

  • Rob

    Scientist,

    Your beliefs about climate science get ever stranger, and ever further from reality. The mere fact of peer review does not mean a paper is any good, or its results believable, and no climate scientist has ever claimed otherwise. Unless you can prove me wrong by showing us a statement to the contrary, then you are shown again to be a liar.

    Why do you continue with ad hominem? You and the blogger are in agreement (peer review != quality work), yet you still deride him? Furthermore, what is he supposed to prove here? He never said climate scientists believe peer review makes their papers good. At best, he implies that the politicians and on-lookers see and use “peer review” to back up the imagined authority of a paper. I even think you’ve dropped the “peer reviewed” bomb in the past postings, so I’m confused by your response this blog entry.

    Maybe you meant to say that you want him to prove that climate scientists don’t publish their full methodologies because they fear criticism even though both constructive or destructive help advance the results in some way? Even this has been shown already to be true. I would posit that this is because they don’t want to lose their comfy funding (whoops, money doesn’t bias the pro-man-made-global-warming studies) and I suppose that a delayed release of data/methodology can still be viewed as a ploy to buy time and even more funding !?!?

  • Scientist

    Rob – no, quite clearly the claim here is that climate scientists believe that if their work is peer-reviewed, then it’s correct and unquestionable. That claim is idiotic. I deride him because his views about climate science are borne entirely of ignorance.

    And no, there is no ad hominem argument in my post. I don’t think you understand what the term means. If I had said ‘this post is ridiculous because you’re an idiot’, that would be ad hominem. I have not done that. Call me rude if you like, but my arguments are not ad hominem.

    jnicklin – In climate science, peer review has come to mean “I’m right, my peers said so, so get lost.” – bullshit.

    If Lord Moncton wrote a paper and submitted it for “peer review” then the peers should be non-PhD individuals with education and track record similar to Moncton. A peer is a member of a group of people of the same age, status, ability. – you think that would make for good science?

  • Luis Dias

    Mr. Scientist (or whatever), whenever I see scientists, when asked for data and information that proves the point they make in their papers, simply asking back “why bother if you’re just going to look for mistakes” I think that really undermines my trust in those papers, for it is clear that the authors are simply completely ignorant (and arrogant at that) of the scientific method by denying it the core of science itself: SKEPTICISM.

    One should keep in mind on why is science different from religion. Many ignorants commonly equate the faith of religion to the faith on science. That ain’t so because there is no faith involved in science, but rather skepticism and unbelief. Evidence and facts speak for themselves and require not appeals to authority.

    Having said that, the latest episode on Climate Audit speaks for itself, and I do believe it ought to be called some kind of a IPPC Gate. Don’t jump into conclusions on what I am about to say though, for Steve McIntyre is still unsure on what steps he will make from now on.

    What happened is simple. Michael Mann made a graph in 99. MBH99 is the paper and it entered in the AR3 report in 2001. McIntyre and McKittrick (?) reached the conclusion that the statistics on that paper are flawed (MM2005 and MM2005a). Wahl and Amman (sorry if I misspelled) made a subsequent paper declaring that MM2005 was wrong. But the core of its proof is hidden in a Supplementary Information (SI) that is not provided. After some rejections it was accepted by the IPCC right on the deadline on 2006 for the AR4 report of 2007.

    This means that the AR4 report of 2007 got to say that MM2005 was wrong and that Michael Mann was right and cited WA2006.

    The news is, they finally archived the stuff. And McIntyre got to see it. And hold and behold, not only the SI didn’t falsify MM2005’s claims, it even confirmed them, albeit the fact that the principal paper completely ignores this and claims the contrary.

    Calls for deceit, lying, and fraud are being made at CA. But this is no schoolyard, the devil is in the details, there can be things still to be cleared out, so I think caution will be needed and McIntyre is cautious enough.

    If facts remain though, it’s egregious.

    Mr. Scientist, as you can see, you …. (put the dark glasses on Caruso style) …. are entirely wrong. (Wham! comes the music score)

  • Alex Llewelyn

    you think that would make for good science?

    No, that’s his point…

  • Mesa Econoguy

    I think Luis is a fairly new convert to skepticism.

    His previous statements should be kept in mind (he has no clue about null hypotheses), but I think he’s starting to come around, albeit slightly melodramatically.

    The CA piece is pretty damning. Let’s get more.

    Then let’s talk about the economics of what these people are proposing based on fake science.

    I’ll be happy to explain in person to this fellow economist why he’s wrong.

  • kuhnkat

    Psyentist,

    how many times have you trashed sceptic papers because they haven’t been PEER REVIEWED????

    HAHAHAHAHAHAHAHAHAHAHAHA

  • Luis Dias

    Mesa, you are misinterpreting me badly.

    I have always been a skeptic, that is why I’ve been following this discussion, and that also means that I am also skeptical of skeptict’s claims. Just because the IPCC is being handled badly doesn’t follow that I endorse Lord Munckton’s views, or any other fringe theory for that matter. Not because they are wrong, but mostly because I lack the skills to review them.

    What I don’t like is how scientific method is being treated by some scientists. Understand that while this episode appears to support the idea that the IPCC wants the HS to survive, we also should remember that the AR4 report badly mentions HS. Worse, McIntyre and others believe that the science of Paleoclimate is completely unnecessary for the discussion at hand.

    So lets slip the waters. Obviously though that this kind of case undermines trust upon the standards of science that IPCC should maintain.

  • Luis Dias

    PS: what are babbling about with me not knowing null hypotheses? Its as if you are always trying to compensate for something you lack with hit points…

  • DC

    Don’t care who you are you will bring presuppositions to your work, period. Presupps will taint the findings because it is either proving your point (presupps affirmed) or disproving your point (presupps false) You can’t distance yourself from your core inner beliefs(no religion involved here).

  • Look, IPCC is political. Enough said. Except someday the whole concept of man-made global warming will be properly called ClimateGate! Possibly, ScienceGotGored! Peer reviews? LOL. Arthur Anderson, the largest CPA firm in the world had peer reviews done regularly – but truth and ultimately bankruptcy got them anyway. It ain’t rocket sciene folks.

  • skydiver48

    Just one more thing. Models are what fooled a host of believers. It was the skeptics, or the shorts, that took down Enron. Models that projected future movements in the field of financial science cost investors billions. Few really had the time to understand the models, but none the less, they were accepted by those who got “peer reviewed”. I heard it announced about one year prior to their collapse that Enron had “won”, and their competitors could never catch up. The debate about who would emerge as the king of energy was over.
    The IPCC models purporting to prove man-made global warming include just as many or more “subjective” inputs as did the Enron models. Fantastically, Enron’s models produced results that supported desirable values and upheld Enron’s truths. Its just a matter of time before it is absolutely clear that the believers and the western world have been gored and bled of trillions of dollars – in spite of the results of the IPCC models – and in spite of the proclamation that “the debate is over”.

  • Mesa Econoguy

    Luis, I was referring to comments you posted on the Briggs (statistics) website stating you did not share their skepticism about rejection of their research hypothesis in a particular example.

    You either confirm or fail to confirm your hypothesis given your confidence interval, it’s that simple. You don’t get the option to say “well, I’m not so sure about that.” It is, or it isn’t. I recall one of the follow-on posters said something snarky like “You’ve just proved the existence of God.”

    But let’s not focus on that. Welcome to the skeptics fold. The economics of the proposals to “curb” AGW is even more alarming than the scientific shenanigans used to justify them.

    And you are absolutely correct that real damage is being done to the scientific process by this movement.

  • Luis Dias

    I didn’t understand well enough the issue at hand. I’ve always had IQ well above average but I didn’t choose math fields on my learning path, so there were concepts that I still don’t understand well enough. Still, I saw what he did and his conclusions didn’t fit with my intuition.

    So I was skeptical at his position, which is a common trait in me. I understand that it was acknowledged as a “sheep-follower-troll-comment”, but it was nothing of the sort. I was “new” to the debate at hand, and unaware of certain facts that are egregious.

    Having said that, I still think that the political body should continue to regard the IPCC as their scientific advisor machine. The concept is a little monolithic, but given enough pressure and if the “gates” are duly reported to the public, truth will prevail. The people of misconducting behaviors should be expelled though. False consensus are also a problem, but I think this is nothing out of the ordinary in science.

    The main reason of my “belief” that such is the best system comes from the fact that the alternative is to have a disparate myriad of different scientists claiming their own theories, to which any sane politician will say: “Or you all come to grips or nothing will happen”. But it has to have a different way to function, because clearly, it isn’t working as it should.

    I recall one of the follow-on posters said something snarky like “You’ve just proved the existence of God.”

    StevenMosher was still bruised and cranky at the other discussion about atheism and theism where I kicked his ass good :).

  • The Idiots Have Spoken. Humans Cause Climate Change. Case Closed.

    Here’s some peer review for ya. I guess this means you can close down your web site, eh? How ridiculous.

    http://environment.newscientist.com/channel/earth/mg19926683.300-humans-cause-climate-change-us-body-accepts.html?feedId=online-news_rss20

  • papertiger

    About the postscript. Lubos Motl pointed out that climate, and by extension climate science, is the study of what humans expect the weather will be in the future based on past experience. The operative phrase is “human expectation” which puts climatology squarely in the realm of anthropology. Social studies is very appropriate as a description.

  • There’s a little misdirection going on here. It’s the non-scientist politicians, media, and others who are relying on the phrase “peer-reviewed” as a passkey. The scientists, for their part, are relying on the politicians, media, and others to carry water for them. AFAIK, very few of the scientists are protesting the misuse of their work; a few of them even applaud the hyperbole. For them, the ends justifies the means, even if the means undermines its own validity (i.e., the politicization of science means that science will become political).

  • I was taught “the scientific method” in 5th grade. Funny how climate scientists seem to have never heard of it…

  • Scientist

    papertiger – that is a deeply ridiculous characterisation. By the same logic, chemistry is the study of what humans expect chemicals to do in the future based on past experience, astronomy is the study of what humans expect stars, planets and galaxies to do in the future based on past experience, medicine is the study of what humans expect human bodies to do in the future based on past experience, and so on.

    Deniers spend a lot of time thinking up new ways to try and denigrate climate science, but they don’t often come up with something as immature and flawed as that.

    William Squire – funny how you obviously have no clue at all about what climate scientists do.

  • An Inquirer

    Perhaps the following point is obvious, but I did not notice it being explicitly addressed. A key reason why GCM are able to achieve great backcasts is the ability of aerosols to act as dummy variables in the models. Perhaps several of you have had similar experience as I have had in building and / or evaluating models. You can get great fit by inserting dummy variables, and as a result, there is great emphasis on the impact of the drivers on the dependent variables. (Of course, if you cannot come up with legitimate values of the dummy variables in the future, then your forecasts will be unreliable.)
    I have spent considerable time examining the sources of aerosol values that GCMs use. First, I am impressed with the amount of data that is out there on this issue. However, at the same time, aerosols are a gold mine of opportunities to insert dummy variables and hypothesized relationships. One can easily choose values that will produce a great fit and be misled into believing that other drivers (such as CO2) are responsible for changes in the dependent variable.
    Researchers should be allowed the ability to update and refine models based upon best available data. For example, some early GCMs forecasted greater warming at both poles. However, as new data of observations are inserted, models now generally forecast warming at the North Pole and delayed warming at the South Pole. Of course, this dichotomy is driven by recent observations forcing relationships to change inside the models, but it probably would be more difficult for the dichotomy to emerge if not for the presence of “dummy variables.”
    The author of the blog is quite generous in comparing Hansen’s 1988 forecasts with GISS. Only in climatology would it be acceptable for a forecaster be allowed to validate his model results with his estimate of observed values — when his estimate of observed values are not reproducible nor verifiable. (To some degree his estimate of observed value are verified by its similarity to HadCrut estimate values, but HadCrut also has issues with UHI which apparently have been examined even less than in GISS. And we can get into comparison with satellite data which I am not going to address in this post.)

  • Scientist

    If the key to hindcasting is assigning arbitrary values to aerosol forcing, then how come the predictions made immediately after Pinatubo erupted about the effect on global temperature of the aerosols emitted were so accurate?

  • Mongo

    I rarely post comment in any of the blogs I visit, but this one grabbed me. While not trained in climatology or any related disciplne, I am well educated and read in general science. Some material is pretty dense and a stretch, but in general I understand it’s basics.

    I cherish healthy debate, as both sides sharpen their supporting information to justify their position. When peer reviewed studies/papers/etc are used as they are now, without the rigor of open and honest debate, we end up with this situation. In general, this resembles a house of cards built on sand when viewed with any kind of open mind.

    After years of watching this…I’m not sure what to term it…debacle ?…come to where it’s at, all I can think of is two words. The below 8 lines describe it.

    Illusion of Invulnerability: Members ignore obvious danger, take extreme risk, and are overly optimistic.

    Collective Rationalization: Members discredit and explain away warning contrary to group thinking.

    Illusion of Morality: Members believe their decisions are morally correct, ignoring the ethical consequences of their decisions.

    Excessive Stereotyping:The group constructs negative sterotypes of rivals outside the group.

    Pressure for Conformity: Members pressure any in the group who express arguments against the group’s stereotypes, illusions, or commitments, viewing such opposition as disloyalty.

    Self-Censorship: Members withhold their dissenting views and counter-arguments.

    Illusion of Unanimity: Members perceive falsely that everyone agrees with the group’s decision; silence is seen as consent.

    Mindguards: Some members appoint themselves to the role of protecting the group from adverse information that might threaten group complacency.

    If you haven’t guessed those two words by now, the above describe group think. It seems to be epidemic.

    BTW – I was open to the idea of AGW/ACC or whatever term is being thrown around, but as time has passed I am convinced that man-made CO2 has, at best, an insiginificant effect on climate. Prove otherwise and I will change my mind.