Monthly Archives: December 2010

Suddenly, Skepticism of Peer-Reviewed Science is OK

Cross-posted at Coyote Blog

Wow, suddenly skepticism, and even outright harsh criticism, of peer-reviewed work is OK, as long as it is not in climate I suppose.

On Thursday, Dec. 2, Rosie Redfield sat down to read a new paper called “A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus.” Despite its innocuous title, the paper had great ambitions. Every living thing that scientists have ever studied uses phosphorus to build the backbone of its DNA. In the new paper, NASA-funded scientists described a microbe that could use arsenic instead. If the authors of the paper were right, we would have to expand our….

As soon Redfield started to read the paper, she was shocked. “I was outraged at how bad the science was,” she told me.

Redfield blogged a scathing attack on Saturday. Over the weekend, a few other scientists took to the Internet as well. Was this merely a case of a few isolated cranks? To find out, I reached out to a dozen experts on Monday. Almost unanimously, they think the NASA scientists have failed to make their case. “It would be really cool if such a bug existed,” said San Diego State University’s Forest Rohwer, a microbiologist who looks for new species of bacteria and viruses in coral reefs. But, he added, “none of the arguments are very convincing on their own.” That was about as positive as the critics could get. “This paper should not have been published,” said Shelley Copley of the University of Colorado.

The article goes on to describe many potential failures in the methodology.  None of this should be surprising — I have written for years that peer-review is by no means proof against bad science or incorrect findings.  It is more of an  extended editorial process.  The real test of published science comes later, when the broader community attempts to replicate results.

The problem in climate science has been that its proponents want to claim that having research performed by a small group of scientists that is peer-reviewed by the same small group is sufficient to making the results “settled science.”  Once published, they argue, no one (certainly not laymen on blogs) has the right to criticize it, and the researchers don’t (as revealed in the Climategate emails) have any obligations to release their data or code to allow replication.   This is just fresh proof that this position is nuts.

The broken climate science process is especially troubling given the budgetary and reputational incentives to come out with the most dramatic possible results, something NASA’s James Hansen has been accused of doing by many climate skeptics.  To this end, consider this from the bacteria brouhaha.  First, we see the same resistance to criticism, trying to deflect any critiques outside of peer-reviewed journals

“Any discourse will have to be peer-reviewed in the same manner as our paper was, and go through a vetting process so that all discussion is properly moderated,” wrote Felisa Wolfe-Simon of the NASA Astrobiology Institute. “The items you are presenting do not represent the proper way to engage in a scientific discourse and we will not respond in this manner.”

WTF?  How, then, did we ever have scientific process before peer-reviewed journals appeared on the scene?

But Jonathan Eisen of UC-Davis doesn’t let the scientists off so easily. “If they say they will not address the responses except in journals, that is absurd,” he said. “They carried out science by press release and press conference. Whether they were right or not in their claims, they are now hypocritical if they say that the only response should be in the scientific literature.”

Wow, that could be verbatim from a climate skeptic in the climate debate.

And finally, this on incentives and scientific process:

Some scientists are left wondering why NASA made such a big deal over a paper with so many flaws. “I suspect that NASA may be so desperate for a positive story that they didn’t look for any serious advice from DNA or even microbiology people,” says John Rothof UC-Davis.

A Really Bad Idea

I know lots of your disagree with me on this, but it needs to be said.  I have had people argue that , well, its about management of public funds, but compared to what the average university wastes, this is trivial.  The funds management issue is just window dressing, in my view, for people looking for a heaping helping of retribution. Cross posted from Coyote Blog

Regular readers will have no doubts about my skepticism of the theory of catastrophic man-made global warming.  In particular, in these pages and at Coyote Blog, I have repeatedly criticized the details of Michael Mann’s work on the hockey stick.  I won’t repeat those issues today, though some of the past articles are indexed here.  Or watch my video linked to the right, it has plenty of stuff about the hockey stick.

That being said, efforts by Republicans in Virginia to bring legislative or even criminal action against Mann for his work when he was at the University of Virginia is about the worst idea I have heard in quite some time.  Though nominally about forcing public disclosure (something I am always in favor of from state entities) the ultimate goal is to drag Mann into court

Cuccinelli has said he wants to see whether a fraud investigation would be warranted into Mann’s work, which showed that the earth has experienced a rapid, recent warming

[As an aside, this is actually NOT what Mann's hockey stick work purports to show.  The point of the hockey stick is to make the case that historic temperatures before 1850 were incredibly stable and flat, and thus recent increases of 0.6-0.8C over the last 150 years are unprecedented in comparison.   His research added nothing to our knowledge about recent warming, it was on focused on pre-industrial warming.   The same folks that say with confidence the science is settled don't even understand it].

For those frustrated with just how bad Mann’s work is and upset at the incredible effort to protect this work from criticism or scrutiny by hiding key data (as documented in the East Anglia climategate emails), I know it must feel good to get some sort of public retribution.  But the potential precedent here of bringing up scientists on charges essentially for sloppy or incorrect work is awful.

Bad science happens all the time, completely absent any evil conspiracies.  Human nature is to see only the data that confirms ones hypotheses and, if possible, to resists scrutiny and criticism.  This happens all the time in science and if we started hauling everyone into court or into a Senate committee, we have half of academia there  (and then likely the other half when the party in power changed).  Team politics are a terrible disease and the last thing we need is to drag them any further into science and academia.

Science will eventually right itself, and what is needed is simply the time and openness to allow adversarial scrutiny and replication within academia to run its course.  Seriously, are we next going to drag the cold fusion guys in to court?  How about all the folks in the geology field that resisted plate tectonics for so long.  Will we call to account the losers in the string theory debate?

If legislators want to help, they can

  • Make sure there are standards in place for archiving and public availability of any data and code associated with government funded research
  • Improve the governments own climate data management
  • Ensure that state funding is distributed in a way to support a rich dialog on multiple sides of contested scientific issues.