Cross-posted at Coyote Blog
Wow, suddenly skepticism, and even outright harsh criticism, of peer-reviewed work is OK, as long as it is not in climate I suppose.
On Thursday, Dec. 2, Rosie Redfield sat down to read a new paper called “A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus.” Despite its innocuous title, the paper had great ambitions. Every living thing that scientists have ever studied uses phosphorus to build the backbone of its DNA. In the new paper, NASA-funded scientists described a microbe that could use arsenic instead. If the authors of the paper were right, we would have to expand our….
As soon Redfield started to read the paper, she was shocked. “I was outraged at how bad the science was,” she told me.
Redfield blogged a scathing attack on Saturday. Over the weekend, a few other scientists took to the Internet as well. Was this merely a case of a few isolated cranks? To find out, I reached out to a dozen experts on Monday. Almost unanimously, they think the NASA scientists have failed to make their case. “It would be really cool if such a bug existed,” said San Diego State University’s Forest Rohwer, a microbiologist who looks for new species of bacteria and viruses in coral reefs. But, he added, “none of the arguments are very convincing on their own.” That was about as positive as the critics could get. “This paper should not have been published,” said Shelley Copley of the University of Colorado.
The article goes on to describe many potential failures in the methodology. None of this should be surprising — I have written for years that peer-review is by no means proof against bad science or incorrect findings. It is more of an extended editorial process. The real test of published science comes later, when the broader community attempts to replicate results.
The problem in climate science has been that its proponents want to claim that having research performed by a small group of scientists that is peer-reviewed by the same small group is sufficient to making the results “settled science.” Once published, they argue, no one (certainly not laymen on blogs) has the right to criticize it, and the researchers don’t (as revealed in the Climategate emails) have any obligations to release their data or code to allow replication. This is just fresh proof that this position is nuts.
The broken climate science process is especially troubling given the budgetary and reputational incentives to come out with the most dramatic possible results, something NASA’s James Hansen has been accused of doing by many climate skeptics. To this end, consider this from the bacteria brouhaha. First, we see the same resistance to criticism, trying to deflect any critiques outside of peer-reviewed journals
“Any discourse will have to be peer-reviewed in the same manner as our paper was, and go through a vetting process so that all discussion is properly moderated,” wrote Felisa Wolfe-Simon of the NASA Astrobiology Institute. “The items you are presenting do not represent the proper way to engage in a scientific discourse and we will not respond in this manner.”
WTF? How, then, did we ever have scientific process before peer-reviewed journals appeared on the scene?
But Jonathan Eisen of UC-Davis doesn’t let the scientists off so easily. “If they say they will not address the responses except in journals, that is absurd,” he said. “They carried out science by press release and press conference. Whether they were right or not in their claims, they are now hypocritical if they say that the only response should be in the scientific literature.”
Wow, that could be verbatim from a climate skeptic in the climate debate.
And finally, this on incentives and scientific process:
Some scientists are left wondering why NASA made such a big deal over a paper with so many flaws. “I suspect that NASA may be so desperate for a positive story that they didn’t look for any serious advice from DNA or even microbiology people,” says John Rothof UC-Davis.