Climate Science Process

The Program Code – Perhaps Far More Damning than the Emails

19 Comments

A reader wrote me that the comments found in the Hadley CRU program code are possibly far more damning than the emails, and in fact this appears to be the case given these excerpts at Anthony Watts’ site.

In the past, I have written that as an experienced modeller, I am extremely suspicious when anyone’s models very closely match history.  This is a common modelers trick – use various plugs and fudge factors and special algorithms to force the model to match history better (when it is used to “back-cast”) and people will likely trust the model more when you use it to forecast.   For a variety of reasons, I have been suspicious this was the case with climate models, but never could prove it.  One example from the link above

Looking back over history, it appears the model is never off by more than 0.4C in any month, and never goes more than about 10 months before re-intersecting the “actual” line.  Does it bother anyone else that this level of precision is several times higher than the model has when run forward?  Almost immediately, the model is more than 0.4C off, and goes years without intercepting reality.

Now we are closer, with programming code comments in the various climate programs that say things like this (from the code that apparently does some of the tree ring histories)

. FOIA\documents\osborn-tree6\mann\oldprog\maps12.proFOIA\documents\osborn-tree6\mann\oldprog\maps15.proFOIA\documents\osborn-tree6\mann\oldprog\maps24.pro ; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions
; of growing season temperatures. Uses "corrected" MXD - but shouldn't usually
; plot past 1960 because these will be artificially adjusted to look closer to
; the real temperatures.

or this

  • FOIA\documents\osborn-tree6\briffa_sep98_d.pro;mknormal,yyy,timey,refperiod=[1881,1940]
    ;
    ; Apply a VERY ARTIFICAL correction for decline!!
    ;
    yrloc=[1400,findgen(19)*5.+1904]
    valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
    2.6,2.6,2.6]*0.75 ; fudge factor
    (...)
    ;
    ; APPLY ARTIFICIAL CORRECTION
    ;
    yearlyadj=interpol(valadj,yrloc,x)
    densall=densall+yearlyadj

The link above has 30+ similar examples.  The real insight will be when folks like Steve McIntyre and his readers start digging into the code and replicating it — then we will see what it actually does and what biases or plugs or overrides are embedded.  Stay tuned.

  • John Anderson

    Maybe a better title would have been, “Actions Speak Louder Than Words”, as it’s plain from what was done *to* the data than what Hadley CRU said was done *with* the data.

  • All these code-comments I’ve read here and on other sites are absolutely damning, yet AGW alarmists and their propagandists in the media keep clapping their hands to their ears and shouting “we can’t hera you!”

    I wonder at what point they start blaming this on a conspiracy by the oil and coal companies.

  • NikFromNYC

    Here is a chart of the code you posted:

    http://i49.tinypic.com/m9vcxv.jpg

  • I just contacted PSU Earth and Mineral Science Dept. and asked if they will investigate the program codes indicating academic fraud. Their statement about Dr. Mann and the controversy only addresses “the emails”. As an alumnus, I will withhold any contributions to the University pending a thorough investigation of the matter. “Hide the decline!”

  • Joseph A Olson, PE

    Did not take long for this conspiracy to be foisted on the oil and coal industires. The UK Guardian climate reporter and mega-warmist, George Monbiot at first is outraged and apologetic. Then he continues to support the failed AGW hypothesis by promoting, in the same article, his book “Heat” and “Heat is ON” by Gelbspan and “Climate Cover-UP” by Hoggan/Littlemore. The Hadley hacking is only the tip of this ugly iceberg and far more revelations will soon follow. The big scientific question now is ‘who’ orchestrated this fraud and ‘why’ was this fraud necessary. To answer those questions you must study the history of ‘human caused global warming’ as explained in “Strange Tale of Green House Gas Gang”, posted at ClimateRealist.com. It is all part of the Progressive/Eugenic plan. They just counted on this happening before the end of a natural warming cycle. Sorry (not).

  • Nik

    As a lawyer turned journalist I am confused by that these cover up attempts were meant to achieve. There seems to be no reasonable explanation.

    Not releasing data and code re scientific matters which affect every inhabitant of the earth is strange. What could have been the motive?

    There is also the failure of the scientific lterature to demand that all data and codes should be submitted so that it can be challenged. Arguably the editors of the journals bear greater responsibility than the submitting authors, for the editors failed in their mission and their public duty.

  • I’m guessing that it will be a while before SM and the gang can replicate anything with any rigor. From the looks of it, the data is a disaster, and the code a close kin. The programmer (Harry) responsible for the code and comments above spent three years trying to make it all work before apparently giving up in frustration, given the state of the data he was dealing with. It is going to take a concerted effort on Steve’s and everyone’s parts to try to make sense of it all.

    It would be nice if an independent source could fund some real programmers, possibly headed by Steve, and have them dedicate themselves full-time to the project. It would be an enormous contribution to the honest efforts to bring transparency and rigor to the whole long-term climate studies effort. Honestly, it would probably be best if a dual-track approach were taken. First, clean the data up so that it can be properly analyzed. Second, one group should focus on “re-efforting” the original Fortran code. Bringing that to a steady-state–with all bugs, kludges, fudges, errors, etc. completely documented–would be there primary task. The point would be to a) document all of the issues with the code as it already exists, and b) to fix it. The other group should take a more modern approach. Using well-known, and accepted, statistical methodologies (using R, or whichever data analysis tool seems most appropriate), they should attempt to squeeze any patterns that may exist out of the datasets that are there.

    Who knows. Maybe the AGW-alarmists are right after all. But there is absolutely no way of knowing until the CRU data is completely cleaned up and analyzed anew. I count myself amongst the skeptics, but I’m by no means so dug in that I won’t change my mind if the facts change. This would be a big step towards bringing clarity to the situation.

  • Come on, everyone, let’s cut to the chase and blame George W. Bush.

  • Ram300

    “As a lawyer turned journalist I am confused by that these cover up attempts were meant to achieve. There seems to be no reasonable explanation.
    Not releasing data and code re scientific matters which affect every inhabitant of the earth is strange. What could have been the motive?”

    To hide the truth from public view?

  • anon

    The debate about the contents of the HARRY_READ_ME.txt the validity of the programming and modelling techniques is something only experts and argue over.

    However, the lay person only needs to know this about the file which they can verify for themselves — specifically he HARRY_READ_ME.txt file) is a THREE YEAR journal of a CRU programmer describing everything he tried with the data and models in an attempt to reproduce existing results CRU had published. Comments in the file make it clear that “HARRY” tried FOR THREE YEARS to recreate CRU’s published results AND FAILED.

    Do you see the REAL significance of this because it is absolutely fatal to the credibility of anything CRU has produced.

    What we have here is a documented THREE year effort by a CRU programmer, who had access to all the data, access to all the code, access to all the people who developed the code and the models and still HE could still NOT duplicate CRU’s OWN results. If he can’t it simply means the CRU’s results cannot be reproduced even by themselves and so there is no point anyone else even trying — CRU themselves have proven it’s a waste of time and so they themselves have proven their own results are plain rubbish. That means any “peer reviewed” document CRU produced along with any other papers that cited the CRU papers are based on data the CRU themselves can’t verify.

    Besides, the absolutly sorry state of affairs in the data handling and software managment the HARRY_READ_ME.txt reveals, the utter and total mess of CRU data and software this document reveals is WHY CRU has not released its data and model software.

    Given the CRU is one of the most, if not the most cited sources of climate data — upon which trillions of dollars of economic policy is being set, the importance of what the HARRY_READ_ME.txt file reveals becomes scary.

    A very nice layman’s summary of some of the issues in the HARRY_READ_ME.txt can eb found here

    http://www.devilskitchen.me.uk/2009/11/data-horribilis-harryreadmetxt-file.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+TheDevilsKitchen+%28The+Devil%27s+Kitchen%29&utm_content=Netvibes

  • Cameron

    To the Climatic Research Scientists of the University of East Anglia, et.al.:

    Stop hiding behind the “illegally” obtained defense. This was the action of a whistleblower, someone inside who had direct knowledge of this fraud. I have a feeling as with all good scandals, the first discovery of impropriety is only the tip of the iceberg!

    Hey, Al, what now…..what’s the next global crisis you can bank on?

  • anonymous coward

    I enjoyed the following comment from RealClimate:

    “Release all the data and all the models. Let everyone, even big oil and king coal, take a crack at it. What is there to hide? If the skeptics play games like the proponents do it will only weaken their case.

    Trying to argue it out here is a waste of time.

    A co-worker once got a theory in his head on what moved stock prices. He build model, collected data, and refined and refined until the backtesting was perfect. He sent the output to me. I said fine, send me the models and data sets so I can check it out. He said he would not because they were proprietary. He still wanted my help selling is “product.” That really is the situation the CRU people are in. They simply have to toss open ALL the data, models and code. If it is correct it will stand if not it will not.

    BTW, his model apparently did not account properly for LIBOR spread gaps circa 2008. LOL.
    Google LTCM if you want another example of too much faith in proprietary models.”

  • Cameron

    The “proprietary” arguement doesn’t work here. This is data that was obtained through government sponsored research, used to formulate public policy, and is the basis for trillions of dollars of policy that is yet to be implemented.

    Furthermore, how can science test itself without full disclosure?

    We are not talking about Apple creating a new product, or National defense issues, this is data that should be available to the entire scientific community for validation.

  • sabril

    It’s a minor point, but I thought I would mention that one of the first things you learn in a computer programming class is to not use “magic numbers.” Numbers like 1400 and 19 should not be appearing in the middle of a block of code because they make the code difficult to understand and change. Instead, you set up a constant with a name like Num_Years_of_NZ_Data and set it equal to 1400 or whatever.

    Given this sort of sloppiness (which would get an F in any programming class) it’s hardly a surprise that the program is a complete nonsensical irreproducible mess.

  • RM

    Cameron, haven’t you heard; that CO2 causes global cooling? Wait for it.

  • kbingh

    My guess is they did not want to use professional programmers because they needed to keep the code hidden except within a small group of trusted colleagues. Most likely it was the scientists themselves that produced the code and they only had a rudimentary experience in programming thus the sloppiness of the code. It is astounding that the premier global warming institution would keep such shoddy code and records for that matter. The only explanation is they had something to hide and could not risk trusting real programmers.

  • Danram

    Nik wrote: “Not releasing data and code re scientific matters which affect every inhabitant of the earth is strange. What could have been the motive?”

    The answer is pretty simple, Nik. This was never about science. This was always about money, power and ideology. The political left has seized upon AGW as the way to force a multi-trillion dollar redistribution of the world’s economic resources under the direction of governments around the world. This is a leftist’s wet dream.

    Also, you have to consider who has vested interests in AGW. If there is no AGW, then untold millions of reserach dollars dry up. If there is no AGW, then entrenched, well-funded bureaucracies like the IPCC have no reason to exist. Ergo, there must be global warming, even if the data have to be fudged to get it.

  • ZelieDad

    The motive for the fraud, as for all fraud, is money and power.
    State research funding in the millions $ has poured in (and been wasted) on this nonsense.
    Now we are all poorer because the resources could have been spent on honest science. Or not spent at all by government.

    It is not just the researches that were dishonest. The granters of the government resources are the greater problem. If government were not passing out money to fund the fraud that would increase their power, there would have been no motive for the fraud.

    Dishonest scientists on their own could not have carried the fraud for a decade. It took the state for that.

  • Poster NukemHill thinks that the situation is reparable: “First, clean the data up so that it can be properly analyzed. Second, one group should focus on “re-efforting” the original Fortran code. Bringing that to a steady-state–with all bugs, kludges, fudges, errors, etc. completely documented–would be [their] primary task. The point would be to a) document all of the issues with the code as it already exists, and b) to fix it. The other group should take a more modern approach. …”

    I agree that the code needs a thorough going over, and the mistakes, kludges, etc., documented for all to see. On the other hand, given what gets reported (and not reported) by the weather stations, the fact that they move around, and, more fundamentally, the fact that none of them were designed to be part of a world-wide thermometer accurate to a tenth of a degree C., it’s not clear what can be salvaged. The whole gridding thing is a nightmare, and it results from the fact that there was no experimental design; the numbers just come in. Too much room for judgment calls; too much to argue about.