I thought this article in the NY Times about the failure of models to accurately predict the progression of swine flu cases was moderately instructive.
In the waning days of April, as federal officials were declaring a public health emergency and the world seemed gripped by swine flu panic, two rival supercomputer teams made projections about the epidemic that were surprisingly similar — and surprisingly reassuring. By the end of May, they said, there would be only 2,000 to 2,500 cases in the United States.
May’s over. They were a bit off.
On May 15, the Centers for Disease Control and Prevention estimated that there were “upwards of 100,000” cases in the country, even though only 7,415 had been confirmed at that point.
The agency declines to update that estimate just yet. But Tim Germann, a computational scientist who worked on a 2006 flu forecast model at Los Alamos National Laboratory, said he imagined there were now “a few hundred thousand” cases.
We can take at least two lessons from this:
- Accurately modeling complex systems is really, really hard. We may have hundreds of key variables, and changes in starting values or assumed correlation coefficients between these variables can make enormous differences in model results.
- Very small changes in assumptions about processes that compound or have exponential growth make enormous differences in end results. I think most people grossly underestimate this effect. Take a process that starts at an arbitrary value of “100” and grows at some growth rate each period for 50 periods. A growth rate of 1% per period yields an end value of 164. A growth rate just 1 percentage point higher of 2% per period yields a final value of 269. A growth rate of 3% yield a final value of 438. In this case, if we miss the growth rate by just a couple of percentage points, we miss the end value by a factor of three!
Bringing this back to climate, we must understand that the problem of forecasting disease growth rates is grossly, incredibly more simple than forecasting future temperatures. These guys missed the forecast my miles of a process that is orders of magnitude more amenable to forecasting than is climate. But I am encouraged by this:
Both professors said they would use the experience to refine their models for the future.
If only climate scientists took this approach to new observations.