Hot on the heels of last months reporting of a discrepancy in the ocean surface temperatures, a new paper in Nature (by Domingues et al, 2008) reports on the revisions of the ocean heat content (OHC) data – a correction required because of other discrepancies in measuring systems found last year.
Before we get to the punchline though, it’s worth going over the saga of the OHC trends in the literature over the last 8 years. In 2001, Syd Levitus and colleagues first published their collation of ocean heat content trends since 1950 based on archives of millions of profiles taken by oceanographic researchers over the last 50 years. This showed a long term upward trend up, but with some very significant decadal variability – particularly in the 1970s and 1980s. This long term trend was in reasonable agreement with model predictions, but the decadal variability was much larger in the observations.
As in all cases where there is a data-model mismatch, people go back to both in order to see what might be wrong. One of the first suggestions was that since the spatial sampling became much coarser in the early part of the record, there might be more noise earlier on that didn’t actually reflect a real ocean-wide signal. Sub-sampling the ocean models at the same sampling density as the real observations did increase the decadal variability in the diagnostic but it didn’t provide a significantly better match (AchutaRao et al, 2006).
Other problems came up when trying to tally the reasons for sea level rise (SLR) over that 50 year period. Global SLR is a product of (in rough order of importance) ocean warming, land ice melting, groundwater extraction/dam building, and remnant glacial isostatic adjustment (the ocean basins are still slowly adjusting to the end of the last ice age). The numbers from tide gauges (and later, satellites) were higher than what you got by estimating each of those terms separately. (Note that the difference is mainly due to the early part of the record – more recent trends do fit pretty well). There were enough uncertainties in the various components so that it wasn’t obvious where the problems were though.
Since 2003, the Argo program has seeded the oceans with autonomous floats which move up and down the water column and periodically send their data back for analysis. This has at last dealt with the spatial sampling issue (at least for the upper 700 meters in the ocean – greater depths remain relatively obscure). Initial results from the Argo data seemed to indicate that the ocean cooled quite dramatically from 2003 to 2005 (in strong contradiction to the sea level rise which had continued) (Lyman et al, 2006). But comparisons with other sources of data suggested that this was only seen with the Argo floats themselves. Thus when an error in the instruments was reported in 2007, things seemed to fit again.
In the meantime however, calibrations of the other sources of data against each other were showing some serious discrepancies as well. Ocean temperatures at depth are traditionally made with CTDs (a probe that you lower on line that provides a continuous temperature and salinity profile), Nansen bottles (water samples that are collected from specified depths) or XBTs (eXpendable bathy-thermographs) which are basically just thrown overboard. CTDs are used over and again and can be calibrated continuously to make sure their pressure and temperature measurements are accurate, but XBTs are free falling and the depths from which they are reporting temperatures needs to be estimated from the manufacturers fall rate calculations. As the mix of CTDs, bottles, XBTs and floats has changed over time, minor differences in the bias of each methodology can end up influencing the trends.
(If this is all starting to sound very familiar to those who looked into the surface stations or sea surface temperature record issues, it is because it is the same problem. Almost all long historical climate records were not collected with the goal of climate in mind.)
In particular, analysis (or here) of the XBT data showed that it was biased warm compared to the CTDs, and that this bias changed over time, and was dependent on the kind of XBT used (deep versus shallow). Issues with the fall rate calculation were well known, but corrections were not necessarily being applied appropriately or uniformly and in some cases were not correct themselves. The importance of doing the corrections properly has been subject to some ongoing debate (for instance, contrast the presentations of Levitus and Gourteski at this meeting earlier this year).
So where are we now? The Domingues et al paper that came out yesterday, along with a companion paper from essentially the same group (in press at Journal of Climate) have updated the XBT corrections and dealt with the Argo issues, and….
… show a significant difference from earlier analyses (the new analysis is the black line). In particular, the difficult-to-explain ‘hump’ in the 1970s has gone (being due to the increase in warm-biased XBTs at that time). The long term trend is slightly higher, while the more recent trends are slightly lower. Interestingly, while there still decadal variability, it is much more obviously tied to volcanic eruptions than was previously the case. Note that this is a 3-year smooth, so the data actually goes to the end of 2004.
So what does this all mean? The first issue is tied to sea level rise. The larger long term trend in ocean warming reported here makes it much easier to reconcile the sea level estimates from thermal expansion with the actual rises. Those estimates do now match. But remember that the second big issue with ocean heat content trends is that they largely reflect the planetary radiative imbalance. This imbalance is also diagnosed in climate models and therefore the comparison serves as an independent check on their overall consistency. Domingues et al show some comparisons with the IPCC AR4 models in their paper. Firstly, they note that OHC trends in the models that didn’t use volcanic forcings are consistently higher than the observations. This makes sense of course because each big eruption cools the ocean significantly. For the models that did include volcanic forcings (including the model we used in Hansen et al, 2005, GISS-ER), the match is much better:
(Note that the 3-year smoothed observations are being compared to annual data from the models, the lines have been cut off at 1999, and everything is an anomaly relative to 1961). In particular, the long term (post 1970) observational trends are now a better match to the models, and the response to volcanoes is seen clearly in both. The recent trends are a little lower than reported previously, but are still within the envelope of the model ensemble. One interesting discrepancy is noted however – the models have a slight tendency to mix down the heat more evenly than in the observations.
This isn’t going to be the last word on OHC trends, and different groups are going to be publishing their own versions of this analyses relatively soon and updates to the most recent years are still forthcoming. But the big picture is that ocean heat content has indeed been increasing in recent decades, just like the models said it should.
Corrections to observations are made without regard to the theory.
I don’t think this is generally true or even possible, since observers are generally believers in the theory in question, and thus potentially influenced in any subjective aspect of the observation.
Sometimes the prevailing theory/hypotheses will be used in an effort to formulate corrections, reject outlier data, etc. Whenever this is done is seems the author should provide the rationale to avoid criticism for a type of circular reasoning.
Rejecting outlier data without a lot of careful analyses and very solid justiication proving that the data was measured incorrectly and not because it doesn’t fall where it “should” is a big no-no in science. And scientists actually check each other for that kind of thing. That’s part of what peer review is for, but more than that if you earn a reputation as someone who manipulates data as a means to prove your pet theory or model, funding starts to dry up, your papers start to get rejected and basically people stop trusting your work and scientific integrity. Scientists are pretty blunt and pretty cruel that way. They have to be.
RE: #101:
It certainly is possible for corrections to be made without regard to theory! Suppose you discover that every temperature sensor made in a particular shop in a particular week under-reports temperature by 1 degree. You verify that by testing a sample of them in the lab. Theories of climate have nothing to do with that.
Now you use serial numbers to find all those particular temperature sensors that were deployed, and increase your records of their observed temperatures by 1 degree. You do so for _all_ those sensors, regardless of whether their observations to date have agreed with your theory.
Any data points that, when uncorrected, exceeded your theory’s prediction, now exceed your theory’s prediction even more due to the corrective addition of 1 degree. You do _not_ refrain from applying the correction to those data points. You apply the correction based purely on criteria having nothing to do with your theory, and you see whether your theory comes out better or worse.
This sort of correction happens all the time, in all branches of empirical science. As Figen Mekik pointed out in #102, an essential part of science is an active and sometimes vicious gang of your peers who are all too willing to point out your mistakes and biases.
Re #100 [Joseph Hunkins] “Rightly or wrongly, I think Hansen’s dramatic interpretations of the threats from AGW are going to be the key media “reference points” going forward.”
I certainly hope so – and if you’re right, that in itself will show how mistaken those who are arguing that Hansen has made a tactical blunder are.
101. Mr. Hunkins makes a sweeping generalization about how scientists view data versus theory and how they treat raw data. I would be very interested in an analyisis of this in a specific area of climate science or oceanography. Where do you see this happening? Please be specific and provide references.
Re: 102. “Rejecting outlier data without a lot of careful analyses and very solid justiication proving that the data was measured incorrectly and not because it doesn’t fall where it “should” is a big no-no in science.”
That’s not strictly speaking true. Any time an experiment has a heavy tailed error distribution it can result in measurements you might as well throw away even though the measurement process was as correct as it can be.
A lot of people used to think that naturally occurring error distributions would be close to Gaussian, based on the central limit theorem, and that actually covers a lot of situations. But it does not cover all situations.
Ultimately, the question of whether a datum should be taken at face value or not is one for robust statistical methods. One reasonable approach is to apply only methods of inference and estimation which are smooth in the Prokhorov metric. This roughly and qualitatively means that the answers you get are insensitive to a small proportion of gross errors in the data and small errors in the remainder of the data. In the past few decades many very useful ways to obtain this sort of property in inference and estimation have been devised.
For example, it was well known for a long time that the median was much less susceptible to error than the mean, but the median was also a much less efficient estimate – in other words you payed a big penalty from the median “throwing away” so much of your data. But it has also long been known that other estimates which are equally robust, but which are more efficient (for example the Hodges-Lehmann estimate (median of pairwise means) is very efficient if the error distribution is symmetric).
I suspect that one of the main reason people in the climate field do not reflexively only use Prokhorov smooth methods all the time (as we do in my field of finance) is that they know they might have to explain their work to an unsophisticated audience (such as government officials).
Joseph Hunkins, Figen Mekik is correct–you would never reject data simply because it did not fit a particular theory. In so doing you might be throwing away a chance for a very important discovery–and scientists are interested in nothing more than this.
While it is true that theory can introduce bias into an analysis, they way this happens is much more subtle than this–e.g by influencing our ideas of what can be measured or perhaps looking for errors in outliers more than in data that fit the theory. However, a good scientist can and will try to check for these biases–and if he doesn’t his peers certainly will, and not kindly.
There are ways of checking data for self-consistency without reference to a theory. Such methods can also be used to detect bogus data. As an example, suppose you have a moderate-sized sample, and you have a single outlier. Is it a mistake, or is it physics? One way to check is to gather more data and see if you get more outliers and whether they start to resemble a feature–e.g. a second mode, a thick tail, etc. If you can’t find any more data, you can see if you can find “similar” data (generated by at least some of the same processes) and look that way. In addition, you can look at the order plots, do bootstrapping… The data analysis toolbox is huge and crammed to the gills with interesting tools.
RE: 107 and Joseph Hunkins:
Probably, Joseph, you are applying your knowledge of science as it is usually taught in high school and even some undergraduate college classes, as the purest form of experimentation. That is, if your hypothesis is not supported by your pure experiment, you must accept that result as absolute, give up that hypothesis and the theory that created it, modify your theory so it produces a different hypothesis, and then run a brand new experiment.
The actual practice of science doesn’t work in only that way. It works differently not because scientists don’t rigorously follow the scientific method as my previous paragraph described it. Rather, science never has and never should work only as my previous paragraph described it. “Normative” (prescriptive–ideal) decision making uses every scrap of information that is available. The form of science that my previous paragraph describes is only one small part of the ideal scientific method, and it is not possible at all when dealing with irreproducible historic data that you’ve already explored thoroughly.
A gentle introduction to the messiness of real science is the topic of “quasiexperimentation.” I was brought up on Cook and Campbell’s book having that title, but there are lots of more recent and brief overviews. By the way, that book is not about climatology. It is about research methodology in general, though its examples are from behavioral science. Climatology in no way gets an exemption from scientific method. All these explanations by me and other bloggers are not excuse-making for climatology. We’re explaining how science does and should work.
My comment was not clear enough above. I am not suggesting that observations are generally unreasonably influenced by theory. I would suggest that that happens sometimes, and is a potentially serious problem. My point is that divorcing theory and observation is not as easy as was suggested, and scientists sometimes fail to do this. A good example where this is *routinely* done is “creation science” where accredited scientists reject overwhelming evidence simply because it is incompatible with the key features of creation “science”.
Ray I think the notion that data is never rejected because it is incompatible with theory is far too optimistic. More importantly, there is often some room for subjectivity when applying corrections and even in initial measurements (e.g. reading a thermometer).
Paul my point was not as dramatic as you seem to think, though observational and correction bias would make a fantastic (and unfundable?) study,
Andrew, If you look at what Figen said, I think you will find that it is consistent with your more detailed explanation. I am grateful to you for emphasizing the statistical nature of outlier identification, though. It’s a problem I often confront in my day job.
Re Gussbob @94: I suspect I’ll regret the resulting thread diversion, but exactly what change in insolation do you propose that is capable of producing a change of 22 degrees in 50 years?
Jan Rooth @96 was more specific than I in suggesting sudden and massive release from methane clathrates. His comment on short duration stems from the fact that methane is a potent greenhouse gas, but reduces to less potent CO2 and H2O in the atmosphere fairly rapidly.
RE 111, I think the life span for CH4 (of which there are many gigtons in the Arctic region just waiting to be released with the warming that’s already in the pipeline) is about 10 years. With our rapid warming and great CH4 releases compounding, I think this could spiral the warming way out of control in a positive feedback fashion.
See David Archer’s post, https://www.realclimate.org/index.php/archives/2005/12/methane-hydrates-and-global-warming/langswitch_lang/po
110. “Andrew, If you look at what Figen said, I think you will find that it is consistent with your more detailed explanation.”
No, I won’t. The part that is definitely irreconcilable is where he says:
“Rejecting outlier data without a lot of careful analyses and very solid justiication proving that the data was measured incorrectly and not because it doesn’t fall where it “should” is a big no-no in science.”
The problem is that he implies that rejecting data is only justifiable when one can show that the data was “measured incorrectly”. There are distributions of errors for which this is not true – for which it is better to ignore or down-weight some data intrinsically as part of the estimation.
For example, consider estimation of something like a stability boundary of a chaotic system. (Or to use the recent slang “tipping point”). If you can only take data from a system for a finite time, you don’t have the luxury of finding out what eventually happens; so there will be some cases of instability which do not manifest themselves in a short time, and have excursions which are less than for some cases of stability. If the mixing time of the chaos is faster than the time scale which you can observe, or if there may be some randomness in the system, you are in a situation where the best option may well be to use a statisical model to determine stability; probably a logistic regression to estimate the probability of instability. In this case, you run the danger of a purely artificial estimation problem called “separation” – it will happen if you correctly identify the all stable and unstable examples as such. Because you will only have a finite set of data, there will then be a range of model parameters which are equally likely – i.e. the likelihood function is degenerate (the Fisher Information is singular). The parameter errors which result from this are huge, and the model – despite getting all the data to “lie on the theoretical line” is not so useful for prediction. What one normally does in this situation (as pointed out by Firth) is to add an information-free prior distribution (Jeffrey’s rule) and what this ends up doing is down-weighting the data which are most influential. So even though the observations of those data are correct, and even though the naive model agrees with those data, and the outcomes of the model for those data are the most important, the best thing to do is to down-weight those data, and move the “theoretical line” to AGREE LESS with them. In particular – if you happen to have observations which happen to land exactly on the stability boundary – the very thing you are trying to estimate – you will be best off IGNORING THEM COMPLETELY. In other words, if you actually take data exactly where you want it, then irrespective of whether the measurement is correct – or completely broken – then you will throw that data out.
This sort of example – where TOO MUCH AGREEMENT BETWEEN THEORY AND EXPERIMENT leads to a pathology resolved by rejecting good data – is not something that most scientists keep in the back of their mind, unless they are thinking about some sort of malfeasance.
“I am grateful to you for emphasizing the statistical nature of outlier identification, though. It’s a problem I often confront in my day job.”
I actually think just about every modern scientists is probably up to their neck in this stuff, whether they know it or not.
I think Ray was referring to the part where I said without further analyses. BTW, and maybe a minor point but I am a she. :)
Joseph Hunkins, “Creation science” is not scientists, and those who purport to practice it are not scientists. There may be some scientists out there who are over zealous in purging their data of “outliers,” but they aren’t doing science when they do this. In so doing, they preclude ever discovering new science, so I would contend that any scientist worth his or her salt will instead use statistical techniques and stringent investigation to identify potential outliers. Experimentalists love nothing better than to rub the nose of a theorist in a theoretical failure. It is the closest the experimentalist will ever get to the feeling of sacking a quarterback in the superbowl. I do not know of anyone who would forego that possibility merely to conform with theory.
This has been quite the enlightening exchange this morning. My thanks to all commenters.
I am a little troubled by an approach to data that starts off with “There is a divergence in the data between it and my theory. What errors in data gathering might there be that would explain this divergence?”
Doesn’t this tend to bias the corrections?
In an ideal world the hunt for problems in the data would be balanced, i.e. as much attention would be paid to “what errors in data gathering might there be that causes a false congruence between my theory and the data”.
But the real world, and real people, are not perfect.
[Response: Who said that this was the starting point? We have stressed here over and again that discrepancies that exist perforce demand that three things be looked at concurrently: 1) is there an issue with the data; 2) is there an issue with the theory (or it’s quantification in the model)? and 3) are we correctly comparing like with like? In climate science, as in every science, the resolution of any discrepancy is distributed amongst these alternatives. – gavin]
All you climate change deniers can put Dr. Hansen’s statement in your pdf reader, read it and weep along with the rest of us.
Twenty years ago we had twenty years to do something about the rise in CO2 in our atmosphere. We’re almost out of time.
http://www.columbia.edu/~jeh1/
June 23, 2008
Just a layman’s question : could Benford’s Law be used to check the quality of data, and if so, in which instances?
John Lederer, Why do you jump to the conclusion that the motivation to find the error came from the discrepancy with theory? Any decent experimentalist would want to find a real discrepancy–that’s how they get famous. The discrepancy has existed for a very long time–only now has it been explained. If the motivation were to make theory and data match by whatever means, don’t you think they could have done so more quickly?
John Lederer,
I did not write that the only time scientists look for errors in observations is when the observations deviate from the theory!
Of course scientists do their best to make the observations accurate right from the get-go. But problems sometimes get missed, and problems sometimes don’t manifest until much later.
When there _is_ a deviation from the theory, it is a clue that there _might_ be a problem with the observations. It’s a clue just like it’s a clue when the shop that built the sensor tells you they fired a worker who built sensors numbered 456 through 655, for falsifying the quality control records. Regardless of where the clue comes from, the scientists have a responsibility to follow up.
As Ray wrote in #117, you don’t _only_ investigate the observations, you investigate the theory as well. Then you let the chips fall where they may.
Ray – I can only hope you are right to have what seems like boundless optimism that bias and subjectivity rarely rear their ugly heads among scientists and then are flushed out by the peer review process.
Tom Dayton, I would say that the interest in points that don’t fall on the theoretical curve is two-fold:
1)They could be in error
2)They could be new physics
You certainly don’t want to preclude the possibility of discovering new physics for the sake of conformism (conformism is not an adjective usually applied to scientists). You also don’t want to proclaim new physics for any tiny disagreement. So it is natural to look at these points AND to look at what theory predicts very closely. As Asimov said, “The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka!’, but ‘That’s funny …”
Nick Gott #97:
My variation on this theme:
When the data is corrected towards confirmation of AGW, it is tampering with the data to fit the models.
When the data is corrected in the other direction, it is proof that the denialists were right all along and there is nothing to worry about.
Gavin, Looking at the second figure in your writeup, it appears that the model runs with volcanic aerosols are matching the long-term trend in ocean heat content very well. When one takes a closer look however, there is a possibly important observation. Firstly, during each of the major volcanic events, the models show a higher amplitude change in ocean heat than is present in the Dominguez curve. In fact, for the Agung eruption, the Dominguez reanalysis is completely out of phase with the model ensemble mean. It is not clear from the labeling, but I assume the upper model realizations are for runs with no volcanic forcing added. Now my question: Do you think that the volcanic forcing is overdone in most of the recent-vintage AOGCM model runs? In other words, if the model aerosol imput is tweeked until a good history-match is achieved with observed volcanic forcing (as measured by observed decreases in ocean heat content), will they then produce a long-term trend in TOA radiative imbalance that is too high relative to observations?
[Response: The aerosol distribution for Agung is the least well observed of recent volcanoes, and so the uncertainty in the forcing is non-negligible – and the observed data is has higher uncertainty too. So it could be tweaked either way. However, the volcanoes are acting like a release valve, deflating the OHC for a couple of years at a time. So while there will be less OHC going forward from every eruption (Glecker et al 2005 found impacts even today from Kratatoa), that doesn’t impact the trend subsequently. Think of it like a sweet jar – if you add in 2 sweets a day for weeks, and every so often scoop out a handful, the long term trend is lower, but the trend in between scoops is the same. – gavin]
Joeduck, the only thing I have faith in is that humans will act in a manner that they believe best furthers their interests. I also have faith that intelligent people will be better able to perceive where their interests lie. As a scientist, it is in my interest to detect any errors I make before they are detected by my peers. It is in my interest to detect any errors my peers will make. It is in my interest to know the difference between errors and discrepancies with theory that are indicative of new physics. That is how one’s prestiege and influence increase as a scientist. You have to realize that science is a collective, competitive enterprise. It doesn’t depend on the judgment of any single individual, but rather on the collective judgment of a huge community of peers–all of whom think they’re smarter than you and are just waiting for the chance to prove it.
And I would like to thank you for a novel experience–this is the first time anyone has ever accused me of boundless optimism.
RE: #124 & #103:
I agree with you, Ray.
I anticipate someone else now will object that it’s not “fair” to adjust the theory to match observations that didn’t match the theory. So I’ll respond in advance:
First of all, the goal is to make the theory reflect reality. That’s a different goal than when you’re betting on your prediction of a soccer match’s outcome. When betting on a soccer match you’re not allowed to adjust your prediction as the game progresses, because the only purpose of betting is to test your predictive skill. (Plus the secondary goal of drinking beer in a pub. Or maybe I’ve got the priorities reversed….) But in science the goal is good theory, which largely means good match of theory to observation. (There are other aspects of theory quality, such as fruitfulness and explanatory power.)
Then there is the fact that climate models are models of (theories of) physical processes. Adjusting such a model requires you to let the chips fall where they may, just as correcting observations for a flaw in an instrument does (see #103). If your model under-estimates the June 1979 temperature by one degree, you cannot simply add to the model a line “If it’s June 1979, add a degree.” Instead you must change the model’s description of some physical entity or process, and accept the consequences not just for June 1979, but for every other date–even if that makes the model fit worse on other dates. That’s why you don’t change the model based simply on mismatch of one or a few observations. You use the mismatch as a stimulus and guide to figuring out what parts of the model might be sensible to change, based on supportive rationale and data in addition to the mismatch that caught your attention.
I think that several of you have leaped to a conclusion other than what I intended to suggest when I pointed to a possible bias in finding errors.
Real world data collection and recording is messy — the recent example of buckets, intake inlets, and SST should suggest how messy it can be.
When I have a theory and real world data seems to support my theory except for some untoward mismatches here and there, I am likely to look closely at the data for those mismatches and see if there might be some problem in its collection or recording that biased it away from the theory. Data gathering being messy, I am likely to find some problems.
If on the other hand the data and my theory matches, I am likely to tell myself what a smart lad I am and treat myself to a quiet restrained jig of delight. I will check the data but I am likely to believe that it is right and my search for error be less than dogged.
Such an approach is not that of a perfect scientist, but I am not perfect and neither is anyone else.
Thus we are likely to have two possible errors in our method: a less than diligent search for errors when data matches our theory, and a less than diligent search for the types of errors that would move data that does not match our theory even further away.
That is the result of being human.
There are two ways to fight against this human proclivity. One way is rigorous self-discipline. The other is to and provide full, easy access to the data and the detailed means of gathering it to those who oppose our theory.
Rigorous self-discipline is commendable, but we don’t have perfect self-discipline. We just were not made that way.
RE: 129
John Lederer, now I understand the points you have been trying to get across. I agree that the risks you describe exist.
There is a third way that the scientific process fights against that human proclivity: Other folks are developing their own theories in competition with you. If they can’t get their theories to predict as accurately as yours does, they will be frustrated and maybe even suspicious, and will demand that you reveal all the relevant aspects of your theory, even if you did not do so originally.
Tom, Ray, John, Figen, others: Many good points by all of you above, but I think John’s points in 129 about the difficulties of maintaining a pristine set of observations are particularly relevant to the Global Warming discussion. MM’s complaints about temperature stations and the GISS corrections seem reasonable to me as do concerns about data sharing. Reading the comments above leads one to wonder why these remain contentious issues.
… humans will act in a manner that they believe best furthers their interests. I also have faith that intelligent people will be better able to perceive where their interests lie.
Ray I agree with first part, disagree with second. Like you I know a lot of smart folks, but unlike you I find intellectuals cling *as stubbornly as others* to their questionable ideas. [Shermer “Why People Believe Weird Things” suggests why, for example, hoax medicine like homeopathy is more popular among the well-educated]. Theoretical complexity often makes it easy to be stubborn about beliefs because so few have the time to learn the underlying math/science. This relates to peer review in an interesting way – friends and insiders are less likely than outsiders to challenge defects. Are critics correct when they suggest this “friendly review” system common in climate science where authors know each other well?
RE: #130:
John Lederer, lest I be misunderstood:
I am very, very, very confident that the existing mechanisms in climatological science far more than adequately cover those risks.
Re #126: Gavin, to follow up, it seems to me that the issue is whether climate modelers are inadvertently dipping into the sweet jar with two hands to obtain the history match. If in the model climate, there are three sweets per day added to the jar (instead of two), then two handfulls being scooped out periodically might be required to balance the books.
Losing the analogy, if the aerosols added to the model climate are greater than those added to the real climate, and this produces a history match, this must then point to the lack of fidelity between the model climate and actual climate. Maybe some parameterizations are not good, or there are incomplete processes included in the model, or weather processes and feedbacks may not be handled appropriately, or possibly some other unknown issues. The point is that the history match would be for the wrong reasons, and such would not portend skill for future predictions. Hence, this reasoning is what drove my previous question about aerosols.
The bottomline for me (on the ocean heat content time series) is that sure, the climate system has been in a positive radiative imbalance over the last several decades, and sure, this is likely attributed largely to human-caused changes in GHG forcing, but I am not convinced that this new paper is a vindication of model forecasting skill. It seems possible, based on the Dominguez reanalysis, that in nearly all climate models, their net TOA radiative imbalance due to the added GHGs may be higher than that observed in the actual climate. In other words, most of the current models have the sensitivity to GHG increases too high, and this has been compensated for by adding too much aerosol forcing, thereby giving the superficial appearance of a robust history match. As supporting my curiosity, I might reference slide 3 in the Wijffels powerpoint presentation presented during the workshop. It is not hard to make the case that the amount of aerosols added to the model produces a big difference in the long- term trend in ocean heat content. Is my curiosity about this issue off base in your opinion?
[Response: You need to distinguish volcanic from tropospheric aerosols. The timing and magnitude of the former are much better constrained than the latter. In general though you are correct – good hindcasts only show the consistency of the calculation, but do not show that this is the only way that such consistency can be achieved. However, the bigger the number of matches, the harder it is for alternative scenarios to do so. Thus, getting the mean SAT trend right is easier than the SAT trend+stratospheric trend, which is easier than SAT+strat+OHC, which is easier than SAT+strat+OHC+short responses to volcanoes… etc, You get the idea. In a Bayesian sense, the more matches there are – especially if they are independent – the higher your posterior likelihood is that the original theory was correct. This is how it always works. – gavin]
Joseph,
I appreciate and agree with most of your points. Of course measurements have errors associated with them and of course some of these errors may be caused inadvertantly by the inherent biases of the observer. A lot of the science rests on quantifying those errors and uncertainties. But my point is that a good scientist would be upfront about those errors, and actually it’s more than that. A scientist NEEDS to be upfront about it if he/she wants to stay in the business as a respected member of the scientific community.
Where I disagree is that we (climate scientists) aren’t all a bunch of friends who will be gentle with each other’s work. I have been pretty lucky to work with the scientific questions of interest to me and with leading people in the field. But friendliness goes only so far. The peer review process is often anything but friendly, especially if the reviewer chooses to be “anonymous.” And I know, as do all my colleagues, that this luck I speak of can change fast if I start to fudge data or make too many assumptions. As colleagues we really hold each other’s toes to the fire, otherwise our science will lose its integrity. You learn to develop a thick skin and accept and critically evaluate negative feedback on your work. Fast! And actually it’s that most horrendous review you receive that makes your work much stronger in the end when it is finally accepted for publication. Though you may be peeved at first, that harsh review is often very appreciated because it helps avert much potential embarrassment when the work is finally out there in litertaure for many many years.
RE: #131
Joseph Hunkins wrote “Are critics correct when they suggest this “friendly review” system common in climate science where authors know each other well?”
Joseph, I don’t know the climate science community. But the communities I do know–scientific research methodology and behavioral science–seem to have the same degree of overlapping authoring, peer reviewing, committee sitting, grant approving, and so on, as does climate science. In my scientific communities, “insider trading” usually is handled by formal checks and balances, but when that fails it is handled by informal checks and balances–usually more brutally.
In most scientific fields, there is so little money that competition is fierce. That combines with fierce idealism. (You don’t become a scientist to get rich.) Young scientists sometimes are shocked at how quickly their friends become vicious critics, and then just as quickly become friends again. An analogy is the business world, where “Sorry, it’s just business” is a common conversation between friends. Likewise, prosecuting lawyers and defending lawyers can be the best of chums outside the courtroom, but horribly vicious to each other inside.
Evidence is in most of the stories you read about any field of science. Good science reporters (e.g., those of Science News and Scientific American) stick into nearly every story about one scientist’s discovery, skeptical comments by that scientist’s peers. You’ll notice that sometimes those comments are downright dismissive or even hostile.
“Chum” has two meanings in scientific communities, and one of those is related to “shark.”
Joeduck and John Lederer, The thing is, you can’t just think of it in terms of “data” and “theory”. Each experiment has errors associated with it, and our anticipation of the errors likely given the experimental method affect our expectations of how much we should trust the data. Likewise, we will have much more confidence in theoretical predictions (and hence distrust of “outliers”) for a theory that has been well validated previously. What is more different scientists have different levels to which they trust data and experiment, and the folks taking or gathering the data are usually not the same folks constructing the data. There is an inherent antagonism between theorists and experimentalists. A theorist likes nothing more that to look at an experimentalist’s data and say, “See, I told you so!” An experimentalist, on the other hands, fantasizes about the day when he or she will be able to slap the theorist across the face with a datasheet that contradicts the theory. It is not usual for an experimentalis to turn away meekly and say, “Yeah, my data kind of sucks.” They will naturally be reluctant to pass up the opportunity for fame and glory. And the theorist will be secretly sweating, checking equations, examining assumptions, etc.
If you look at the classic example of how the concept of the ether was overturned. The wave theory of light had a lot of very strong evidence and there was at the time no evidence against it. The concept of a wave propagating without a medium was unthinkable, so by implication, the ether had to exist. Yet, it stubbornly refused to reveal itself. Rather than give up, experimenters kept coming up with more and more sensitive experiments until Michelson and Morley drove the nail into the coffin. Now this led to not just relativity, but also the quantum revolution. However, more important, a conceptual revolution was required. So, really, the conservatism in rejecting the ether was appropriate. I don’t think the sort of bias you are talking about is a major problem in science as a collective enterprise–and that is how science actually gets done.
Definition:
Friendly reviewer–one who warns you before completely trashing your research
Keep in mind–when I review your research, my credibility–even my career–is on the line. I’ve known husband/wife researchers who trash eachother’s research if it deserves it.
The most you can expect from a reviewer is constructive criticism.
Ray, Figen: I don’t think the sort of bias you are talking about is a major problem in science as a collective enterprise–and that is how science actually gets done.
I agree with this strongly. However I’m not clear to what extent it is a *minor problem*, and the stakes are so high that I would be comforted to see, for example, more critical discussion of Jim Hansen’s GISS data as well as his generalizations than I expect to see here at RC due to that “no friendly fire” factor.
Figen unless I am mistaken *harsh insider criticism* of a colleague as high up as Jim Hansen would be a challenge to one’s career – is that a fair generalization?
[Response: Not at all. – gavin]
If it is every true to a small extent then …. Houston, we have a problem.
Joe,
If I received a detailed though harsh review from Jim Hansen, I may be a little mortified at first, but I would cherish the fact that he took the time and took my work seriously enough to provide a harsh review. Then, after getting over the emotions of both elation that he reviewed my work at all and dejection that he didn’t like it, I would sit down and go through his review point by point to see how I could improve my work. That would be the opportunity of a lifetime to seriously improve my work and potentially (hopefully) make a huge impact in the science.
I am being sincere in this. And though not from Hansen, I have received many harsh reviews from some pretty hefty scientists and am alive to tell the tale :) Actually I am much better off for it. Nothing is as sobering and as educational as a good, long review.
Gavin and Figen I guess I was not clear enough. Are there many examples of colleagues criticizing Hansen’s sometimes extraordinary claims about pending catastrophe?
[Response: What extraordinary claims are those? I’d prefer you took something directly from one of his many writings rather than an interpretation of them on random websites. If they are indeed extraordinary (which remains to be seen), we can see whether he has been criticised and by who. – gavin]
RE: #140
Joe, if your suspicions are correct, you should see evidence in the history of climate science. Take a gander at Spencer Weart’s free site, which has even more content than his book. http://www.aip.org/history/climate/
Keep in mind that the medium-to-long term history is the scale of most relevance, because the corrective mechanisms we’ve been writing about do take a bit of time. The major reason they take a bit of time is that new observations are a key to the scientific corrective mechanisms, and making new observations takes more time than, say, writing an Op-Ed piece.
Gavin –
Sure, I’ll find some Hansen quotes that IMHO are extraordinary and post them, but that approach won’t answer my question. The issue is simple. In the private sector if you call out your superiors you will often be squashed. I suspect, based mostly on common sense and reports from dissident climatologists, that a variation on this happens in the sciences, though far more subtly.
In fact Hansen himself was subjected to the type of non-scientific, political and ego-driven pressures I am talking about. To his credit he resisted and reported it (nationally to headline news around the globe). You seem to think that type of pressure always comes from climate skeptic camps, and I’m saying this is not a reasonable assertion and in fact is a dangerous one that blinds those within the climate community to their own personal bias challenges.
I’m *hypothesizing* that bureaucratic social pressures are keeping new NASA researchers from some research, and certainly from generalizations, they might otherwise consider/express.
Tom Dayton – yes, there should be evidence of this and I will follow up at that site – thx.
Do you find the claims of people like Pielke and Chris Landsea dubious, ie you are comfortable that grants and reviews are done with no regard to any political considerations?
[Response: I have reviewed and participated on many panels that award climate related funding and I have never seen even a hint that political considerations were important or even relevant. The important factors are interest, tractability and competence. Look up what the NSF (for instance) actually funds before spouting off about how it is all political. – gavin]
I do not know the extent to which this happens or poisons the well. All I’d assert as obvious is that humans – scientists or not – do not divorce themselves from these social and ego pressures. [edit]
RE: #142
Joe, I understand that your experience is in the private sector, so that’s the most salient example for you to bring to bear. (I’ve been in academia, the private sector, and government.) The process and community of science is more multifaceted and wide-ranging than the private sector is.
In particular, the process of publishing in peer-reviewed scientific journals crosses lots of boundaries, involving reviewers and editors from anyplace as long as they have appropriate expertise. Likewise, granting agencies use reviewers from all over, not just from their agency, not just from their branch of government, not just from government at all.
There are lots of reasons for granting money. Occasionally it is even done just to end a long-lived and distracting controversy, by letting the applicant rigorously test some hypothesis that the reviewers have very low confidence will be supported. But that is done only occasionally, because if the result is as the granters expect, then the results are unlikely to be published, meaning it was a waste of money. But occasionally journals will publish such papers, for the same reason of ending a distracting controversy.
Sometimes journals will even publish a paper purely to make a point, as in “Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials” http://bmj.bmjjournals.com/cgi/content/full/327/7429/1459
Joseph Hunkins, The thing that I think you are missing is that in science it is considered petty to take offense when you are criticized on the quality of your science. I can and do criticize my group leader on the soundness of the science, and he does the same with me. I am not free to call my colleagues “ignorant jerks” without some expectation of repurcussions. I have seen graduate students go toe to toe with Nobel Laureates when the students thought they were right. The worst they got was maybe a somewhat condescending nod of approval and a correction. It would be naive to think that there is no politics in funding, but in science it is generally acknowledged that we have a common goal of advancing the state of knowledge as far as we can take it given a limited pool of resources. This tends to breed funding decisions based on the merits of the research and the researchers.
This meritocratic process can be short-circuited, but it is generally done by nonscientists who have priorities different from advancing the state of knowledge. Yes, there are probably young researchers who would like to look into an idea they have, but the thing that keeps them from doing it is not bureaucracy, but lack of funding for the task–and the funding decisions are usually made by nonscientists.
Scientists want more than anything else to keep doing science–to understand some portion of the Universe that fascinates them. They will suppress a lot of ego to do so–and some of them have a lot of ego to suppress.
I was probably too subtle in my last comment.
Anybody who thinks climatology is not following “the scientific method” should read this paper: “Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials” http://bmj.bmjjournals.com/cgi/content/full/327/7429/1459
That’s hilarious Tom. I actually work with some folks (they are not scientists) who are very skeptical of gravity!
RE: #144:
Joseph Hunkins, the “Mediarology” page by Stephen Schneider is relevant to both the issue of social and other biases in science, and to the appropriateness of an assortment of methods in science: http://stephenschneider.stanford.edu/Mediarology/MediarologyFrameset.html
RE: #145:
That parachute paper I cited is not only funny, it is dead serious. It was actually thoroughly researched and written exactly as thousands of similar papers are. If you read it while mentally substituting “enzyme X” for every mention of parachutes, and “disease Y” for every mention of falling out of an airplane, you’ll see that its approach is completely legitimate.
The relevance to global warming denial is that many poorly-schooled denialists claim science requires us to effectively ignore the well-established physical causal mechanism of greenhouse gases preventing escape of radiation, in favor of tightly controlled experiments on the entire planet. In fact, scientists use all manner of information for decision making, including knowledge of plausible causal mechanisms such as parachutes’ effects on air resistance and therefore speed, and CO2’s effects on radiation emitted from the Earth.
Lots of thoughtful comments above.
I think it was inflamatory for me to call Jim Hansen’s climate claims “extraordinary” but since I said I’d post something I’d suggest the following is not well supported by data or IPCC’s summary of the situation:
As an example, let us say that ice sheet melting adds 1 centimetre to sea level for the decade 2005 to 2015, and that this doubles each decade until the West Antarctic ice sheet is largely depleted. This would yield a rise in sea level of more than 5 metres by 2095.
Of course, I cannot prove that my choice of a 10-year doubling time is accurate but I’d bet $1000 to a doughnut that it provides a far better estimate of the ice sheet’s contribution to sea level rise than a linear response.
In my opinion, if the world warms by 2 °C to 3 °C, such massive sea level rise is inevitable, and a substantial fraction of the rise would occur within a century. Business-as-usual global warming would almost surely send the planet beyond a tipping point, guaranteeing a disastrous degree of sea level rise.
Although some ice sheet experts believe that the ice sheets are more stable, I believe that their view is partly based on the faulty assumption that the Earth has been as much as 2 °C warmer in previous interglacial periods, when the sea level was at most a few metres higher than at present.
My biggest concern about the AGW *science* discussion is the degree to which it’s considered acceptable to emphasize the possibility of catastrophic change while failing to point out the far more likely scenarios (such as IPCC’s).
[Response: Hansen’s statements are a model for how to express a dissenting opinion in a scientific discussion without insults, accusations of bad faith and unsupportable statements of certainty. People are free to disagree and criticise, but you will actually find very few do, because no-one is very confident that they can put an upper bound on sea level rises in the next century that is negligible – and that includes the IPCC authors. At recent meetings I’ve attended on ice sheet dynamics, the concern is palpable that we are not in a position to do so. This doesn’t just worry Hansen. – gavin]
Gavin your optimism about how little politics wind up influencing research is encouraging. Do you see those principles as extending here to RC? It sure seems there is a great *reluctance* to welcome (or even allow?) scientists with legitimate credentials who pose challenges to the views that prevail here, but I suppose this could be that they are simply reluctant to step into the fray.
[Response: People challenge us here all the time, and there is no “reluctance” to deal with serious scientists – all of us have such interactions at meetings and workshops continuously. What gets tiresome is the continual parade of junk masquerading as neo-Galilaen revelations. But if you have someone in mind, encourage them to participate. – gavin]
RE:#144 A minor clarification: internal funding decisions, such as those that are made by the Director of the National Science Foundation and various subcommittees, are made by scientists. The amount of money that each funding agency receives is decided by politicians. My PhD advisor was the Director of NSF for a year, which gave me a little more insight into how NSF works.
Re # 147 Tom Dayton “That parachute paper I cited is not only funny, it is dead serious. It was actually thoroughly researched and written exactly as thousands of similar papers are.”
Sorry, Tom, but I can’t help but think it is satire.