This is a continuation of the last thread which is getting a little unwieldy. The emails cover a 13 year period in which many things happened, and very few people are up to speed on some of the long-buried issues. So to save some time, I’ve pulled a few bits out of the comment thread that shed some light on some of the context which is missing in some of the discussion of various emails.
- Trenberth: You need to read his recent paper on quantifying the current changes in the Earth’s energy budget to realise why he is concerned about our inability currently to track small year-to-year variations in the radiative fluxes.
- Wigley: The concern with sea surface temperatures in the 1940s stems from the paper by Thompson et al (2007) which identified a spurious discontinuity in ocean temperatures. The impact of this has not yet been fully corrected for in the HadSST data set, but people still want to assess what impact it might have on any work that used the original data.
- Climate Research and peer-review: You should read about the issues from the editors (Claire Goodess, Hans von Storch) who resigned because of a breakdown of the peer review process at that journal, that came to light with the particularly egregious (and well-publicised) paper by Soon and Baliunas (2003). The publisher’s assessment is here.
Update: Pulling out some of the common points being raised in the comments.
- HARRY_read_me.txt. This is a 4 year-long work log of Ian (Harry) Harris who was working to upgrade the documentation, metadata and databases associated with the legacy CRU TS 2.1 product, which is not the same as the HadCRUT data (see Mitchell and Jones, 2003 for details). The CSU TS 3.0 is available now (via ClimateExplorer for instance), and so presumably the database problems got fixed. Anyone who has ever worked on constructing a database from dozens of individual, sometimes contradictory and inconsistently formatted datasets will share his evident frustration with how tedious that can be.
- “Redefine the peer-reviewed literature!” . Nobody actually gets to do that, and both papers discussed in that comment – McKitrick and Michaels (2004) and Kalnay and Cai (2003) were both cited and discussed in Chapter
2 of3 the IPCC AR4 report. As an aside, neither has stood the test of time. - “Declines” in the MXD record. This decline was
hiddenwritten up in Nature in 1998 where the authors suggested not using the post 1960 data. Their actual programs (in IDL script), unsurprisingly warn against using post 1960 data. Added: Note that the ‘hide the decline’ comment was made in 1999 – 10 years ago, and has no connection whatsoever to more recent instrumental records. - CRU data accessibility. From the date of the first FOI request to CRU (in 2007), it has been made abundantly clear that the main impediment to releasing the whole CRU archive is the small % of it that was given to CRU on the understanding it wouldn’t be passed on to third parties. Those restrictions are in place because of the originating organisations (the various National Met. Services) around the world and are not CRU’s to break. As of Nov 13, the response to the umpteenth FOI request for the same data met with exactly the same response. This is an unfortunate situation, and pressure should be brought to bear on the National Met Services to release CRU from that obligation. It is not however the fault of CRU. The vast majority of the data in the HadCRU records is publicly available from GHCN (v2.mean.Z).
- Suggestions that FOI-related material be deleted … are ill-advised even if not carried out. What is and is not responsive and deliverable to an FOI request is however a subject that it is very appropriate to discuss.
- Fudge factors (update) IDL code in the some of the attached files calculates and applies an artificial ‘fudge factor’ to the MXD proxies to artificially eliminate the ‘divergence pattern’. This was done for a set of experiments reported in this submitted 2004 draft by Osborn and colleagues but which was never published. Section 4.3 explains the rationale very clearly which was to test the sensitivity of the calibration of the MXD proxies should the divergence end up being anthropogenic. It has nothing to do with any temperature record, has not been used in any published reconstruction and is not the source of any hockey stick blade anywhere.
Further update: This comment from Halldór Björnsson of the Icelandic Met. Service goes right to the heart of the accessibility issue:
Re: CRU data accessibility.
National Meteorological Services (NMSs) have different rules on data exchange. The World Meteorological Organization (WMO) organizes the exchange of “basic data”, i.e. data that are needed for weather forecasts. For details on these see WMO resolution number 40 (see http://bit.ly/8jOjX1).
This document acknowledges that WMO member states can place restrictions on the dissemination of data to third parties “for reasons such as national laws or costs of production”. These restrictions are only supposed to apply to commercial use, the research and education community is supposed to have free access to all the data.
Now, for researchers this sounds open and fine. In practice it hasn’t proved to be so.
Most NMSs also can distribute all sorts of data that are classified as “additional data and products”. Restrictions can be placed on these. These special data and products (which can range from regular weather data from a specific station to maps of rain intensity based on satellite and radar data). Many nations do place restrictions on such data (see link for additional data on above WMO-40 webpage for details).
The reasons for restricting access is often commercial, NMSs are often required by law to have substantial income from commercial sources, in other cases it can be for national security reasons, but in many cases (in my experience) the reasons simply seem to be “because we can”.
What has this got to do with CRU? The data that CRU needs for their data base comes from entities that restrict access to much of their data. And even better, since the UK has submitted an exception for additional data, some nations that otherwise would provide data without question will not provide data to the UK. I know this from experience, since my nation (Iceland) did send in such conditions and for years I had problem getting certain data from the US.
The ideal, that all data should be free and open is unfortunately not adhered to by a large portion of the meteorological community. Probably only a small portion of the CRU data is “locked” but the end effect is that all their data becomes closed. It is not their fault, and I am sure that they dislike them as much as any other researcher who has tried to get access to all data from stations in region X in country Y.
These restrictions end up by wasting resources and hurting everyone. The research community (CRU included) and the public are the victims. If you don’t like it, write to you NMSs and urge them to open all their data.
I can update (further) this if there is demand. Please let me know in the comments, which, as always, should be substantive, non-insulting and on topic.
Comments continue here.
By the way, I meant that in the best possible way, and in a sense of good faith and clear scientific curiousity
Anand Rajan,
FOI requests serve no purpose in science. We already have a mechanism for sharing data: ask nicely and say “Please!” Why should scientists be treated as criminals merely for doing their job?
And have you forgotten that it is only 12 years since scientists were subpoenaed before Congress and excoriated for the heinous crime of telling the truth (you remember, the chairmen of the hearings were Doolittle and Delay–and no I’m not making this shit up!).
The vast majority of climate data are now in the public domain–so what have the denialists done with it? Bupkis. Worse than nothing, because they wind up taking time away from legitimate scientists. I feel very safe in predicting the next breakthrough in climate science will not come from the brain trusts over at CA or WTFUWT.
I am a lay person, a skeptic with a small s, I guess, slogging through the emails. I am up to January 2005. Tom Wigley writes to Mann on 1/20/05 what follows and eventually gets a variety of responses. Is this as bad as it sounds? Can someone decipher here?
” This is truly awful. GRL has gone downhill rapidly in recent years. I think the decline began before Saiers. I have had some unhelpful dealings with him recently with regard to a paper Sarah and I have on glaciers — it was well received by the referees, and so is in the publication pipeline. However, I got the impression that Saiers was trying to keep it from being published. Proving bad behavior here is very difficult.
If you think that Saiers is in the greenhouse skeptics camp, then, if we can find documentary evidence of this, we could go through official AGU channels to get him ousted.”
[Response: There really were some terrible papers published and it isn’t surprising that people started questioning the quality of the editorial decisions. As far as I know, Saiers served out his full 3 year stint as editor though. – gavin]
In fact, an extended quote from Peter Watts (rifter.com as linked above) is worth a shot:
—–
Science is so powerful that it drags us kicking and screaming towards the truth despite our best efforts to avoid it. And it does that at least partly fueled by our pettiness and our rivalries. Science is alchemy: it turns shit into gold. Keep that in mind the next time some blogger decries the ill manners of a bunch of climate scientists under continual siege by forces with vastly deeper pockets and much louder megaphones.
As for me, I’ll follow the blogs with interest and see how this all shakes out. But even if someone, somewhere, proves that a handful of climatologists deliberately fudged their findings — well, I’ll be there with everyone else calling to have the bastards run out of town, but it won’t matter much in terms of the overall weight of the data. I went running through Toronto the other day on a 17°C November afternoon. Canada’s west coast is currently underwater. Sea level continues its 3mm/yr creep up the coasts of the world, the western Siberian permafrost turns to slush. Swathes of California and Australia are pretty much permanent firestorm zones these days. The glaciers retreat, the Arctic ice cap shrinks, a myriad migratory species still show up at their northern destinations weeks before they’re supposed to. The pine beetle furthers its westward invasion, leaving dead forests in its wake— the winters, you see, are no longer cold enough to hit that lethal reset button that once kept their numbers in check.
I could go on, but you get my drift. And if the Climate-Change Hoax Machine is powerful enough to do all that, you know what?
They deserve to win.
Posted in: climate, scilitics by Peter Watts
——-
#1030, thank you for that link to Little Green Footballs. Great find – I was pretty certain that the TimesOnline article was bunk, what with no sources and no mention of “deleted data” anywhere else to be found.
In the beginning: Friends of Science, Talisman Energy and the de Freitas brothers
http://deepclimate.org/2009/12/02/in-the-beginning-friends-of-science-talisman-energy-and-the-de-freitas-brothers
Here is the first of an occasional series that will look back at the origins of various major players among Canada’s climate contrarians. And, quite appropriately, the honour of inauguration belongs to none other than our old, um, acquaintances, Friends of Science.
For the first time, we can confirm both financial and logistical support from an Albertan oil company, Talisman Energy, along with circumstantial evidence of the early involvement of a second, Imperial Oil (ExxonMobil’s Canadian subsidiary). We’ll also look at the key roles played by the de Freitas brothers, geologist Tim and climate skeptic Chris. And the story leads right to the heart of a key controversy reignited by the stolen CRU emails, namely the ongoing perversion of the scientific peer review system by “skeptic” scientists.
Gavin – in regards to the IPCC discussion of McKitrick and Michaels (2004) and Kalnay and Cai (2003); they are in Chapter 3 of the IPCC FAS WG1, not chapter 2, in case folks want to read more.
Thanks to everyone at RC for your efforts!!
gate have a lot of parallels.
Watergate resulted in the conviction of some participants and the resignation (and later pardon) of the U.S. President.
How Climategate will end up is anyone’s guess today, but it is safe to say that it will have a major impact.
But there is one crucial difference between the two. This is the instant flow of information today, with the Internet, leaked emails, blog sites (including RC), youtubes, etc., none of which existed almost 40 years ago when ‘Deep Throat’ leaked information to the Washington Post reporters.
Anti-parallel really, if it were parallel we’d have everybody chasing after the hackers (burglers) with the ‘skeptics’ franticly covering up! To be parallel we’d have the hackers going to jail.
Heh, I’m responding to my own post. I’ll say this, let this be a lesson learned, that being that all the cards should, of course, always be on the table
After this is done (e.g. the new data sources page etc.) to a reasonable person’s, or committee’s satisfaction, there should come a point wherein climate scienists are no longer in any way obliged to continue to waste valuable time responding to purely harassing requests for the same data over and over again. Where they can finally tell the #%&*! to bugger off.
[Response: Unlikely. – gavin]
I get that the CRU can’t release data because others put limits on what they can’t do with it.
What you don’t bring up, then, is how you can use that data at all.
If your research is based on data you can’t show, rest assured that all such research is worthless, at least until such a day as you can show the relevant data.
If anyone else tried to publish results based on data they couldn’t reveal, they’d be ridiculed.
For some reason, that does not apply to you guys, most likely because you currently are saying what the politicians want you to say. You’ve been accused of all sorts of scientific misconduct, the emails show clearly that what really has been going on was far worse than anyone thought. Do yourselves a favour: Don’t assume that this will blow over.
In the interests of showing respect for the scientific method, would you also please show us how much of the results of the latest IPCC reports that was based on data that the CRU “regrettably” can’t share?
[Response: Nothing. Because using the purely public domain GISTEMP version gives almost exactly the same result. -gavin]
#1040 David B. Benson “suggests wanting to look at 60–90 years of data to have high confidence of detecting small signals from the noisy record”.
The “small signals” are the much sought-after AGW evidence. (Small, eh?).
The “noisy record” isn’t “noise” at all. It’s all the other big climate stuff we haven’t yet properly understood.
Ryan:
No, not being conspiracy-theory crackpots, we don’t.
RE <a href="https://www.realclimate.org/index.php/archives/2009/11/the-cru-hack-context/comment-page-21/#comment-146174"?Sean
This made me think that at some point every paleontologist has probably expressed frustration that more fossils are not available, and I am sure that climatologists wish they also had more information. Both disciplines do the best with what they have and seek out more.
Added – but this does not mean that paleontologists think evolution is false.
> Do you not agree that there is a possibility of false facts, basically lies, being put into these reports to achieve a hidden agenda?
Barton Paul Levinson: No, not being conspiracy-theory crackpots, we don’t.
It’s not a question of conspiracy, it’s a question of political realism. Government-paid scientists have a clear conflict of interest when producing findings likely to boost the size and scope of government.
[Response: Complete BS. I already have a govt job, why do I need the govt to get bigger? The idea that thousands of scientists are studying climate in order to raise your taxes is ludicrous. -gavin]
No Gavin, it’s not ludicrous.
Generally speaking – no matter where you work – the better off your employer is, the better _your_ prospects are too. In this case, the bigger government gets, the more goodies all round there are to be had in government employ; more and bigger jobs, more research grants, etc.
Equally, it makes sense for government to try and disburse our taxes in a way most likely to benefit itself. There is a symbiotic relationship there.
[Response: Sorry, but you coming up with a ideology that seems to make sense to you is not the same as reality. I know many of the people whose integrity you are besmirching and the reality is completely divorced from your neat little theory. Show me one shred of empirical evidence to support your idea. Is research from prvivate universities different from research from state universities? Is research from Europe different from research from the US? Are grants funded by NSF different in ant meaningful way from research by NASA? Proof by assertion is nothing of the sort. -gavin]
Re: Comment 1052. ray Ladbury wrote:
“FOI requests serve no purpose in science. We already have a mechanism for sharing data: ask nicely and say “Please!” Why should scientists be treated as criminals merely for doing their job?
[…]
The vast majority of climate data are now in the public domain–so what have the denialists done with it?”
Ray,
Whether or not you feel that FOI serve no purpose in science, the fact is that they serve a purpose in law. The rule of law is a neccessary prerequisite of society so it doesn’t matter whether or not these requests are “inconvenient” or “annoying” to scientists, any more than it matters whether I find speed limits “convenient” or not. If I choose to speed anyway, and am caught, then there is a legal penalty to pay for my choice. If there is evidence that FOI laws were flaunted then it should be investigated and punished if the evidence stands up.
Regarding the data, what you say is simply not true. the data in the public domain has already been collected (“raw” data) and processed (“homogenized”) to turn it into consistent working datasets. There’s absolutely nothing wrong with that because it’s been collected from different sources using different protocols. The raw data is essentially useless for any analysis until that homogenization has been done.
However, if the homogenization process is flawed it can introduce a lot of errors or bias into the working dataset and, without the original raw data there is no way to verify if this has happened. It is the original, raw, data and details of the homogenization process that the sceptics have been trying to obtain in order to check the validity of the working data sets.
That’s actually pretty important when you think that all the peer-reviewed evidence in favour of AGW is based on those working datasets and the authors of those papers are relying on the validity of that data in forming their conclusions.
That raw data and homogenisation process is not in the public domain yet.
On Dec 1, Amity Shlaes had a column up on Bloomberg about Obama’s prospects for turning around his “failed” presidency. Tellingly, she attributed the hacked e-mails to scientists at the “U.K.’s Intergovernmental Panel on Climate Change”!!! I notified them about the error, and they corrected it. However, the column still states that the referenced scientists “were concealing evidence that might contradict global-warming alarmism”.
Either Shlaes is as confused about the IPCC as she is about the Great Depression, or she was deliberately trying to transfer the smear to the more well-known body. I wonder…..
Regards,
Martin
Gavin
I am sorry, you response to PeterK (1065)is unnecessarily hostile. Many scientist’s profiles have been raised dramatically by their research into AGW and the amount of money and resources that they have received has sky rocketed. This does not necessarily mean that their research is wrong. However, it does mean that they have a vested interest in the paradigm.
If you can point out (as you often do) that sceptics have a vested interest, at least be fair minded enough to point out that people in govt and universities where there is intense competition for funding may have a vested interest as well.
Also, at least be fair minded enough to acknowledge that many people who advocate intense government intervention to combat AGW, may be pursuing an interventionist political agenda under the guise of fighting AGW. This does not falsify the argument that AGW is real and that something should be done about it. In fact acknowledging this will probably strengthen your case.
Michael S (1068),
PeterK practically accused Gavin and other scientists of cooking the books to help their employment prospects. Gavin’s response was remarkably measured, given PeterK’s assertion, and avoided attacking the messenger.
–Martin
@ PeterK 1088
Think about this for a minute. If the government accepted the case made by climate scientists, why would they continue to fund climate research at the levels they currently do?
Wouldn’t they divert those research funds to research into alternative energy and adaptation?
If it was just about the research dollars, climate scientists would be saying “No, we don’t know enough, we need more funding for research into X, Y and Z.”
Well, since my last response was not posted- not sure why – let me try again.
I am not of the opinion that scientists pushing the climate change agenda are interested in pushing more government control – why would they be? But their science could be used to control entire populations, to wage wars on “uncooperative” nations, to stop developing nations from prospering, and yes, to impose taxes infringing on the rights of individuals for their behavior. In light of last night’s presidential address, I have no doubt that the reasons given for intervening political actions have nothing to do with the actual reasons behind the scenes.
In such politically charged science, therefore, it is especially important for those conducting the science to ensure that alternative perspectives are not suppressed, to present a debate as a debate, to not be biased or one-sided, to openly admit that there may be mitigating factors influencing two observed trends and relentlessly seek what those factors might be, to be as open and transparent as possible. Those emails unfortunately show clear evidence to the contrary.
I have never witnessed an open scientific debate turn to a one-sided “fact” in such a short period of time as man-made climate change has in the last two to three years. I find it incongruous that, on this issue alone, the skeptics are not the scientists themselves, when, on every other scientific debate that has arisen in the last few decades, the scientific view has always been the skeptical one. It seems to me this trend is directly correlated to an increase in government funding, the formulation of the IPCC, and the popularity of “An Inconvenient Truth.” But, then again, what do I know, I’m just a scientist who studies trends in culture and public opinion, and the resulting consequences for society as a whole.
Joe inadvertently makes a point against the denial movement:
“Regarding the data, what you say is simply not true. the data in the public domain has already been collected (”raw” data) and processed (”homogenized”) to turn it into consistent working datasets. There’s absolutely nothing wrong with that because it’s been collected from different sources using different protocols. The raw data is essentially useless for any analysis until that homogenization has been done.
However, if the homogenization process is flawed it can introduce a lot of errors or bias into the working dataset and, without the original raw data there is no way to verify if this has happened. It is the original, raw, data and details of the homogenization process that the sceptics have been trying to obtain in order to check the validity of the working data sets.”
Why don’t the deniers just make their OWN homogenization process, and publish in the scientific literature? That would be far better. Perhaps they are incompetent.
Please continue comments on the new thread.