This is a continuation of the last thread which is getting a little unwieldy. The emails cover a 13 year period in which many things happened, and very few people are up to speed on some of the long-buried issues. So to save some time, I’ve pulled a few bits out of the comment thread that shed some light on some of the context which is missing in some of the discussion of various emails.
- Trenberth: You need to read his recent paper on quantifying the current changes in the Earth’s energy budget to realise why he is concerned about our inability currently to track small year-to-year variations in the radiative fluxes.
- Wigley: The concern with sea surface temperatures in the 1940s stems from the paper by Thompson et al (2007) which identified a spurious discontinuity in ocean temperatures. The impact of this has not yet been fully corrected for in the HadSST data set, but people still want to assess what impact it might have on any work that used the original data.
- Climate Research and peer-review: You should read about the issues from the editors (Claire Goodess, Hans von Storch) who resigned because of a breakdown of the peer review process at that journal, that came to light with the particularly egregious (and well-publicised) paper by Soon and Baliunas (2003). The publisher’s assessment is here.
Update: Pulling out some of the common points being raised in the comments.
- HARRY_read_me.txt. This is a 4 year-long work log of Ian (Harry) Harris who was working to upgrade the documentation, metadata and databases associated with the legacy CRU TS 2.1 product, which is not the same as the HadCRUT data (see Mitchell and Jones, 2003 for details). The CSU TS 3.0 is available now (via ClimateExplorer for instance), and so presumably the database problems got fixed. Anyone who has ever worked on constructing a database from dozens of individual, sometimes contradictory and inconsistently formatted datasets will share his evident frustration with how tedious that can be.
- “Redefine the peer-reviewed literature!” . Nobody actually gets to do that, and both papers discussed in that comment – McKitrick and Michaels (2004) and Kalnay and Cai (2003) were both cited and discussed in Chapter
2 of3 the IPCC AR4 report. As an aside, neither has stood the test of time. - “Declines” in the MXD record. This decline was
hiddenwritten up in Nature in 1998 where the authors suggested not using the post 1960 data. Their actual programs (in IDL script), unsurprisingly warn against using post 1960 data. Added: Note that the ‘hide the decline’ comment was made in 1999 – 10 years ago, and has no connection whatsoever to more recent instrumental records. - CRU data accessibility. From the date of the first FOI request to CRU (in 2007), it has been made abundantly clear that the main impediment to releasing the whole CRU archive is the small % of it that was given to CRU on the understanding it wouldn’t be passed on to third parties. Those restrictions are in place because of the originating organisations (the various National Met. Services) around the world and are not CRU’s to break. As of Nov 13, the response to the umpteenth FOI request for the same data met with exactly the same response. This is an unfortunate situation, and pressure should be brought to bear on the National Met Services to release CRU from that obligation. It is not however the fault of CRU. The vast majority of the data in the HadCRU records is publicly available from GHCN (v2.mean.Z).
- Suggestions that FOI-related material be deleted … are ill-advised even if not carried out. What is and is not responsive and deliverable to an FOI request is however a subject that it is very appropriate to discuss.
- Fudge factors (update) IDL code in the some of the attached files calculates and applies an artificial ‘fudge factor’ to the MXD proxies to artificially eliminate the ‘divergence pattern’. This was done for a set of experiments reported in this submitted 2004 draft by Osborn and colleagues but which was never published. Section 4.3 explains the rationale very clearly which was to test the sensitivity of the calibration of the MXD proxies should the divergence end up being anthropogenic. It has nothing to do with any temperature record, has not been used in any published reconstruction and is not the source of any hockey stick blade anywhere.
Further update: This comment from Halldór Björnsson of the Icelandic Met. Service goes right to the heart of the accessibility issue:
Re: CRU data accessibility.
National Meteorological Services (NMSs) have different rules on data exchange. The World Meteorological Organization (WMO) organizes the exchange of “basic data”, i.e. data that are needed for weather forecasts. For details on these see WMO resolution number 40 (see http://bit.ly/8jOjX1).
This document acknowledges that WMO member states can place restrictions on the dissemination of data to third parties “for reasons such as national laws or costs of production”. These restrictions are only supposed to apply to commercial use, the research and education community is supposed to have free access to all the data.
Now, for researchers this sounds open and fine. In practice it hasn’t proved to be so.
Most NMSs also can distribute all sorts of data that are classified as “additional data and products”. Restrictions can be placed on these. These special data and products (which can range from regular weather data from a specific station to maps of rain intensity based on satellite and radar data). Many nations do place restrictions on such data (see link for additional data on above WMO-40 webpage for details).
The reasons for restricting access is often commercial, NMSs are often required by law to have substantial income from commercial sources, in other cases it can be for national security reasons, but in many cases (in my experience) the reasons simply seem to be “because we can”.
What has this got to do with CRU? The data that CRU needs for their data base comes from entities that restrict access to much of their data. And even better, since the UK has submitted an exception for additional data, some nations that otherwise would provide data without question will not provide data to the UK. I know this from experience, since my nation (Iceland) did send in such conditions and for years I had problem getting certain data from the US.
The ideal, that all data should be free and open is unfortunately not adhered to by a large portion of the meteorological community. Probably only a small portion of the CRU data is “locked” but the end effect is that all their data becomes closed. It is not their fault, and I am sure that they dislike them as much as any other researcher who has tried to get access to all data from stations in region X in country Y.
These restrictions end up by wasting resources and hurting everyone. The research community (CRU included) and the public are the victims. If you don’t like it, write to you NMSs and urge them to open all their data.
I can update (further) this if there is demand. Please let me know in the comments, which, as always, should be substantive, non-insulting and on topic.
Comments continue here.
(A little humor)
I wonder why some contrarians haven’t jumped to the much less obvious conclusion (rozaR s’maccO – backwards) that these emails are not a result mainly of changes in solar activity. Fossil fuel energy supplied to the computers is from the sun, and cosmic rays could certainly affect electronic equipment. Why should anyone infer human behavior from these emails – after all, they can’t predict what P. Jones will have for breakfast tomorrow morning, so how could they possibly know what his job description is? There are some neural networks in the heads of scientists that have yet to be fully described – the anthropic models are incomplete and can’t reproduce the Hoola Hoop fad in simulations! And sensitivity to emails is only 0.9 times what it was thought to have been. Which means it is really only 0.9*0.9*0.9*0.9 … about 0.3 times what it was thought to have been. Not to mention that having more emails automatically boosts the rate of scientific discovery, regardless of content, or of funding for labs, computers, and satellites! In fact, the internet can’t have any net effect on our lives, because people managed to survive the Stone age. There have been changes in emails in the past, so people sending emails now can’t be responsible for them. Climate science has been around for over 100 years, while emails have been around a fraction of that time – clearly, emails cannot have any affect on science; science causes emails. Besides which, the Second Law of Infodynamics dictates that emails can only flow in one direction, so any back-and-forth must be a fraud – and anyone who says otherwise must have delusions of grandeur! (The first law of Infodynamics requires that information has to come from somewhere, and since the exchange of emails results in accumulation of information, this violates physics). An email can’t even really exist – if you print it out, it looks different on paper than on screen. And we don’t really know they’re from the UK, so we should assume these were written in Aramaic. Hey, they don’t make any sense at all in Aramaic. I guess these people don’t know any language at all, since in Aramaic, it is all gibberish. Anyway, the vast majority of all emails are written about volcanic eruptions and have nothing to due with climatology (please ignore contradiction with beginning of paragraph).
Gavin, in your response
-[Response: With all due respect you have things very confused. Nothing depends on the 12 trees in the Yamal record. Toss them out completely if you aren’t happy with them. The instrumental temperature records are plenty enough to demonstrate the warming since then without recourse to trees. You don’t like dendroclimatology at all? Fine, toss that too. It just isn’t that key to anything important (see here). And I think you may be a little confused about CRU does – they chiefly collate one version of the instrumental temperature data. They don’t make projections, and they don’t have one of the main climate models. Your statement about vineyards in the medieval period is very out of date as well. In fact, your whole argument neglects the fact that CRU are just a very small part of the climate science establishment, and every single one of their important contributions has been replicated by independent groups around the world. – gavin]
I see nothing in your response to indicate that the main drivers of temperature increases since the time frame after the Yamal records were mainly driven by the warm cycles of the PDO and AMO. Since I can not find a single post to dispute that any warming has taken place in that time frame, why are you posting general statements about warming without indictating the forces behind it. What is expected to happen now that we have shifted to a cold PDO? What happens in 10 to 15 years when the AMO goes cold? Will we go back to tree ring data?
[Response: Huh? A temperature record on it’s own does not prove any attribution to a cause. That requires some kind of model – statistical, GCM or whatever, but attribution is a whole different thing. But there is no evidence that AMO and PDO have cause the long term trends in the data. (Hint, look at how they are defined). No one is going to use trees in lieu of modern instrumental and satellite data for temperatures today or tomorrow or even in 15 years time. – gavin]
“will have for breakfast tomorrow morning, so how could they possibly know what his job description is?”
Of course, if he has cereal at 6:00 AM one morning, we can infer he has cereal every morning at 6:00 AM.
That about covers it.
Who is also a well-known far-right libertarian crank who is very, very busily demonstrating that he doesn’t know squat about what he’s posting on, regarding the snippet of code he believes totally destroys all of climate science.
I made a handful of posts there, bid adieu, and won’t be back.
My software engineering cred is at least comparable to ESR, but despite that, I don’t think (as he does) that this means I know more about climate science that professionals in that field do.
He’s an ass, and he’s wrong, and if you spend just a few moments thinking about his “oh my god, this proves all of climate science is wrong, because I found a snippet of code demonstrating hacks to account for the divergence problem which is a huge smoking gun secret” followed, later, by his saying “oh, I’ve known of the divergence problem for a long time” … it’s all BS.
Please go away until you’ve studied GISS Model E’s documentation and code.
And quit telling people to “free the code” that’s been online for … a long time, now.
Oh, and also, please stop beating your wife and free her. We know you must be, just like you know that GCM code is not “free”.
Hey Ron R,
Which of the gratuitus assertions made in the quote by your guru, Peter Raven, (other than the fact that the earth is overpopulated) can stand the test of verification to be anything other than pure hyperbole!
He claims that mankind has altered the composition of the atmosphere ‘profoundly’. Well, I guess that if increasing the concentration of a trace gas by a few hundredths of a percent is profound then he’s correct.
Come on! A quarter of the topsoil, a fifth of the agricultural land? I’d like to see the basis for the calculations that genreated those numbers. His claims about the loss of a ‘major proportion’ of forests and increases in species extintion by ‘several hundred times’ sound like they come from the hysterical folks at WWF or NRDC.
By the way, did you intentionally leave out the part about his eugenic solutions to these problems?
“Where do we stand in our efforts to achieve a sustainable world? Clearly, the past half century has been a traumatic one, as the collective impact of human numbers, affluence (consumption per individual) and our choices of technology continue to exploit rapidly an increasing proportion of the world’s resources at an unsustainable rate … during a remarkably short period of time, we have lost a quarter of the world’s topsoil and a fifth of its agricultural land, altered the composition of the atmosphere profoundly, and destroyed a major proportion of our forests and other natural habitats without replacing them. Worst of all, we have driven the rate of biological extinction, the permanent loss of species, up several hundred times beyond its historical levels, and are threatened with the loss of a majority of all species by the end of the 21st century.” -Peter Raven, past president of AAAS, the American Association for the Advancement of Science.
[Response: Huh? A temperature record on it’s own does not prove any attribution to a cause. That requires some kind of model – statistical, GCM or whatever, but attribution is a whole different thing. But there is no evidence that AMO and PDO have cause the long term trends in the data. (Hint, look at how they are defined). No one is going to use trees in lieu of modern instrumental and satellite data for temperatures today or tomorrow or even in 15 years time. – gavin
Sarcasm!
As for the PDO…
Christy, J. R., R. W. Spencer, W. B. Norris, W. D. Braswell, and D. E. Parker (2003),
Error estimates of version 5.0 of MSU/AMSU bulk atmospheric temperatures, J.
Atmos. Oceanic Technol., 20, 613- 629.
Douglass, D.H., and R. S. Knox, 2005. Climate forcing by volcanic eruption of Mount
Pinatubo. Geophys. Res. Lett., 32, doi:10.1029/2004GL022119.
Forster, P. M., and J. M. Gregory (2006), The climate sensitivity and its components
diagnosed from Earth Radiation Budget data, J. Climate, 19, 39-52.
Gregory, J.M., R.J. Stouffer, S.C.B. Raper, P.A. Stott, and N.A. Rayner (2002), An
observationally based estimate of the climate sensitivity, J. Climate, 15, 3117-3121.
Intergovernmental Panel on Climate Change (2007), Climate Change 2007: The Physical
Science Basis, report, 996 pp., Cambridge University Press, New York City.
Schwartz, S. E. (2007), Heat capacity, time constant, and sensitivity of the Earth’s
climate system. J. Geophys. Res., 112, doi:10.1029/2007JD008746.
Spencer, R.W., W. D. Braswell, J. R. Christy, and J. Hnilo (2007), Cloud and radiation
budget changes associated with tropical intraseasonal oscillations, Geophys. Res.
Lett., 34, L15707, doi:10.1029/2007GL029698.
Spencer, R.W., and W.D. Braswell (2008a), Satellite measurements reveal a climate
system less sensitive than in models, Geophys. Res. Lett., submitted.
Spencer, R.W., and W.D. Braswell (2008b), Potential biases in cloud feedback diagnosis:
A simple model demonstration, J. Climate, November 1
[Response: Is that sarcasm too? Not one of those papers has anything to do with the PDO. And you think that citing the IPCC report supports your case? Funny! – gavin]
#613
The CRU emails contain an email from McIntyre accusing a scientist (not Mann) of blocking his personal IP address. As some of you may know, these types of accusations have been made at various times by McIntyre.
That reminds me of the time where McIntyre devoted a blog post to speculation that his ClimateAudit website was blocked in a parental control blacklist database.
http://www.climateaudit.org/?p=6217
This was back in the days when I commented occasionally. I proved convincingly (or so I thought) that the problem was triggered by specific content, and there was no “blocking” as such:
http://www.climateaudit.org/?p=6217#comment-344660
http://www.climateaudit.org/?p=6217#comment-344860
But some people just want to believe what they want to believe:
http://www.climateaudit.org/?p=6217#comment-344880
Gnrnr:
In your field, instrument calibration isn’t ever necessary?
In your field, long-term data series don’t confront instrument calibration issues as technology changes over century timescales?
I don’t believe this, either. On the other hand, I will believe you if you tell me that in your field, field conditions (which gavin mentions) aren’t a problem, because maybe you’re just sticking a probe up some person’s ass, or are instrumenting some laboratory, rather than working in the field.
Come to my lair, my friend, and I’ll teach you everything you need to know to convince you that you’re right, that no change in the concentration of a trace gas by a few hundredths of a percent could possibly be profound.
Bring a blank will and be prepared to sign over your assets to me first, though.
You’ll have no reason not to if you have the faith of your convictions …
Re: unable to release data from Met data belonging to other countries… read Willis Eschenbach’s account of his request to you for just the list of stations and how your team stonewalled him. Surely the station mapping data was not proprietary since it wasn’t the station data itself and you could’ve easily complied with his request without forcing him to pursue an FOI request. This looks really bad – it looks like you are hiding something or at the very least acting far outside of the spirit of honest science. I can’t wait to hear an explanation. Read it here: http://wattsupwiththat.com/2009/11/24/the-people-vs-the-cru-freedom-of-information-my-okole%E2%80%A6/ (skip down to “Guest post by Willis Eschenbach …”).
I agree with many posters here…treat this as you would open source software..in cases where you aren’t allowed to release data, you can still release code and instructions on how you can reproduce the analysis/models so someone who does buy the data can replicate your results. Something so important as this demands this level of behaviour at a minimum.
To Gavin – RE my comment 602: answers to your questions:
what was this for? and what paper did it appear in? it appears to be (do I know absolutely for sure – no – how could I -but appears to match the content of the papers
Note one program is dated 3/4/99 just prior to the 2000 paper and is called “briffa_sep98_d.pro” so might have been authored by or involved Briffa’s data. The latest paper states “To overcome these problems, the decline is artificially removed from the calibrated tree-ring density series, for the purpose of making a final calibration. The removal is only temporary, because the final calibration is then applied to the unadjusted data set (i.e., without the decline artificially removed). Though this is rather an ad hoc approach, it does allow us to test the sensitivity of the calibration to time scale, and it also yields a reconstruction whose mean level is much less sensitive to the choice of calibration period”
please therefore publish the remainder of my post with the code and please can you or anyone explain the following:
1. why didn’t the papers reveal the “fudge factor”(s)- not my words the programmer’s- used to produce the calibration of tree ring density? They happen to show an almost exponential rise post 1958. I find no justification of this.
2. why do said “fudge factor”(s)-there are twenty to be applied to different time periods – increase some years, decrease others, and leave others the same whereas the papers say in effect only increase because of a recent decline – and why is that even permissible?
2. why is it permissible to apply a “VERY ARTIFICIAL” adjustment to the “DECLINE” because of the “problems otherwise induced by” the “recent decline in high latitude tree-ring density?”
3. Although the 2 papers only mention “adjusting” the data for purposes of obtaining a “temporary” “calibration”, is it not true that when a “VERY ARTIFICIAL” adjustment is used to create a “VERY ARTIFICIAL” calibration, and this “VERY ARTIFICIAL” calibration is then applied to the raw data, what you end up with is “VERY ARTIFICIAL” data?
4. Why does the code I cited proceed to plot the “fudged” data with the “VERY ARTIFICIAL” adjustments and does not plot the raw data with the “fudged” calibration (as the paper states)? If you are only using the “fudged” data for calibration purposes, why plot it?
5. The programming files also use the words “VERY ARTIFICIAL…” , but the papers make this sound so routine and don’t use that adjective. Why use the word VERY unless you are implying “too much?”
I’m really just curious and don’t have any agenda or axe to grind, so please, anyone, assure me that there is a reasonable scientific rationale for these issues. Thanks!
[Response: (I reformatted for clarity). The issue is that unless you look at the papers (and one seems not to be online anywhere, and the other only the CRU webserver (which is down), you can’t tell what was done and why and whether it was in fact justified. Maybe if someone has actually read the papers, they can comment? – gavin]
DocMartyn says: 25 novembre 2009 at 5:25 PM
> [invasive European] earthworms … did they appear in the White Mountains,
> the Inyo Mountains, and the Panamint Range in California?
No mention of them that I can find, but you should ask a reference librarian to help you with this kind of question. None appear in those areas on this map, which has some further references: http://www.archbold-station.org/abs/staff/pbohlen/publications/Hendrix&Bohlen_Bioscience_2002.pdf
#587 Sloop
Thanks for the interesting and thought provoking response.
First off, on the issue of accessability of publicly funded climate research data, code, and methods – as a matter of principle, I have a hard time accepting that accessability should be limited in any way. The issue of Steve’s “approach” to dealing with the climate research “community” should not detract from this fundamental principle. With perhaps very limited and trivial exceptions, publicly funded data, code , etc needed for independant verification does not belong to the “community”.
I commend you for recognizing the potential value of “contrarian” science. I think the concept in your proposal is interesting although I don’t share your view on SM’s credibility. In any case, perhaps the best interests of the concept would be better served with independant participants who do not have vested interests in the outcome.
Re: ccpo, 1085, 5, 40 etc and others
You should get a proper historical or preferably geological context to what you are preaching. I have been reading these posts in utter amazement. Here’s my favorite quote from this evening’s time spent reviewing comments about this email hack:
“Where I’m sitting was once the bottom of a shallow sea with a near tropical climate. Today it’s snowing and the sea is 1000 miles away. Now that’s some serious climate change and man had nothing to do with it”
Get some perspective. Take off the blinkers.
BTW, If it sounds like a duck and looks like a duck …..
[Response: Logical fallacy. If forest fires can occur naturally, does that mean arson does not exist? Quack. – gavin]
regarding 650
NZ raw data vs Corrected.
http://nzclimatescience.net/images/PDFs/global_warming_nz2.pdf
Here is a link to the paper in question.
In short it contains no corrections for anything. We know that AT LEAST the met stations were changed from min-max thermometers to hourly recording. Exactly when I don’t know. NIWA has yet to respond, but since the question was raised in Parliament it IS likely that they actually WILL have to pull themselves together and answer.
There is no particular reason to expect that the NZClimateScience organization got it completely right. They have a strong bias. I personally am surprised a little, to see the more rapid increase here that NIWA reports. We’re in the middle of the Southern Ocean, and not a big enough Island to have strong climate trends that are very independent of that ocean.
It’s curious enough, on the other hand, our vinyards are thriving (where have I heard that song before?). Matter of fact that reminds me that we cracked open a very nice red and there is some left :-)
respectfully
BJ
With all the attacks going on here I think it is time to bow out. I am not here to argue the science anyway, just the ethics. But I will leave you with this highly negative story from CBS(BTW Gavin you are mentioned by name):
http://www.cbsnews.com/blogs/2009/11/24/taking_liberties/entry5761180.shtml
The irony of this situation is that most of us expect science to be conducted in the open, without unpublished secret data, hidden agendas, and computer programs of dubious reliability. East Anglia’s Climatic Research Unit might have avoided this snafu by publicly disclosing as much as possible at every step of the way.”
I hit send before I was finished Good Luck Gavin. I hope everything works out for the best.
Mark: The code you cite does not apply the correction factor. The line where it is applied is commented out.
[Response: (I reformatted for clarity). The issue is that unless you look at the papers (and one seems not to be online anywhere, and the other only the CRU webserver (which is down), you can’t tell what was done and why and whether it was in fact justified. Maybe if someone has actually read the papers, they can comment? – gavin]
Would you take a look at them gavin, when you’re able? And if you find that it was unjustified, would it change your view on the strength of global warming evidence?
“[Response: Yawn. If you want raw data go to GHCN. If you want current temperatures from exclusively public-domain sources go to GISTEMP. If you want all the raw data that went into those figures go to NOAA Paleoclimate. If you just want to rag on scientists, go somewhere else. – gavin]”
I’m sorry that you are tired. But you misunderstood my request. I am not requesting raw data. I want to be able to recreate exactly the global warming that you are predicting. I want to see the modeling software and the data that was input into it to determine that we have serious global warming issues. Preferably without the comments removed from the software since it appears to mostly be spaghetti code.
And I don’t want to disparage scientists at all. At least, not the traditional type of scientist that shows his methodology and data to anyone who asks for it. Please don’t put words in my mouth, Gavin. I love them, and am one. Thanks in advance!
[Response: If you want to run models used in AR4, try either GISS ModelE or NCAR CCSM – both source codes are publicly available. The NCAR model is better documented. That code is exactly what was used in the AR4 runs. – gavin]
I’ve been doing some of my own global temperature anomaly models (which are relatively simple, with variables for the sun, greenhouse gases, aerosols, volcanoes, and ENSO). All I can say is that 2010 is going to surprise a few people. Given the projected El Nino forecast, it’s not just going to be the hottest year on record. It’s on pace to get shattered by more than 0.1C. And with the Sun the coldest it’s been since the 1930s, I’m curious how this is going to get spun. (If the Sun was at a solar max as per the last couple of solar cycles, the temperature record is estimated to be broken by 0.25C, according to my model). Now, who knows for sure what the actual ENSO activity will be, and who knows when the next time the next big strato-volcano comes, but it’s going to take quite a bit for it not to happen.
(ENSO forecast found here at NOAA’s website):
http://www.cpc.noaa.gov/products/analysis_monitoring/lanina/images/nino34SSTMon.gif
Gavin,
The lawsuit that is being filed against you could beused as an opportunity to grill its denialist sponsors. What I would suggest is an attempt to expose the fact that the a sceptics because they are letting political ideology control their beliefs on scientific matters.
Don’t attack their ideology. They seem to be people who do not understand the inexorability of natural processes. Theybdo not understand that the Universe has no politics and it doesn’t matter how good they are at convincing people. They are looking for reasons to continue believing what they want to believe and cannot understand that this is wrong.
Quoting from “Mail Online”. I hope that Hudson will reveal who sent him the stolen emails:
Climate change scandal deepens as BBC expert claims he was sent leaked emails
The controversy surrounding the global warming e-mail scandal has deepened after a BBC correspondent admitted he was sent the leaked messages more than a month before they were made public.
Paul Hudson, weather presenter and climate change expert, claims the documents allegedly sent between some of the world’s leading scientists are of a direct result of an article he wrote.
In his BBC blog three days ago, Hudson said: ‘I was forwarded the chain of emails on the 12th October, which are comments from some of the world’s leading climate scientists written as a direct result of my article “Whatever Happened To Global Warming”.’
That essay, written last month, argued that for the last 11 years there had not been an increase in global temperatures.
Read more: http://www.dailymail.co.uk/news/article-1230943/Climate-change-scandal-BBC-expert-sent-cover-emails-month-public.html#ixzz0XxC2ToVf
[Response: This came up before, Hudson was forwarded a single chain of emails involving his piece, which subsequently was part of the hack. He wasn’t saying he saw all the stolen emails. – gavin]
Re: CRU data accessibility.
National Meteorological Services (NMSs) have different rules on data exchange. The World Meteorological Organization (WMO) organizes the exchange of “basic data”, i.e. data that are needed for weather forecasts. For details on these see WMO resolution number 40 (see http://bit.ly/8jOjX1).
This document acknowledges that WMO member states can place restrictions on the dissemination of data to third parties “for reasons such as national laws or costs of production”. These restrictions are only supposed to apply to commercial use, the research and education community is supposed to have free access to all the data.
Now, for researchers this sounds open and fine. In practice it hasn’t proved to be so.
Most NMSs also can distribute all sorts of data that are classified as “additional data and products”. Restrictions can be placed on these. These special data and products (which can range from regular weather data from a specific station to maps of rain intensity based on satellite and radar data). Many nations do place restrictions on such data (see link for additional data on above WMO-40 webpage for details).
The reasons for restricting access is often commercial, NMSs are often required by law to have substantial income from commercial sources, in other cases it can be for national security reasons, but in many cases (in my experience) the reasons simply seem to be “because we can”.
What has this got to do with CRU? The data that CRU needs for their data base comes from entities that restrict access to much of their data. And even better, since the UK has submitted an exception for additional data, some nations that otherwise would provide data without question will not provide data to the UK. I know this from experience, since my nation (Iceland) did send in such conditions and for years I had problem getting certain data from the US.
The ideal, that all data should be free and open is unfortunately not adhered to by a large portion of the meteorological community. Probably only a small portion of the CRU data is “locked” but the end effect is that all their data becomes closed. It is not their fault, and I am sure that they dislike them as much as any other researcher who has tried to get access to all data from stations in region X in country Y.
These restrictions end up by wasting resources and hurting everyone. The research community (CRU included) and the public are the victims. If you don’t like it, write to you NMSs and urge them to open all their data.
[Response: Halldor, thanks. I’m going to move this up to the top. – gavin]
If I can add one thing to this now ridiculously overblown topic: Gavin et al.: the attacks on your work are on a scale unprecedented since the Inquisition. Not even the tobacco industry had the gall to haul scientists before the senate multiple times, attempt to silence senior research scientists by using a stooge government to rewrite their press releases and now this. I don’t know how you keep going but a lot of us really appreciate that you do.
One thing I am convinced of: if there is a serious alternative hypothesis to explain the climate, the denial bunch had the resources to find it, and the fact that they are using this sort of tactic instead says it all.
May I suggest editing out the whole of comment #133 by Alan Millar (25 November 2009 @ 8:34 AM) on the now-closed Copenhagen thread as it is *identical* to his #533 (25 November 2009 at 10:17 AM) on this thread.
It got soundly thrashed here (e.g. Tamino at #548, 11:17 AM), and makes no more sense when reposted unchallenged on a fresh thread just before closing time. Besides which, it is off topic there according to Eric’s clear strictures.
Obviously, I do hope you are not reading this today…
Marco,
Just listen to yourself:
“567.I just HAVE to respond to Mike M (#546 at this moment):
The issue at hand is not that Steve McIntyre is a sceptic. The issue is that he is obfuscating. Yes, he has found mistakes before, great. But he’s taken those mistakes and enlarged them to superhuman proportions. Sometimes he has also claimed mistakes (see the whole Briffa issue) where there essentially were none, and resulting in allegations of fraud uttered by his cheering crowd. He *could* have been a valuable asset to climate science (and science in general) if he had a constructive attitude. He doesn’t.
Do remember that the mistake he found in the GISTEMP series is STILL widely reported (by skeptics) as evidence that Jim Hansen (and Gavin Schmidt by extension) don’t know what they are doing, and that none of their results are to be trusted.” End quote
So what that some idiots made a mountain out of a molehill regarding Steve McIntyre’s uncovering of some errors in the data? Some idiots make the same mountian out of a molehill in order to make silly alarmist statements. So what? The fact you are so defensive about Steve McIntyre is really worrying.
Also the grand feeble excuse about the CRU data not being theirs to release through FOI is really ridiculous. Since when is publicly funded climate data secret information? If we are all being asked to drastically change our lives over a “theory” based on a “computer model” then there should be no embargo on that data. You cant have your cake and eat it at the same time, which is what you imply by defending Phil Jone’s deplorable attitude toward a full and vigourous independent analysis of the data used to make very serious longterm decisions which will effect every human being on this planet.
The longer you guys support this suppression of data, FOIs and properly critical and independent peer review process the more agnostics such as myself will become sceptical.
Really you guys are your own worst enemies. And this is coming from someone who agrees that global warming is occuring and believes its perfectly feasible that humans are responsible for some of it.
[Response: There is much publicly collected data outside the US (which is much better in this regard) that is not available without payment. The Ordanance Survey maps in the UK for instance. Met Offices are often tasked to be revenue generating and so restrict access to some data to paying customers except in some academic uses (such as the data sent to CRU). While understandable, this is antithetical to the openness and transparency that people are (rightly) demanding. But you are blaming the wrong people here. (Note that the CRU data is not a computer model in the sense you imply – by the way, all computer model output used in AR4 is publicly available at PCMDI or ClimateExplorer). – gavin]
I can’t help but admire your perseverence, Gavin. I disagree with some of your more regular contributors – opening up the discussion has been helpful.
I don’t find your explanation of the FOI-request refusal satisfactory, given the contents of the emails concerned and the divergent (ho ho) explanations that were given at the time. I’m sure that this issue will develop over the coming weeks.
[Response: The response to the 2007 FOI request is the same as the response in Nov 2009. – gavin]
In any case, the only way to achieve lasting closure will be for an independent review to take place. I wonder if you’d support this?
JCS: I am not convinced by anything I have read, seen, studied or experimented that there is a definitive correlation between CO2/Greennhouse gases and climate variability.
BPL: Then you can’t have spent much time on it–about 15 minutes, maybe? Try here:
Houghton, John T. 2002 (1977). The Physics of Atmospheres.
Petty, Grant W. 2006 (2002). A First Course in Atmospheric Radiation.
Goody, R.M. and Y.L. Yung 1989. Atmospheric Radiation.
Hartmann, Dennis 1994. Global Physical Climatology.
Weart, Spencer 2008 (2003). The Discovery of Global Warming.
Philander, George S. 1998. Is the Temperature Rising?
You might also try going to Google Scholar and checking out the names Kasting, Walker, Hays, Berner, and Lasaga.
Hi,
I was told yesterday that the data sharing agreements between CRU and the national met services don’t exist after all, and that where they do, they’re verbal only.
[Response: Not quite true. Some of the written agreements were provided as a response to an FOI request, but many of them are apparently based on more informal practice and agreements. I don’t see how that means they can be ignored with impunity. It would be better if they didn’t exist, but just imagining that they don’t is not a solution. – gavin]
Granted, I was told this by Ross McKitrick (following a seminar he gave in which he was very well restrained but did appear to be a little gleeful at the whole situation), but it got me thinking.
Why do people go to the middle people for information? Isn’t it obvious to go straight to the NMSs?
I’ve learnt this week that continuing to follow climate change science after my degree is fun, but frustrating. I’m getting more than a little downhearted by it all, so RealClimate has to keep doing what it’s doing to try and keep some sort of sensibility on the web. Thanks Gavin et al!
JimM says, ” guess I’ve been around long enough to see more than a few scientific “facts” be later found to be mistakes, and theories that we were fairly certain of be ultimately disproven.”
OK, JimM, I’ll bite, hows about you tell us cite some theories where better than 90% of the researchers in the field agreed AND where the theory had been examined by the entire scientific community and found to be solid AND where the basic theory had been the same for about a century AND where there were mountains of empirical data supporting the theory…
The choice is really very simple: We can make policy consistent with the science or we can go 180 degrees against the science. Science or Anti-science. Pick.
RE Joe Duarte @ 636
Via Open Mind, I found this bit of information. Apparently, the line of code (below the ESR-quoted section) that uses this “artifical” adjustment is actually a comment (preceded by a semi-colon). Caveats: I haven’t seen the whole program and I am not a programmer and couldn’t pass for one on television. As “wingding” notes:
And Happy Thanksgiving, Gavin and US-based RC crew. I’m kinda thankful for their research contributions and that RC is around.
Darn. The link didn’t go through. It’s here:
http://allegationaudit.blogspot.com/2009/11/mining-source-code.html
The human capacity for deception is bizarre at times.
It’s sad, how easy it is to take quotes out of context and manipulate public perceptions. I hope that you and your colleagues handle it well, because public perception is going to have a strong influence on decisions that policy-makers make. It seems that some are deluded enough to believe that there’s a global conspiracy of some sort.
I read an editorial about this leak that basically said that responding to these criticisms with scientific explanation is like responding to being called a bastard by showing someone your birth certificate. Getting angry won’t be useful, but it should still be recognized that this is a PR attack more than anything, so responding as if its a scientific criticism may not be the most rational course of action. Don’t know what else to say, because I know nothing about PR.
Have a happy Thanksgiving. Best of luck.
FHSIV, #64
If you don’t think that “Trace gas” has much significance (385 parts per million CO2) then take that insignificant amount out of the atmosphere and the Earth becomes a frozen ice cube.
Just to say what a good model of calm thinking this whole thread is. thank you, Gavin
I saw this comment:
“It really does not matter why you cannot release the data. Nonscientists will not trust the science until the process is 100% transparent.”
And I second it.
You’re going to keep getting hell, until *everything* is documented, released and open.
That aside. Well done to the guy “Gavin” who is responding to all these comments. I can imagine its hard work!!!
OK, Patrick that’s a really funny bit of parody. Some may not appreciate it–this hack is, after all, a serious incident–but I sure do. Sauce for the goose is sauce for the gander, after all.
Or turkey, in the US–happy Thanksgiving to all.
Mark Yoxon asks: “In any case, the only way to achieve lasting closure will be for an independent review to take place. I wonder if you’d support this?”
OK, first, lasting closure of what? Do you really think denialists will every STFU, no matter how strong the evidence against them?
And what do you mean by “independent review”? It seems to me you got a problem here, since you pretty much have to be a climate scientist to understand the science, and they’ve pretty much already reached a determination–as of 50 years ago.
May I suggest a way of dealing with FOI requests for data locked up by national weather organizations. Write up a form listing all the weather services that have placed restrictions on the usen of their data with space on it for them to give their release. Send it to the person making the FOI request and tell them to get all the appropriate weather sevices to sign it. Release the data if they successfully do so.
It’s funny that near no-one is acknowledging the very large and smelly gorilla in the room..that is human population. In 1900 it stood at 1.65 Billion, now it’s 6.8 Billion. The rate of multiplicity of this large mammal over the last 100 years is unprecedented. With each nett human addition to our population there is another Co2 factory being born. Also the problem is there is getting such little space in the cities for burials of the dead that the vast majority of our dead are cremated releasing more CO2 and microparticulate compounds. Humans after all take a lot of energy to turn into ash. How much CO2 and greenhouse gasses does the average human produce through their lifetime?. I am talking here on rate of change of our pop. Not the equilibrium. For every new human over and above the death rate there is more livesotck bred, whether it be goat, chicken or if you are lucky enough beef. There is more rice farmed, more vegetables and fruit etc. Every additional human is doing their share of climate forcing. As more countries become importers as their standard of material well being is increased there is an exponential increase in their wastage.
I’ve been reading Dr Tim Garret’s study from the Univ. of Utah and he says that stabilizing the climate with the uncontrollable ‘heat engine’ of human and allied livestock popultions’ will be impossible. A global complete collapse in all regional economies for a very sustained period is our only hope. Another words..turn off the heat engine..turn off capitalism. Before you all ridicule this study..sit quietly for a while and ponder this concept..isn’t that what your inherent intuition knows is true?
Hiding the decline again
This web-page shows very clearly the effect that hiding the decline has:
http://www.uea.ac.uk/mac/comm/media/press/2009/nov/homepagenews/CRUupdate
At the bottom of the page, the upper diagram shows 3 hockey-stick plots with the ‘hide the decline’ algorithm in place
– so it gives the impression the proxies all have very good correlation in the 20th Century, and they diverge somewhat going back in time.
– so the effect of ‘hiding the decline’ is to make these proxy-plots appear more ‘skilful’ than in fact they are.
This is illustrated very clearly by the lower plot, which shows (I think) the same data plotted without a real-temp blending operation.
This shows that, actually, the proxy-plots don’t track real-temps very well, and in particular none of them seem to be able to track temps accurately above the 0.0C baseline.
The green Briffa plot is particularly bad, relying as it does, entirely on tree-rings.
Note: these plots are all cut-off at 1960 or 1980 to avoid showing too much divergence
– which, still, therefore, has the effect of implying greater ‘skill’ than in fact they possess.
So I am note sure how one can reasonably make claims about temperatures 1000 years ago, if we know these proxies (in particular tree-ring proxies) have not been demonstrated to be capable registering temperatures higher than a certain level.
(i.e. like most complex biological processes, isn’t it reasonable to assume that tree-ring growth has an ‘optimum’ temperature range, above and below which growth rates will diminish)
Just to re-iterate, it seems to me that the ‘hide the decline’ algorithm has the effect of making hockey-stick plots appear to be more skilful than in fact they are.
– and the effect of this is to mislead the viewer of these results.
[Response: We can disagree about that, but that was a figure from 10 years ago. The equivalent figure that is up-to-date is figure 6.10 in the IPCC AR4 report. There is no blending of data in those smoothed plots. – gavin]
Re: 634 J Smith. Well said! I’m with you on that one!!
You know, the level of ignorance reflected in the posts of the denialists makes a pretty strong argument against releasing data, code, emails or anything else to them.
How about it, guys? Prove me wrong. Why don’t you actually produce an analysis of value–one that actually advances the state of understanding of climate–with the data that are already available. Go ahead, we’ll wait.
Re #681 and elsewhere, the thing about oral ongoing data sharing agreements, is that you break them and you don’t get anymore data, so oral or written, they are enforced.
The Mc’s are, as usual, both wrong and political in this. Other words appear to apply also. The same thing was true about data sharing wrt the Yamal tree rings. McIntyre had the data almost as soon as he asked for it, Eli would bet that somewhere in the actual CRU Emails, is one that McIntyre was forwarded or sent, which shows that he should have known that he got the exact same set that was used in Briffa’s papers.
McIntyre’s incessant whinging that he was not told in detail about various national met services data being proprietary, something that is obvious, where the devil did he think the data came from, is just another indication of his goals.
This was obvious early, and it has a lot in common with the commented out line discussed above.
Scientists often carry out what-if models, in other words, if I do this, what happens. Often you do things that you expect to move the result in a certain way as a stress test of your model. Sometime surprises turn up, your result and your expectation are different so you investigate which was wrong. These lines get left in and commented out so you can come back to them later if needed.
When Mann told McIntyre where his FTP site was (there are indications that the original site was open earlier) McIntyre whined for months that the batch files were not exactly the same as the ones used in the MBH papers. Why should they have been? That was a working archive, not a museum piece.
Eli’s greatest compliments to Gavin. As long as we are discussing the motives of scientists, an important thing to keep in mind is that the denialists have never been on the side of an issue that was not harmful to health, wealth and happiness alone or in the various possible permutations. The best that could be said for them is that they have an extreme pro-big-corporation slant (whether defense, tobacco, fossil fuel, CFCs and more).
Just to cheer you all up, I wrote a poem for you all. It starts:
The night Mike wore his lab coat,
and made scientific discoveries of one kind and another,
the denialists called him a fraudster
and Mike said: “I’ll prove you wrong!”
So they sent his emails to the media without any context.
That very night on his blog,
a jungle of obfuscating comments grew,
and grew,
until the discussions became enflamed,
and spilled out into the internet all around.
(read the rest here: http://www.easterbrook.ca/steve/?p=990)
#648, David, You might try these sites for more up to date and comprehensive, try these. You might catch a interesting upward trend. You then might correlate this with the flattening of global temp.
http://www.ijis.iarc.uaf.edu/en/home/seaice_extent.htm
http://arctic-roos.org/observations/satellite-data/sea-ice/ice-area-and-extent-in-arctic
http://ocean.dmi.dk/arctic/icecover.uk.php
http://arctic.atmos.uiuc.edu/cryosphere/IMAGES/current.area.jpg
I wouldn’t hold my breath for the Arctic to be ice free. However what will be interesting to see is if the upward trend continues. As I said above, we will just have to wait and see, and have to remember “nature” does what it wants, regardless of the politics.
Only those with ideological blinders on will get their predetermined notion of widespread fraud confirmed.
What do you do if you don’t agree with the science (or with the perceived political implications thereof), but don’t have any real evidence to back up your position?
You could try breaking in the computer system of a renowned institute, to then release the stolen emails and documents via internet. If your catch is big enough, there will surely be something that could be spun to embarrass the scientists in question (and, by extension, discredit the whole field). Especially emails written before they had their morning coffee serve that purpose really well.
More here: http://ourchangingclimate.wordpress.com/2009/11/26/scientists-under-attack/
Gavin, I don’t envy the task you’ve taken on, but I have the utmost respect for how you’re handling this, both in providing needed context, answering numerous questions, and remaining calm and collected while doing so (at least in appearance…).