Much of the discussion in recent days has been motivated by the idea that climate science is somehow unfairly restricting access to raw data upon which scientific conclusions are based. This is a powerful meme and one that has clear resonance far beyond the people who are actually interested in analysing data themselves. However, many of the people raising this issue are not aware of what and how much data is actually available.
Therefore, we have set up a page of data links to sources of temperature and other climate data, codes to process it, model outputs, model codes, reconstructions, paleo-records, the codes involved in reconstructions etc. We have made a start on this on a new Data Sources page, but if anyone has other links that we’ve missed, note them in the comments and we’ll update accordingly.
The climate science community fully understands how important it is that data sources are made as open and transparent as possible, for research purposes as well as for other interested parties, and is actively working to increase accessibility and usability of the data. We encourage people to investigate the various graphical portals to get a feel for the data and what can be done with it. The providers of these online resources are very interested in getting feedback on any of these sites and so don’t hesitate to contact them if you want to see improvements.
Update: Big thank you to all for all the additional links given below. Keep them coming!
#50–I apologize for perhaps being a little harsh in my previous reply to you. I admit that climate change is probably not as well settled as gravity, although I think that there is little doubt that it is occurring. I did use the term deniers in the comparison to creationists, and I do think that is a reasonably valid comparison, in that they are both denying something for which there is ocnsiderable evidence. You said you are a skeptic and despite my previous post, I wouldn’t want to put honest skeptics in with deniers. If you really are a skeptic, I hope that you will examine what the skeptical web sites are saying and look at what this site and others are saying about what is happening. I think if you really look into it, you will find that a lot of the skeptical sites are distorting what the real situation is.
About #278 John o’Sullivan – Didn’t Michael Crichton get his big break by writing fiction about an alien microbe?
Mike Flynn #259: judging from the examples you cite, I wouldn’t venture out of my home, if I wanted that degree of certainty. I certainly wouldn’t cross a bridge or drive a car.
Kevan Hashemi #289: there’s a lengthy discussion on Steve McIntyre’s Climate Audit site predating the theft of the emails and documents, where they picked up the fact that the tree ring data ends in 1960, but somehow failed to read numerous mentions of the fact that this data becomes unreliable after 1960. While I have some sympathy with complaints that raw data wasn’t available, it’s hard to believe McIntyre is a serious scientist if he fails to mention (possibly failed to read?) the obvious explanation contained in the papers he is attacking. One example, from a paper cited on that same Climate Audit page: K. R. Briffa, T. J. Osborn, F. H. Schweingruber, Large-scale temperature inferences from tree rings: a review, Global and Planetary Change, Volume 40, Issues 1-2, Global Climate Changes during the Late Quaternary, January 2004, Pages 11-26, ISSN 0921-8181, DOI: 10.1016/S0921-8181(03)00095-X:
The strongest conclusion I can draw from all this is that if anyone is being deliberately dishonest, its McIntyre.
Just heard the Diane Rehm panel discussion on Copenhagen, which included Michael Mann along with a “resident scholar” from the American Enterprise Institute. I commend it to anyone, especially in the skeptic community, who still has questions about the email incident as well as the reality of the recent warming.
The whole episode is available as a podcast here: http://podcastdownload.npr.org/anon.npr-podcasts/podcast/305/510071/120952159/WAMU_120952159.mp3
Basically, Kenneth Green of the AEI served up clueless talking-points and Michael Mann smacked them down quite nicely. The scientist (Mann) was able to get in the last word on most of the exchanges with the political operator (Green), a refreshing departure from the typical talk-radio format.
Folks here who have “fence-sitting” friends/relatives should make certain that their friends/relatives download this podcast to their iPods/iPhones/iWhatevers and give it a listen.
It probably won’t change many minds that have been made up, but I’m confident that it will convince many that haven’t. Dr. Mann did a bang-up job on that interview. I hope he does more.
Re: Comment by dale — 30 November 2009 @ 10:46 AM
Hypocrisy knows no bounds, apparently:
http://mediamatters.org/blog/200911290004
“Capital Punishment for James Hansen. Climategate is high treason.”
Well what I can conclude is that if climate science is somehow unfairly restricting access to raw data upon which scientific conclusions are based, there definitely has to be certain reasons to it. Raw data is data not formatted maybe they have a different process before making data available to all.
I was a little disappointed not to see my climatology page listed in the resources… I list a lot of time series data and other useful tables. Oh, well.
And can anyone point me to extinction coefficients for Rayleigh scattering? I’ve looked on Google Scholar, but I must be using the wrong search words. I know it’s proportionate to wavelength to the negative fourth power, but that doesn’t help me with ke = ka + ks.
Philip Machanick #303: Okay, let me make sure I get this straight.
1. The tree-ring data deviates from surface thermometers after 1960, and post-1960 data was not used at all in making the hockey stick graph.
Many people have been saying the surface temperatures are wrong because of urban heating, and indeed I see evidence of urban heating when I reconstruct the surface trend myself. And yet, tree rings were rejected because they disagreed with the rise in temperature measured by surface thermometers. The fact that this disagreement has a name, “The Divergence Problem”, and that it is “Widely Known”, does not detract from the fact that most of us scientists would not react to a data conflict by rejecting one data stream and declaring the other to be flawless.
[Response: Logical fail. That actual temperatures have been rising is unequivocal and demonstrated by ocean temperatures, retreating glaciers, melting snow, etc. etc. etc. The surface met station data do not need to be flawless (and of course they aren’t) for you to know that a big decrease in one particular set of tree ring proxies is not representative of temperature. – gavin]
2. The tree-ring data was used to show temperature stability over the previous four centuries.
I guess different scientists have different styles, but when I find that a measurement method does not work, I don’t use it. In this case, it appears that the tree-ring data, having been discredited after 1960, was trusted over the past 1000 years to show remarkable stability of climate.
[Response: Other tree data and other proxies work fine for this. – gavin]
3. Many widely-publicized graphs, like this one, show tree-ring data from Briffa et al. right up to 2000.
How can it be that we have tree-ring graphs going right up to 2000, and yet the raw tree-ring data itself is not being used? [edit]
[Response: This actually only goes up to 1999 (since that was when it was made), and it can hardly be said to be well-publicised since no-one had noticed it until last week. And, as you know, this uses instrumental data for the modern period (alebit in a not clearly communicated way at the time). – gavin]
Each of these three numbered statements can be defended in isolation, but taken together they are hard to explain. I am, however, open to persuasion.
Gavin #308: Thank you for your answers. When you say “the actual temperatures” I guess I’m not sure what you mean. Do you mean that the surface temperature measurements are the final word in global temperature? If so, I accept your faith in them, but I hope you can accept my lack thereof. The number of stations used to obtain the surface plots dropped from around 8000 stations in 1960 to 1800 in 2000 (see here), which happens to be the period during which you reject the tree rings for disagreeing with the thermometers, and the period during which the world experienced unprecedented urban growth. As to melting ice: if I put a big enough lump in the garden, it will take a thousand years to melt (I live in Boston, not LA), and I would not trust the melt rate as a thermometer, let alone a +-0.1C trend-measuring thermometer. That’s just me, however, but please note that measuring temperature is one of my specialities. So instead of “logical fail”, I’d rather say “disagreement about data reliability”, which seems more dignified.
[Response: Try again. I was not specifically not claiming that the temperature data were flawless (which of course they aren’t). But I was rejecting your claim that the post-1960 MXD proxy being thought wrong was a statement that the surface temperatures are perfect. That is a problem in logic, not a disagreement about data quality. The station number issue is a red herring since there are more than enough stations to characterise global temperature anomalies on an annual basis (something like 100 good ones is all that would be required), and again, there is plenty of supporting evidence for continued warming since 1990 – satellites anyone? I suggest you try your backyard ice block experiment – a thousand years isn’t even close. – gavin]
You say that other tree ring data and other proxies also show that the climate has been stable for the past thousand years. But work like this, which uses every proxy except tree rings, shows a distinct medieval warm period and a subsequent mini ice-age. So I’m not sure I could defend your statement if I was asked to.
[Response: I never claimed that climate was stable for a thousand years. But Loehle’s reconstruction is crap. – gavin]
In your final response, you flatter me by saying that I know how instrumental data is used in the graph I linked to. The truth is: I don’t know, but I’m listening. How do you get 1960-1999 values in the Briffa plot when you have thrown away the post-1960 data? I just want the twenty-word summary.
[Response: The smooth was calculated using instrumental data past 1960. (8 words). The caption stated that both proxy records and instrumental data were plotted but wasn’t clear enough about how that had been done. The recent IPCC figures are much clearer in that respect. – gavin]
Tilo Reber says:
30 November 2009 at 12:25 PM
“As with CRU, all I have to give out is the merged product. – gavin]”
How do you know that the merged product represents the original product. Unless you either have the original, or enough metafile data to point you to the places where you got the pieces of your merged products, then the idea that your merged product represents reality is simply unsupportable.
[Response: You are welcome to go back to the original sources and do it again yourself. The point is that there is only one product (the current database), but the raw data still exists (in it’s less than perfectly accessible form). – gavin]
Also if you read the UKMet Office conditions of use for their data which has been posted here you’ll see that users are precluded from keeping their own copies of the raw data and from passing such data on to third parties. If other MOs have similar rules then it’s clear that no compiler of derived statistics would be able to keep the raw data on file.
Ow. I made the mistake of looking at Kevan Hashemi’s website: he using a method for averaging the anomalies of all weather stations in a dataset, and claims that because his average looks similar to the CRU record, his method is a valid proxy for that record. Because he doesn’t account for geographical siting of weather stations, and because he has determined that his method is a valid proxy, he therefore claims that the CRU record does not account for geographical siting either.
Speaking of logical fails…
How about the Coral Reef Temperature Anomaly Database (CorTAD): http://www.nodc.noaa.gov/SatelliteData/Cortad/
In response to all the skeptics still complaining about the lack of climate data availability, hammering Gavin about it, etc, what would happen if you tried looking for said data on the internet? Lets see.
I just googled “climate change data”
Result; over 46,400,000 hits. That is right. FORTY SIX MILLION AND FOUR HUNDRED THOUSAND hits. (and it only took 0.22 seconds)
Wow, that was easy! And I didn’t even have to file a FOIA to get those selfish, corrupt scientists to spit out their coveted data.
On page one of my search results (you can view the output here; http://www.climateshifts.org/?p=3619) there are 10 results including some newspaper stories arguing that scientists don’t share data and (ironically) also several portals where one can easily download climate data such as;
the Climate Change Data Portal, http://sdwebx.worldbank.org/climateportal/
a NASA master data directory, http://gcmd.nasa.gov/
the NOAA climate Program office, http://www.climate.noaa.gov/cpo_pa/ccdd/
the IPCC data distribution center, http://www.ipcc-data.org/
and the NOAA National Climate Data Center, http://www.ncdc.noaa.gov/oa/climate/globalwarming.html
At least five of the first ten results contain a wealth of climate data. Free for anyone to download, use and abuse. Extrapolating, without any justification whatsoever, to the all the results, means that there are 23,200,000 online data repositories. Obviously, there are not that many. But one could spend several lifetimes analyzing the data from just the first five results.
To paraphrase from Ray Pierrehumbert’s education of Steve Levitt, it is amazing what you can learn with just a tiny bit of curiosity and an internet connection.
And to speak to all the folks claiming they wrote such and such and that mean scientist didn’t respond; there could be hundreds or thousands of people asking Briffa, etc for their data. it just isn’t realistic to expect scientists to respond to EVERYONE who asks if they get popular, are being targeted, etc. I share my data regularly, but if I was getting more than a request or two day, it just wouldn’t be possible to handle the correspondence.
I think open access data is a great idea. Believe me, since scientists are the primary consumers of data, this is the way we want it! But it takes a lot of time and $ to share data in a sophisticated way. What Gavin has set up here; http://www.giss.nasa.gov/tools/modelE/modelEsrc/
is mind blowing. And far beyond the norm in environmental sciences. Lets give him some credit. And recognize that as currently structured NGOs, universities and many governmental organizations reward scientists for science, not for data sharing.
Gavin #309: Okay, you used “instrumental data” to obtain the post-1960 Briffa tree-ring graph, and there was some “smoothing” involved. Which instruments supplied the “instrumental data” that you used in the Briffa graph after 1960? (PS. Please forgive my request for twenty-word summary, which I meant as a way of showing respect for how busy you are, but I think comes across as some kind of challenge.)
Marcus #311: I am honored that you took the time to go to my site, and regret that you did not enjoy the visit. Perhaps you could explain in more detail why “integrated derivative” analysis of the station data is less reliable than CRU’s, by posting your critique here (If I am the fool you say I am, then my analysis is not worthy of this audience). All assaults upon my arguments are welcome and indeed encouraged.
Tilo Reber says:
> How do you know that the merged product
> represents the original product.
And if CRU had indeed given over a file that they claimed was their copy of the “original product” — you’d trust their copy to be correct?
Or would you be claiming that they had to show you certified copies of each original, and proof the people who certified them were certified as well?
Re: #71
In addition to the CRU, NOAA, and NASA global temperature records there is data from the Japanese Meteorological Agency (JMA):
http://ds.data.jma.go.jp/tcc/tcc/products/gwp/temp/ann_wld.html
(hat tip to Michael Schlesinger via Andy Revkin: http://dotearth.blogs.nytimes.com/2009/11/30/more-on-the-climate-files-and-climate-trends/?ref=science )
Rather amusingly, a new lobby group launched in the UK last week, the Global Warming Policy Foundation, which has been amongst the most vocal critics of UEA during the current controversy has now admitted that it had introduced an error into CRU data that it has published on its website to show ‘global cooling’: http://timesonline.typepad.com/science/2009/12/climate-sceptics-get-it-wrong-1.html
Hank Says:
“Posting faux-naive questions with outdated information to try to test your fellow reader wastes everyone’s time who bothers to try to be helpful here.”
Yet this is EXACTLY WHAT I SAID OF BURGY!!!!
FFS.
Two faced rissoles.
HAT-CRUd DATASET TAMPERING SCANDAL
In a sensational development that has set the world of client science in a turmoil, climate change skeptic activists have published photographs of leading climatologists showing several of them afflicted with a severe case of HAT-CRUd, or, in layman’s terms, dandruff. Versions of the same photographs had already been published in leading scientific journals, but, according to the activists, “photoshopped so that the HAT-CRUd is dishonestly camoflaged”.
“How can you trust people who have such questionable personal hygiene?” is one of the more printable lines of their press release. “These people clearly have no credibility, and they want to confiscate my SUV?”
One of the pictured scientists, when contacted, claimed that the white flecks in the pirated photos were “merely artifacts of a lossy data compression algorithm”, but nobody listened to the geeky slob.
Seems to me that it might help to have clear definitions when discussing so-called climate skeptics.
‘Skepticism’ is certainly a misnomer, since it implies a certain amount of open-mindedness.
‘Cynicism’ is the correct word, since climate cynics consistently accuse scientists of wrong doing.
From the dictionary:
“Cynicism: 1. An attitude of scornful or jaded negativity, especially a general distrust of the integrity or professed motives of others”
In contrast to:
“Skepticism: 1. A doubting or questioning attitude or state of mind; dubiety.”
I think it’s time to be more precise in distinguishing between skeptics & cynics in the public debate, as anyone claiming fraud is clearly a cynic, not a skeptic.
I have some thoughts about clear definitions when discussing so-called climate skeptics.
‘Skepticism’ is certainly a misnomer, since it implies a certain amount of open-mindedness.
‘Cynicism’ is the correct word, since climate cynics consistently accuse scientists of wrong doing.
From the dictionary:
“Cynicism: 1. An attitude of scornful or jaded negativity, especially a general distrust of the integrity or professed motives of others”
In contrast to:
“Skepticism: 1. A doubting or questioning attitude or state of mind; dubiety.”
I think it’s time to be more precise in distinguishing between skeptics & cynics in the public debate, as anyone claiming fraud is clearly a cynic, not a skeptic.
Ray (295), by (normal) definition, a skeptic is NOT required to have a fully developed alternative.
Gavin: don’t get too frustrated by the high number of newbies asking old questions. I don’t know what’s happened to traffic to RealClimate, but my blog has had a huge increase in visits mostly via RC (Google Analytics), and I haven’t been posting anything remarkable here, so the most plausible explanation is a lot more new curious readers on RC.
Kevan Hashemi <a href="http://www.hashemifamily.com/Kevan/Climate/"#309: if I can add a little to Gavin’s responses, if any stations are removed from the total, the overall average has to be calculated taking into account the area they used to cover. One approach is to use neighbouring sites as approximations for the missing sites. This is one of the reasons that a temperature anomaly is used rather than absolute temperature. Neighbouring sites may have different absolute temperatures but will usually have highly correlated variation. I don’t know how this is actually done, but the obvious thing to do when losing a site is to find neighbours with variation most highly correlated to the missing site, and weight those up to replace the lost site.
On the Briffa allegations, my approach has been to read his papers on the subject, and they generally explain the divergence problem, so claiming that there has been dishonesty based on finding computer code to generate graphs has no basis in reality. We have no idea what a particular program in this archive actually represents. It may be a what-if thought experiment, a test program to compare actual values with constants to check that scaling is right, a version modified since publication to compare against another data set, … I have a PhD in computer science, not clairvoyance, so there’s a limit to what I can read into a snippet of code with no context. Particularly when the code concerned was stolen under circumstances yet to be revealed. Had there been computer code showing a faked graph and that graph was published as representing the actual reconstructed temperature, then I would have been irate, as I suspect would most other readers of RC (and I hope contributors).
The courteous way you are handling critical post is a credit to you. But, gentlemen, if you do not treat the scientists that have betrayed us harshly then this field of study will be placed alongside Phrenology and Astrology.
Nice to see that efforts are made to compile the GISTEMP code and verify the output. I took a different approach by constructing a global temperature index from scratch, using the same observations used in GISTEMP along with data from other sources. First results, very crude (e.g. without station series homogenization), look quite similiar to the existing estimates, especially from 1960 onward. More details at the link, input and ideas are most welcome (e-mail can be found on the page as well).
Gavin, would it be possible to open up the raw data for plotting the O2/N2 atmospheric mixing ratio? The data is currently protected on this web site: http://bluemoon.ucsd.edu/data.html
If there is an alternate location where this data can be publicly accessed then please advise. Thank you.
Oops!
http://timesonline.typepad.com/science/2009/12/climate-sceptics-get-it-wrong-1.html
#303 – Philip. I believe I said the following: –
” I am a self uneducated retiree with some scientific background.
Does anybody on either side of this argument claim to be able to foresee the future? As far as I can see, computer models have the same predictive ability as chicken entrails, albeit using chicken entrails you get to eat a chicken as a side benefit.
Although it is tempting to believe that the past can be extrapolated into the future, it just ain’t so.
What usually happens is that people (not just scientists) confuse “assumptions” with “predictions”. Newton’s thoughts on gravity. Einstein’s thoughts on gravity etc., were all deemed to be practically useful at the time. Good enough to “predict” eclipses, demolish the luminiferous ether (believed in by a majority of scientists, I believe), and so on.
The future is unknowable. Prove that you can foresee even the next 5 seconds repeatedly with one hundred percent accuracy, and I will “predict” that civilisation as we know it, will end. You can easily work out why.”
I can’t see the “examples” to which you refer. Maybe you looked at another post.
Gavin –
You say : Fortune telling is a class-B misdemeanor in NY state, and obviously no-one here is doing that!
I say : Really? Based on the past, my chicken entrails predict iceless Antarctica, grapevines flourishing in Northern England, Greenland supporting a farming population big enough for Rome to send a Bishop, permafrost thawing, with animals grazing on previously frozen ground.
I then foretell the future beyond this. An Ice Age will descend upon the world. Glaciers will reshape the landscape. A time of cold and dark will prevail!
As I say, you can’t predict the future any better than I with my entrails. (Actually I cannot lie. I don’t use entrails, I just “predict” the past as the future.) Alas, I am forced to confess. Mea culpa.
You say (G) :
However, predictions are the lifeblood of science. Given certain, well-defined, conditions, scientific theories will predict what the result of a specific experiment will be. If the conditions are not met (the temperature is different, or it wasn’t a vaccuum etc.), the prediction doesn’t work.
I say (M) : My point exactly. You use “prediction” instead of “assumption”. And we have to wait until the predicted event either occurs, or doesn’t. As in Newton’s “laws” being found not quite able to explain some subsequent observations. Einstein’s work filled in some of the holes. However, there are some areas of Einstein’s work that don’t quite accord with observations. Oh well, most predictions were close enough. The science is settled?
G.: With respect to the future of the climate system, there are some external forces that are quite predictable – the orbit of the Earth, and its variations, the continuing rise in CO2 emissions, etc. Another example would be the climate consequences of a big volcano. In each case, you can use climate models to project the impacts on temperature, circulation etc.
M. : I don’t believe you! One the one hand, you predict growing CO2 emissions. On the other, you believe they should be lowered until they no longer grow. Please examine your models and predict CO2 levels in the year 2050? Lower or higher and by how much? Maybe this is one of the areas where the prediction doesn’t work.
G: In fact, people have done this. In 1991 Hansen et al predicted the temperature drop that was expected from the Pinatubo eruption well in advance of it happening. In 1988, the same team projected how temperatures would increase under 3 scenarios of rising greenhouse gases. For the scenario that came closest to being realised, the trends were a pretty good match to what actually happened. Other groups have made skillful projections for the El Niño events and their consequences months in advance.
M: I am not a trained “climatologist”, but I assume that a rather large cloud of particulate (aerosol) matter might result in lower air temperatures under the cloud. Is this what happened? NASA said “The predicted decrease in the global surface air temperature by 0.5 C by the end of 1992 with recovery to normal by 1995 has proven to be remark-ably accurate, though not all for the right reasons.” I cannot find a free copy of the paper (strange, that). Other citations use words like “closely”, “maybe”, “possibly”, and “somewhat less than predicted.” One thing that I DID notice was that the Pinatubo eruption reversed the effect of AGW for a year or so. Maybe that’s the answer – flood the atmosphere with aerosols of the right type. Skillful projection is what I do. Can’t bag anyone else for doing the same thing. Would you rather have a factually answer for the wrong reasons or a well reasoned, peer reviewed, ideally referenced incorrect answer?
If you are as scientist, I don’t have to point out the incorrect scientific assumptions over the years.
G: So finding useful information about the future is not impossible, even if it isn’t fortune telling as described in the NYS penal code. – gavin]
M: Yes it is. It’s just disguised as “science”. The best climate predictions
1. Are scary.
2. Won’t occur until the present Government of (insert country of choice here) has finished its current term.
3. Preferably after the predictor retires
4. Generate maximum grant funding for the predictor.
The list goes on.
Don’t get me wrong. The AGW theory may be right. But based on past history, the Earth has to get at least as hot as it has been in the past without human involvement, before I will become any more than moderately concerned. So call me after the Antarctic ice cover is all gone. Obviously, if the ice cores only go back x years, there wasn’t any ice before that time.
I wouldn’t blame you for moderating me out of existence, but let an old man get some enjoyment from his few remaining years.
Regards to all. Live well and prosper
@Rod B#320: by (normal) definition, a skeptic is NOT required to have a fully developed alternative.
In my reading of the history of science, (normal) skepticism doesn’t get very far. Bad science is tossed out not by skeptics, but by better science. Einstein’s skepticism of quantum theory didn’t amount to a hill of beans because he had nothing better to offer in that field. On the other hand, he revolutionized physics, not by being skeptical of Newton, but by offering something better (more complete).
RE: caerbannog
Mike Mann did very well – good to see another good spokesperson for the science. He was very good at framing questions to make the bigger point, in the right context. I think he also acquitted himself well with the policy/big picture questions.
Caerbannog, Diane Rehm is one of the better interviewers out there. She’s always been able to bring in some interesting people and ask great questions. Besides the regular government people, she brings in some heavy hitters from NIH for health issues.
“Interested in direct links to cosmic ray count data,” Brian Angliss — 30 November 2009 @ 2:32 PM
http://cosmicrays.oulu.fi/ ” The full database (since 1964)”
Rod B @ 320, by definition, a skeptic is not required to have a fully developed alternative… this is true, but skeptics should be able to articulate a fact-based reason as to why they are skeptical. Most “skeptics” in the realm of AGW don’t do this. They simply dismiss the 30+ years of collected data, research, and scientific literature with a hand wave. Those are the actions of a denier, not a skeptic, and at worst is intellectually lazy and dishonest.
Sorry, but that’s simply not true, not in science, anyway. You’re saying that “la-la-la I don’t believe you!” is sufficient, which come to think of it, pretty much sums up your contribution to this site over the last few years.
Cool, raw data. As an electrical engineer with 30 years of remote sensing experience, and having taken statistics and probability way back in college, I was easily go through and process raw data in Excel until it couldn’t hold any more data. Then I could have written a simple program to do the same analysis, but being lazy, downloaded one from the web.
The biggest headache was simply reading through the readme’s so I knew how to read the raw data. (For example on the v2 files knowing to divide by 100 to convert the integers to degrees.)
But after going through this (I only processed data that had the measured temperatures), I don’t get any Hockey Sticks, and seems to go down for the last 10 years. What I get matches the Sunspot cycle length vs temp models.
Now I am admittedly not a climatologist, so I don’t know the validity of converting from anything but measured values, but I can do the mathematics. But I also did not keep track of which pieces of data I used, since some of it has missing measurements, or other funkiness, and it was more an exercise in curiosity.
Many of the other posts have sighted availability of data all over the web, but almost all of these are data that is expressed as anomalies and furthermore the ones that I looked at trace back to the ones that you have sited here. Which in terms of raw data sources is not all that huge. But it is a lot of data.
People like me need to be able to follow the math from the raw data (not process it), or we will be skeptical, and I can’t get there from the raw data (so I tried to process it). When I read that people like as Freeman Dyson , Fred Singer, Ian Plimar, John Coleman, John Christy, Richard Lindzen, Henk Tennekes, and many, many others have similar issues that doesn’t help, and is exacerbated by the email scandal.
If I hear “Peer Review” as proof, I want to puke. Peer review is something that you should do to make sure your proof will hold up to everyone else reviewing what you have done.
At present, as I look out the window and read my outdoor thermometer, I think that the astrophysical models of Milankovitch and Sunspot cycle models is more accurate, which should give C02 being absorbed as the oceans cool. But maybe the CO2 models are correct and we can use it to fend of the ice age.
@Ben Lankamp#326: I took a different approach by constructing a global temperature index from scratch, using the same observations used in GISTEMP along with data from other sources.
Nice work. I hope we can your methodology in more detail later. Your work is more valuable than a simple port of GISTEMP.
A snippet of something I wrote elsewhere about levels of confirmation/corroboration.
1) Given the exact data and exact code, an audit can be made on implementation errors.
2) Given the same data but using independent methods, results of a first team can be bolstered/undermined by a second team.
3) Using independent data and independent methods, results of a first team can be bolstered/undermined by a second team.
Each has its place, but #3 has the most scientific value.
Dear Gavin, Your tireless moderation is much appreciated. I would like to print or copy text from the comments (eg. your response to comment 57), but find I cannot do this as I can’t seem to select text from this pop-up. Any advice?
“So call me after the Antarctic ice cover is all gone…. let an old man get some enjoyment from his few remaining years.”
Thanks, ‘old man’ but I’d prefer to leave a planet for my children, and their children, that is capable of supporting human civilisation in roughly the same degree of luxury as it does now. I don’t predict
extinction (the science doesn’t support that!) but it does support significant damages to agriculture,
water supplies, lose of fish, increased disease vectors, migration etc. that are very alarming.
There is ZERO evidence that you can increase CO2 in the atmosphere to 550ppm and see no significant temperature rise. If you can prove this, please do so. It would make me happier.
The is HEAPS (literally heaps and heaps) of evidence that IS NOT BASED ON MODELS that shows that if you increase CO2 in the atmosphere to 550ppm, temperature will increase by 3 degrees, give or take a degree and a half.
Based on THE SCIENTIFIC EVIDENCE which of the following statements do you believe are true?
a) Our current fossil fuel based economy will cause a significant increase in global temperatures over the course of the next century if we fail to moderate CO2 emissions
b) Nothing to see here folks. Carry on as usual
Re: the website Climate4You (#268),
it’s run by Ole Humlum, a U. of Oslo physical geographer of Danish origin, who’s done glacier work on Greenland and Svalbard (e. g. in Holocene 15(3)). He’s got a rambling 34-page essay online in Norwegian, a hodgepodge of historical factoids on climate with some hand-waving about Henry’s law and isotope ratios to suggest that the atmospheric CO2 increase is mainly from natural sources. He recently featured in a Norwegian newspaper debate with Eystein Jansen about the IPCC and the skeptics. Humlum, and a co-author who also ought to know better, wrote inter alia:
‘Nuff said.
What’s the point of posting links that require you to log in with a password? I want to examine how the stability problem in GCMs is actually dealt with.
[Response: What stability problem? However, if you are looking for explanations, you are better of reading the papers than the code. -gavin]
Mike Flynn: if you don’t want to accept any projection or prediction or whatever you want to call it, take my advice and never leave your home again.
Are you confused about what an assumption is? An assumption is something you make up (possibly plausibly, but it’s something you toss in without having to justify). Any reasonable scientific theory is not based on assumptions, but on propositions that are verifiable against reality.
Mike S.: you will not get much traction by citing the likes of Fred Singer and “Ian Plimar” (even if you spell his name right) as authorities. Singer has a long history of attacking science to support causes like tobacco, and Plimer is not a climate scientist. He’s an economic geologist, and his widely publicized (in Australia) book is riddled with errors, indicating he isn’t even a good geologist. Rather than citing bogus authority, stick to understanding the science, even if it’s hard. The people who discovered the role of CO_2 in the climate are also by and large those who first understood how Milankovitch cycles worked. If you think sunspots explain the modern climate trend, I have news for you.
Mike, this so-called email scandal is nothing compared with the concerted effort of industry shills and hangers-on to confuse the public about the science. It’s an old game started with industries like tobacco and asbestos. The only puzzle is why more people haven’t seen through it by now.
Mike Flynn,
are you the fellow who wrote Eifelheim?
Regards,
Martin
Philip (#324), If you apply our method of integrated derivatives to any set of station data, it will generate almost exactly the same curve as the CRU or NCDC methods. That’s our claim. Our existing graph is all the evidence we need to make the claim, just as a graph relating mercury column height to temperature from 0C to 100C is all we need to claim that mercury column height is a measure for temperature. You are welcome to disprove our claim by applying our method (it’s simple, here’s the code) to any other set of station data, calculate the integrated derivative curve, and compare it to an existing CRU or NCDC curve obtained from the same data. If they differ significantly, I’m wrong. But no amount of talk about smoothing and grids and compensation is going to shift me from my position. Why should it? I’m a scientist, I don’t have to be convinced by fancy talk. Now, given that we can generate the same curve with a calculation that takes no account of distribution, disappearance, nor urban heating, we can say with certainty that one of two things must be true for each of these sources of error: either it does not exist in the data, or CRU and NCDC failed to compensate for it. But we show that disappearing stations do have a strong effect upon the resulting trend. Therefore, the CRU and NCDC curves have failed to account for the loss off 80% of their weather stations during the period 1970 to 2000, which is precisely the period when their curve shows such sustained warming.
For anyone still reading this thread, and interested in GISTEMP:
David Jones and I have finally started the Clear Climate Code website, hosted by John Keyes. The first couple of blog entries describe the overall project and the progress on CCC-GISTEMP.
Thanks for making these links available – very helpful.
Can anyone shed light on the following file I came across in the NGRIP site – http://www.gfy.ku.dk/~www-glac/data/ddjtemp.txt ? This is a set of data showing historic temperatures in Greenland based on ice-cores. The following summary accompanies the data:
In Greenland:
The warmth of the climatic optimum during the European stone age 5000 years ago is clearly seen, just as the cool period during the Roman age and the
relative warmth of the Viking age (where the Norsemen settled in Iceland and
Greenland) and the two cold periods of the “little ice age” at 1600 AD and 1875 AD. The warming in Greenland in the 20th century only lasts until 1950 AD. After
that it has become colder.
This doesn’t seem to fit with everything else I’ve read about unprecedented temperatures today – and how can Greenland be getting cooler if the ice sheets have melted so dramatically?
Kevan Hashemi #343: Believe it or not, I’m also a scientist, though I don’t work in climate science. I can also read Pascal, though that was an interesting retro experience. I last used the language about 20 years ago.
Remember what we are trying to do: measure a worldwide trend, i.e., drift in the long-term average. We need to do that in way that minimises the risk of losing data, since the number of weather stations is not fixed over time. If you have your own personal methodology that you show to be highly sensitive to changes in the number of stations, you have only established that your method is flawed, not that someone else’s method is flawed.
Before you run an experiment, you need to set up your hypothesis, and determine what will support it and what will knock it down. You can’t just do an arbitrary calculation then claim it means something. First, using absolute temperatures rather than anomalies means removing or adding weather stations will have an effect that is unrelated to the trend. Here’s an experiment for you to try. Remove every Arctic weather station after a given year, and the average will shoot up after that. Secondly, the derivative method you use will delete any inter-annual trend. Why would you want to do that? It makes no sense whatsoever.
I’m not sure how the fact that your different data manipulations produce a particular result have any relationship to anyone else’s graph. The only thing you can say with any certainty is that coincidences are possible if you try enough random data manipulations. You do one set of manipulations that is in no way similar to those of the other bunch and get a vaguely similar graph. You change the data, and get a different graph. Totally meaningless unless you have a solid justification for your approach.
Try this. Use the same methodology as the original approach, i.e., use temperature anomalies rather than absolute temperatures, and don’t do any strange manipulations like integration and derivatives that produce artefacts not present in the original data (do you have any idea what effect these manipulations have on the statistics of the data set? Are you introducing autocorrelation?). Calculating anomalies is not that hard. Choose the baseline period (any range of years long enough to smooth out differences, 30 years should be enough: NASA uses 1951-1980, CRU uses 1961-90). Average temperatures at each station over that period, then subtract each station’s average from the actual temperature reading. Graph the result. Others on this site will let you know if I have this wrong.
Once you have this straight you can do a proper analysis of the effect of this missing stations. I suggest you read the academic literature on the urban heat island effect before you do this. Google Scholar is your friend.
NASA GISS has a good discussion of why anomalies are used. Read it.
Jonathan, pasting that cite (it’s from 11 years ago) into Google Scholar turns up papers published since then that mention it. Try some of these:
http://scholar.google.com/scholar?hl=en&q=D.Dahl-Jensen+et+al%2C+SCIENCE+282%2C+1998%2C+p.+268-271&lr=lang_en&as_sdt=2001&as_ylo=&as_vis=1
Here is a link to some raw data – US Station Daily Data: Access by State http://www.esrl.noaa.gov/psd/data/usstations/. “the user can examine the temperature and precipitation station data for continental US. All stations available have data from 1950 until 1999 for daily maximum and minimum temperature and for precipitation”. Gavin, mabye you could add a subdirectory for raw data.
Thank you!
Jonathan, since the Arctic melt allows low pressures with thick cloud cover to go in a more northerly route, the Arctic high pressure area is located more commonly over Greenland, allowing it to cool more at nights, or so I’ve understood this. To get a more detailed answer I should try to look to some model results, but I think this is the trick to get at least a partial explanation to the observed discrepancy in trends between Greenland temperatures and the rest of the Arctic. It is colder in winter/nights in there that it has been before.
@Philip Machanick
I apologise for not communicating more clearly. I didn’t complete High School, so my feeble attempt to draw a distinction between an unproven (but workable) assumption, and a Scientific “prediction” or “forecast” made by people who cheerfully admit that nobody understands the complete workings of the Earth’s climatic processes sufficiently to accurately explain the past, let alone the future, was obviously doomed to failure.
I am not sure why I should never leave my home again. Unless you can be a bit more specific, I will carry on life as usual. So far, so good – I’m still alive.
@Martin.
No, sorry. I didn’t write Eifelheim.
@Silk
I am unaware of any scientists who don’t accept that the Earth has experienced glacial periods, followed by (naturally enough) interglacial periods. I assume that the ice melts because the surrounding atmosphere gets warmer.
Conversely, areas of permafrost that were once soil capable of growing grass and similar stuff, (I don’t know what it would be called, but mammoths got quite large eating it), got a lot colder quite suddenly.
I also believe the research that shows that prior to the late Ordivician glacial period, atmospheric CO2 probably reached 4000 ppm, or thereabouts. The exact quantum may be irrelevant.
My query has always been “What mechanism explains the glacial/interglacial temperature differences?”
The Earth may be warming, may be cooling, may be about to lurch one way or the other next week/month/year.
Could you please me know what will happen if CO2 reaches 1000 ppm within the next two years?
Or maybe 7000 ppm. I believe that the atmosphere has reached this level, but I can’t vouch for it from personal experience. I await your guidance.
@Jonathan
Good for you. Parts of the Earth warm up, parts of the Earth cool down. Nobody really seems to know why. They’ll still predict the future for you, for a fistful of dollars! Sorry, couldn’t help myself. Must be my lack of education showing through.
@Gavin
I really WOULD like to know how the atmosphere and climate work. I worked for the OZ Met Bureau a long time ago. Had an interesting experience when the crew of a rather high flying Vulcan confirmed my radar observations of “cloud” tops. Unfortunately, the scientific assessment of the qualified meteorologists was that my observation was theoretically impossible according to published papers by experts in the field.
Some time later, I believe a new cloud growth mechanism was adopted, which fitted better with observation. Possibly RAAF pilots carry more weight than humble Observers.
Some experts continued to disbelieve observed data which conflicted with their theories, and discarded it. Maybe it still happens. I really don’t know.
Regards to all. Live long and prosper.