Dennis Bray and Hans von Storch have been making surveys of climate scientists for a number of years with the reasonable aim of seeing what the community thinks (about IPCC, climate change, attribution etc). They have unfortunately not always been as successful as one might like – problems have ranged from deciding who is qualified to respond; questions that were not specific enough or that could be interpreted in very different ways; to losing control of who answered the questionnaire (one time the password and website were broadcast on a mailing list of climate ‘sceptics’). These problems have meant that the results were less useful than they could have been and in fact have occasionally been used to spread disinformation. How these surveys are used obviously plays into how willing scientists are to participate, since if your answers are misinterpreted once, you will be less keen next time. Others have attempted similar surveys, with similar problems.
As people should know, designing truly objective surveys is very tricky. However, if you are after a specific response, it’s easy to craft questions that will favour your initial bias. We discussed an egregious example of that from Steven Milloy a while ago. A bigger problem is not overt bias, but more subtle kinds – such as assuming that respondents have exactly the same background as the questioners and know exactly what you are talking about, or simply using questions that don’t actually tell you want you really want to know. There are guides available to help in crafting such surveys which outline many of the inadvertent pitfalls.
Well, Bray and von Storch have sent out a new survey.
The questions can be seen here (pdf) (but no answers, so you can’t cheat!), and according to Wikipedia, the survey respondents are controlled so that each anonymised invite can only generate one response. Hopefully therefore, the sampling will not be corrupted as in past years (response rates might still be a problem though). However, the reason why we are writing this post is to comment on the usefulness of the questions. Unfortunately, our opinion won’t change anything (since the survey has already gone out), but maybe it will help improve the interpretations, and any subsequent survey.
There are too many questions in this survey to go over each one in detail, and so we’ll just discuss a few specific examples (perhaps the comments can address some of the others). The series of questions Q15 through Q17, typify a key issue – precision. Q15 asks whether the “current state of scientific knowledge is developed well enough to allow for a reasonable assessment of the effects of turbulence, surface albedo, etc..”. But the subtext “well enough for what?” is not specified. Global energy balance? regional weather forecasting? Climate sensitivity? Ocean circulation? Thus any respondent needs to form their own judgment about what the question is referring to. For instance, turbulence is clearly a huge scientific challenge, but how important is it in determining climate sensitivity? or radiative transfer? Not very. But for ocean heat transports, it might very well be key. By aggregating multiple questions in one and not providing enough other questions to determine what the respondent means exactly, the answers to these questions will be worth little.
The notion of ‘temperature observations’ used in Q16 and Q17 is similarly undefined. Do they mean the global average temperature change over the 20th Century, or the climatology of temperature at a regional or local scale? Or it’s variability? You might think the first is most relevant, but the question is also asked about ‘precipitation observations’ for which a century-scale global trend simply doesn’t exist. Therefore it must be one of the other options. But which one? Asking about what the ability of models is for modelling the next 10 years is similarly undefined, and in fact unanswerable (since we don’t know how well they will do). Implicit is an assumption that models are producing predictions (which they aren’t – though at least that is vaguely addressed in questions 45 and 46). What ‘extreme events’ are being referred to in the last part? Tornadoes? (skill level zero), heat waves (higher), drought (lower), Atlantic hurricanes (uncertain). By being imprecise the likely conclusion that respondents feel that global climate models lack the ability to model extreme events is again meaningless.
Q52 is a classic example of a leading question. “Some scientists present extreme accounts of catastrophic impacts related to climate change in a popular format with the claim that it is their task to alert the public. How much do you agree with this practice?” There is obviously only one sensible answer (not at all). However, the question neither defines what the questioners mean by ‘extreme’ or ‘catastrophic’, or who those ‘scientists’ might be or where they have justified such practices. The conclusion will be that the survey shows that most scientists do not approve of presenting extreme accounts of catastrophic impacts in popular formats with the aim of alerting the public. Surprise! A much more nuanced question could have been asked if actual examples were used. That would have likely found that what is considered ‘extreme’ varies widely and that there is plenty of support for public discussions of potential catastrophes (rapid sea level rise for instance) and the associated uncertainties. The implication of this question will be that no popular summaries can do justice to the uncertainties inherent in the science of abrupt change. Yet this is not likely to have been the answer had that question been directly addressed. Instead, a much more nuanced (and interesting) picture would have emerged.
Two questions of some relevance to us are Q61 and Q62, which ask whether making discussions of climate science open to potentially everyone through the use of “blogs on the w.w.w.” is a good or bad idea, and whether the level of discussion on these blogs is any good. These questions are unfortunately very poorly posed. Who thinks that anyone has any control over what gets discussed on blogs in general? The issue is not whether that discussion should take place (it surely will), it is whether scientists should participate or not. If all blogs are considered, then obviously the quality on average is abysmal (sorry blogosphere!). If the goal of the question was to be able to say that the level of discussion on specific blogs is good or not, then specific questions should have been asked (for instance a list of prominent blogs could have been rated). As it is, the conclusion will be that discussion of climate science on blogs on the w.w.w. is a good idea but the discussion is thought to be poor. But that is hardly news.
One set of questions (Q68+Q69) obviously come from a social rather than a climate scientist: Q68 asks whether science has as its main activity to falsify or verify existing hypothesis or something else; and Q69 whether the role of science tends towards the deligitimization or the legitimization of existing ‘facts’ or something else. What is one to make of them? There are shades of Karl Popper and social constructivism in there, but we’d be very surprised if any working scientist answered anything other than ‘other’. Science and scientists generally want to find out things that people didn’t know before – which mostly means choosing between hypotheses and both examining old ‘facts’ as well as creating new ones. Even the idea that one fact is more legitimate than another is odd. If a ‘fact’ isn’t legitimate, then why is it a fact at all? Presumably this is all made clear in some science studies text book (though nothing comes up in google), but our guess is that most working scientists will have no idea what is really behind this. You would probably want to have a whole survey just devoted to how scientists think about what they do to get anything useful from this.
To summarise, we aren’t in principle opposed to asking scientists what they think, but given the track history of problems with these kinds of surveys (and their remaining flaws), we do suggest that they be done better in future. In particular, we strongly recommend that in setting up future surveys, the questions should be openly and widely discussed – on a wiki or a blog – before the surveys are sent out. There are a huge number of sensible people out there whose expertise could help in crafting the questions to improve both their precision and usefulness.
Re: Walt (#9, #29) –
Questions 11-17 have too many problems to be useful. RC has already discussed 15-17. In 11-14, what does “adequate” mean? Adequate for what? Detecting an anthropogenic influence? Determining that swift action is needed? Quantifying the future precipitation changes throughout the globe?
Consider Q12 in particular. Which models? Some atmospheric models only parameterize convection, while others have sufficient resolution to model it explicitly. Should I consider both? How do I weight the two types in my answer? Does the word “can” mean that I should consider only the best-performing models or the theoretical potential of models, or should I ignore the word entirely and just consider current models?
Hopeless.
Well-done! Thanks for letting us know what’s perking the climate science world, and also for debunking this ‘survey’. The best surveys are short, and come up with the answers to the questions you want. This 76 question monstrosity seems designed to tire the recipient out, with its ill-thought and leading questions. Why would any respectable climate scientist devote her or his time to such an endeavour? I’m with you – nonresponse rate will dominate the results.
An ideal poll is short, to the point, and asks an interesting research question. Something like:
1. In your opinion, is the observed warming of the Earth in the past 100 years due mostly to human emissions, or not?
2. In your opinion, can state-of-the-art climate models predict long-term climate response to doubling of CO2 to within 1.5 degrees Celsius?
3. In your opinion, what would be a good threshold below which to limit CO2 emissions to avoid very serious damage?
That’s what the public wants to know about… why not ask that?
More on the undersea permafrost question, in the form of a release from last year:
http://www.sciencedaily.com/releases/2007/02/070207091024.htm
Obviously, there is some confusion on the topic (including on my part–mea culpa), however the definition of permafrost does not include a chemical component–it comprises, for instance, both water ice and rock, as well as other components of soils, either terrestrial or subsea.
I’m getting into OCD territory here, but OK, one more link:
http://nsidc.org/data/ggd318.html
The survey of 34,000 `scientists’ keeps cropping up. Yesterday on Chicago tonight, an old Chicago area TV weatherman, John Coleman, was interviewed on Chicago Tonight on the local public TV station. He denied there was any warming at all or that it would be a significant problem, referring to those so-called scientists who don’t believe in global warming. He also made the usual claim that the IPCC was motivated by ideology.
Unfortunately, the interviewer, although he tried to challenge Coleman on a couple of matters was entirely inept. I posted something on the WTTW website in response, mentioning for example, the fact that the National Academy of Sciences and other such organizations agreed with the IPCC. There is a good chance the station will read my comment and other similar comments on the air. But I haven’t been able to locate any information about those 34,000, whatever they are. If I remember correctly, they were mostly not even scientists, and, except for a small minority, didn’t have any qualifications in the relevant disciplines.
It would help if ReadlClimate has an easy to find reference to which people could refer when they need to make a quick response.
P.S. In my comment to the station, I suggested they interview Ray in response, so I hope he will forgive me for bringing up his name.
I am having an awful time trying to get past CAPTCHA. This is my third try.
Figen, is this a useful search to get information on these processes? I know that pingos used to be considered as ice heaves but I’ve seen comments that pingos now above the ocean may have been formed when the ocean was higher, and new ones are forming below sea level. I’m not sure how new this idea is.
http://www.google.com/search?q=methane+pingo
http://images.google.com/images?q=methane pingo
http://scholar.google.com/scholar?q=methane+pingo
Wow hank, that’s some awesome info. i didn’t know much about pingos. Thanks!
re 57
Pingos at Nioghalvfjerdsfjorden, eastern North Greenland by Bennike puts me in mind that “Pingo” may not be a geologically precise word. The shape may be more a function of local weather than the formative mechanism
re Marcus 35
Just to add on the: Arctic Ocean seabed methane clathrate release
“Since 1994, he (Dr Semiletov) has led about 10 expeditions in the Laptev Sea but during the 1990s he did not detect any elevated levels of methane.
However, since 2003 he reported a rising number of methane “hotspots”, which have now been confirmed using more sensitive instruments on board the Jacob Smirnitskyi.”
Marcus, I think you are correct to state that any conjecture is premature until peer reviewed, juried, refereed journal articles appear, and I appologise if I seemed to overstate its possible significance.
What I am trying to suggest is that its possible intensity change is perhaps a “new event” and bears more scientific investigation and attention just in case.
http://www.heatisonline.org/contentserver/objecthandlers/index.cfm?id=7115&method=full
Hi Richard Sycamore,
I’m suggesting that scientific knowledge is not increased by taking a poll of existing knowledge. Further, I’m suggesting that a poll where the respondents volunteer to respond is a notoriously poor way of getting an accurate assessment of the population’s real attitudes and beliefs. I can tell you I spent a little time in research (Psychology) where it was generally accepted as a given that you could resort to this method if you had to, but the results would be treated with skepticism. (Take a look at recent political polls, conducted by expert pollsters; who is winning seems to vary depending on who is taking the poll.) So, given that no new knowledge is gained by a poll and the result of any poll should be treated with skepticism, you are absolutely correct, I am suggesting that they are wasting their time and ours.
When you ask people if they would like to contribute their opinion (Bray and von Storch), you get different results from when you sample opinions already stated (IPCC). If you have the opportunity to examine how the person formed that opinion (IPCC) versus when you don’t (Bray and von Storch), it’s even better. The risk of bias is reduced when you take larger sample sizes, and here I am talking about the sample size of the authors. Bray and von Storch are a sample size of two authors; the IPCC report has 37 authors.
Are you suggesting that these 37 authors did not sample widely enough? I’ve looked over the references.
http://www.ipcc.ch/pdf/technical-papers/ccw/references.pdf
They look pretty wide too me. If you made a tree of the references cited in those references, it would grow very large, very quickly. It starts with around 900 direct citations. The number of research hours that this report represents is astronomical.
When I look over how the polls have been used, I do not get the impression that they are used to examine the variance between opinions. It appears more like they are used to argue that the consensus among climate scientists is not what some (most) climate scientists are telling us, when they say that it is real, anthropogenic, and dangerous. I am not saying that is the intent of Bray and von Storch; they are just as subject to misinterpretation as everyone else.
I assume that Bray and von Storch have read the IPCC report, but I can offer no opinion concerning their thoughts on it. I can only give you my reasons for trusting it over the results of any questionnaire survey.
Others,
I’m sorry, but science is not served by a democratic vote of the general population, and in this case the population is best served by accurate science. If you ask the average person really simple questions about Newtonian physics, you will get a lot of answers that are just plain wrong. As the questions get more difficult, the answers will get less and less connected to reality, and the correct answer in this subject depends on an understanding that stretches our best minds. You might as well have asked the general population in the early 40s about whether an atomic bomb were possible or whether setting one off would ignite the atmospheric nitrogen. The climate is like a vast multi-body problem were the movement of one body affects the movement of others, which is hard enough, but unlike gravity, we are are struggling to understand how much movement of one affects movement of the others. Most people have no idea how little they know about it. That’s not a condition that leads to good decisions about what, if anything, to do about any problem.
Most engineer types can appreciate the idea that a change in the thermodynamic balance will result in a new equilibrium state. This can easily be put in terms the general population can understand as well. The fact that change is and will happen is not the question, the question is how quickly and how much will it change. Those answers depend on how everything interacts. The fact that the general population has a deeply vested interest in the answers does mean they are best served by letting them make an uninformed decision. Conducting surveys of general beliefs will not add to the accuracy of those beliefs.
I know that sounds elitist; I will admit to being concerned about letting the average person make this decision for me and my children. I would much rather do without some luxuries than risk some of the extreme scenarios that could play out, but I’m skeptical that the average person is as willing to make sacrifices as I am and I really don’t think they understand the risk.
Rant/muse time.
At the risk of using a really bad example, I’ve run a lot of simulations (computer games) involving competing societies and limited, common resources. Most often the society that dominates the others is the one that figures out how to collect and consume the resources faster than the others. It seems to me that humans are doing this in real life. Americans have been particularly good at it.
If you couple peoples’ competitive nature with the desire for fairness observable in primates (We want at least as much as the other guy is getting.), you get a situation where Americans will not give up the advantages we have and the more populous, but poorer, societies will not give up their pursuit of getting the same kinds of things that we have. The implications of where this will take the climate, without a major shift in where the global economy gets energy, are not good. My belief is that the world is heading toward a train wreck; the best we can do is to try to slow it down so that when it hits, it won’t be as bad.
Re. Leonard Evens, #55, see here.
Aaron writes
” … may not be geologically precise …”
Look it up. You’ll find a variety of structures talked about. Don’t get stuck on “precise” — look at reality.
http://adsabs.harvard.edu/abs/2004AGUFM.C23A0989M
I’d like to suggest an alternative way of going about a survey of opinions that would get around all of the problems listed.
1. Set up a survey website that allows scientists to join as members.
2. The members would be able to enter details into the site database about their experience and their fields of expertise.
3. The members would be able to enter a list of their papers, along with details of each, including optionally the abstract.
4. All members would be able to rate each of the papers of other members on a scale of importance/soundness.
5. To assist busy scientists who have lots of papers, they would be able to enable other members, including lay members, to enter papers for them. (With an approval flag feature implemented to show when the member has approved a paper that has been entered for them.) This would be a task that students for example might be interested in helping with.
6. To prevent abuse, site administrators would need to be able intervene to bar bogus members, or to confirm their credentials. Both members and lay visitors would be able to flag potentially bogus members and papers (but not be able to do so for confirmed members).
7. The site administrators would post survey questions periodically, where-upon members would be notified and asked to participate. Answering questions would not be compulsory and there would be no time limit on answering them. Confirmed members would be able to submit questions to the administrators.
8. When answering questions a certainty rating would need to be given with the answer.
9. Members would be able to rate each question on various quality criteria (eg. ‘ambiguous’, ‘confusing’ etc).
10. Members would be able to return to a question and re-answer it. Each answer would be stored with a time stamp, so that it would be possible to track changing sentiments within the community.
11. Survey summaries would be able to be queried in a manner that presented the results in relation to member creditability as demonstrated by their publication record and the rating of those papers by other members. Answers from members who have been flagged as bogus, or who have been disbarred, would be filtered from the result sets.
Does anyone out there have the expertise to do this? I’ve been contemplating it since I suggested it here about six months ago, and have been busily learning how to create such applications using PHP and ExpressEngine, but I reckon I’m still at least six months away from being able to do it.
O.T. I like this, finally some specifics.
http://wonkroom.thinkprogress.org/2008/10/02/climate-principles-letter/
I apologize for spamming this. I’ve been working on a “Layman’s Guide” to global warming intended to summarize the science and controversy. It covers the broad topics (including mitigation) with a focus on answering skeptical claims along the way. It’s presented as a narrative, where each section builds on the last, and not as a point by point rebuttal like Skeptical Science or Coby Beck’s guide. Most significantly, it is presented both as html and as a flash-based slideshow/presentation with narration.
The HTML version is basically done, aside from a few minor revisions that I intend to make as I create the slideshow. I learned the hard way that it is extremely difficult to record a halfway decent narration, so I’m looking to correct any errors or lapses in judgment first. (At this point I’m not looking to make stylistic changes, or re-organize it). I have no actual expertise in any of this so it is a true “layman’s” guide, but there are some things I understand better than others. Fixing problems with the narration after the fact is problematic.
It is 14 sections, plus introduction. Each section will be about 18 minutes long.
The site is here:
http://cce.890m.com
The sections are:
0. Introduction (my reasons for doing this)
1. Primer and History
2. The Scientific Consensus
3. A New Ice Age (case study of the claim that scientists were predicting a new ice age)
4. The Temperature Record
5. Temperature Reconstructions (Mostly Hockey Stick stuff. Andrew Dessler did a post about my summary on Grist some months back)
6. Solar & Cosmic Rays
7. Attributing Mankind
8. Climate Models
9. Hansen’s ’88 Scenarios (case study)
10. A Changing World (summarizes environmental changes to present)
11. “Who Cares?!” (why this is a bad thing)
12. “Why Now?” (why we’ve run out of time)
13. Facing the problem (high level mitigation issues)
14. Technologies and Strategies (specific solutions)
I’ve posted the slideshow version of the introduction to give an idea of what the final format will look like. I also plan to upload versions for Google Video.
Thanks!
CCE,
We need more good efforts like this, but there is an inherent psychological problem which severely limits their usefulness.
Adults who have not made up their minds on this issue by now, are not likely to have the interest required to make the effort to sort out all the technical points in the controversy.
People who have made up their minds that AGW is a scam, are unlikely to accept any arguments to the contrary, because most of these people firmly believe AGW is some kind of political plot, or based on economic motives. Reading the web sites, I find the deniers will believe any quasi scientific narrative that supports their idea no matter how stupid and incorrect it may seem to someone with any real scientific background. The lack of qualifications of the people who make these bogus scientific statements does not matter. It has been shown that sometimes the more debunking you do, the more the incorrect idea becomes set in the mind.
You can see this demonstrated on more blogs and discussion groups than you would have time to visit.
I haven’t had a chance to read it yet, but this survey looks like it’s full of good examples on how not to write survey questions I can use in research methods class. Examples of all the various mistakes.
Another flash — if a tree falls in a forest & there’s no one to hear it, does it make a sound?
If scientists are unaware of climate change (or some of its awful effects), does that mean it doesn’t exit?
Here’s something we should be made aware of: PEAK FOOD. Due to using finite resources to TEMPORARILY boost food production, and climate change effects, etc. — see: http://peakfood.co.uk
67 Eric, you’re right. Nothing will change until terror sets in. Unfortunately, the pretend panacea of geo-engineering will prevent that. Yep, temps can be stabilized no matter the CO2 level, but the oceans will still die and the poles will still melt. (melt is determined by equator/polar temperature difference, NOT average temps) Modern folks have been conditioned to doing whatever is fun and expecting a Magic Pill, then another Magic Pill that will counteract the harm caused by the first Magic Pill. Well, ultimately, we run out of Magic and the patient dies a miserable death. Permanent Red Tide, anyone?
The article focuses on the questions and not who was asked to participate. Why were there questions about activism and advocacy? The issue is about what the scientists think. I could get any answer I want if I could pick the respondents, e.g. a gun control survey sent to Wyoming republicans.
Cheers,
A scientist with real credentials and peer reviewed publications.
68 lynn: I read the peak food articles. They do state that part of the problem is that Asian, and other countries, who for centuries have eaten plant food diets, are now eating more like us in the West. i.e., a lot of meat, poultry, fish, eggs, and dairy. That takes a lot of land to feed one person, compared to that person eating a vegan diet. About as much as 15 to 1. So we can buy some time by encouraging as many people as possible to eat as near vegan as possible. A very important side benefit would be a much healthier population, with lower health care costs.
Sorry, this is really OT, but I’ve gotta rant to someone about an interesting lingo thing I caught in Gov. Palin’s stock line about AGW. You guys (RealClimate) seem interested in how scientists’ cautious, precise use of language puts them (you, us) at a disadvanage in the “marketplace of ideas”. Gov Palin has provided a great counter-example, as someone not constrained by a preference for clarity.
First, in the Couric interview:
Couric: Is it [Global warming] man-made, though in your view?
Palin: “You know there are – there are man’s activities that can be contributed to the issues that we’re dealing with now, these impacts. I’m not going to solely blame all of man’s activities on changes in climate. ”
2nd, during the VP “debate”:
“I’m not one to attribute every … activity of man to the changes in the climate. ”
Three times, in her most public moments, she has garbled the causal directionality of the same phrase in almost exactly the same way. Three times, she – sorta – declines to blame climate for man’s actions, when asked whether man’s actions are to blame for climate change.
What are the chances of this occuring randomly? We know she’s no scientist, but is her concept of causality really that fuzzy? Or has she learned – or been taught – to answer that question like that, to avoid answering the question at all? This wouldn’t be the only question she hasn’t answered, of course, but has she avoided answering any other question in such a consistent way?
Color me paranoid, but I think it’s intentional obfuscation. My pet theory is that Exxon (indirectly) paid some clever unscrupulous gnome to write her answer.
The kicker is her answer when Gibson asked her if she didn’t believe in AGW. It was something like (paraphrasing, I don’t have the transcript) “… I never said that. You can’t find where I ever said that…”. Which is true, because I can’t find where she ever said anything coherent about this at all!
Its also a great way to keep the issue out of the limelight. Most text-based media wouldn’t use a quote which didn’t make sense, because it can’t be woven into the context.
And who is most interested in keeping the issue hidden and muddled? Hmmm, let me think a minute here, with my brain…
-elkern
Forgot to mention, just in case those reading here didn’t already know it, that animal agriculture produces 18% of the world’s greenhouse gas emissions.
Some brilliant wanted HSS to switch from cow meat to kangaroo meat because the latter’s digestive system barely produces methane. The fences around the pastures need to be a tad higher then :D
As for Sarah Palin, I’m in awe with her. As a none native English speaker she almost seems to say that man’s actions are a result of climate changes. Yes, man are planting palm trees in gardens way more northerly than before ;)
recaptcha: Believe / Expense
Jack, can you cite a source for your statement?
> animal agriculture produces 18% of the world’s
> greenhouse gas emissions.
Where did you get that?
I found this:
http://dx.doi.org/10.1016/j.ics.2006.01.009
And I found the number “18 per cent” here:
ftp://ftp.fao.org/docrep/fao/010/a1445e/a1445e.zip#page=47
but that number needs to be undetstood, and comes with significant cautions
“… sector emissions in particular are extremely difficult to
quantify and the values reported to the (United Nations Framework Convention on Climate Change) UNFCCC for this sector are known to be of low reliability. This sector is therefore often omitted in emissions reporting, although its share is thought to be important.”
elkern (72) says, “Color me paranoid, but I think …. Exxon (indirectly) paid some clever unscrupulous gnome to write her answer.”
O.K. Consider yourself colored.
RE Question #9, “How much do you think the direction of research in climate change sciences has been influenced by external politics in the past 10 years?”
I’m thinking that so much energy has been tied up in refuting contrarians. It’s one thing for science to advance from legitimate, logical, (some) evidence based skepticism — that spurs scientists to strive for greater heights of knowledge. But contrarian attacks that are based more on ideology and less on logic or evidence — or a general sense & knowledge of how the physical world works — I think, is counterproductive. I’m thinking part of the purpose of their critiques is to derail the science — get scientists involved in somewhat side issues.
Another thing that has happened is that the debate, which should be between false-positive-avoiding scientists and false-negative-avoiding laypersons and policy-makers (who would strive to avert harms, requiring much lower confidence than scientists that there’s a dangerous problem, before acting to mitigate it), has been shifted so it is between the false-positive-avoiding scientists and the false-positive-avoiding contrarians (who have much higher standards than scientists, and would not accept AGW is a problem unless they were much more than 95% confident, some seemingly to demand 101% confidence — requiring some bridge to never-neverland).
The contrarians have managed to convince a lot of the public and politicians that the science just isn’t in yet. That’s got to be demoralizing to hard-working scientists.
RE #72 and Elkern’s comments, here’s an insight into the bogus science behind Palin’s claims:
http://www.guardian.co.uk/world/2008/oct/01/sarahpalin.climatechange
Do the names ExxonMobil, the American Petroleum Institute, the George C Marshall Institute, the Heartland Institute, Willie Soon, Sallie Baliunas, and Timothy Ball ring any bells? These and other like-minded persons are the orgs and “scientists” who inform Palin’s knowledge about global warming, according to the article.
Too bad this type of news is only available outside America, and the American media are not picking up on it. We live in the Dark Ages here.
This is all very interesting but isn’t it about time we had a serious in-depth discussion about the GCMs?
#75 Hank:
The 18% came from the FAO’s report: “Livestock’s Long Shadow”. Note that that is for worldwide emissions. I calculated the percent for the USA. It works out to about 8%, due to our using much more fossil fuel than most countries. So for some countries, their percentage is higher than 18%.
Here’s a link to the FAO’s report: http://www.fao.org/docrep/010/a0701e/a0701e00.htm
The main offenders are ruminants, because of their methane emissions. So if you want to eat flesh, you’d be best off with poultry. I’d include fish, but much energy is expended bringing them to your plate.
While I’m at it, you might want to read “The China Study”, by T. Colin Campbell, on the health effects of eating animal foods. There are many other books on the subject, but I think that one is the best.
I really like your idea above of discussing questions publicly as part of the survey process. This would help reduce leading question problems as well as help design the survey instrument to provide the community with the most helpful kinds of feedback.
#75 Hank:
I’ll be honest. I never read that entire FAO report myself. I took others’ word for the 18%.
This afternoon, I searched through the report, http://www.fao.org/docrep/010/a0701e/a0701e00.htm, and found the summary in section 3.4, Pg. 112, Part 4. They do list the total at 18% of all AGW emissions, worldwide.
I’ve also read that that exceeds the emissions of the world’s entire transport sector. Guess I’ll now have to verify that.
#61 Chris G:
If you mean actual computer games, such as civilization, age of empires, etc, almost all of those are deliberately designed to favor the rapid acquisition of resources. So you’ve observed these games work as designed. They don’t indicate anything about the real world.
I don’t disagree with your claim that societies that gather the most resources tend to dominate others – that is, after all, largely what the historical record appears to show (with some important exceptions). But experience with computer games does not support your claim.
I can say that, whatever the other problems with this survey it has not yet been FREEPED by any of the climate sceptic groups I am aware of, including the folks on the climate sceptics mailing list (who got hold of it last time).
Jack, the document I linked to above (the FTP link) is, I think, using the same numbers you found. Here’s a cite to it. FAO is documenting their longterm serious attempt to work out answers to these questions that are useful for comparing different food sources including the costs to the climate. It’s an amazing amount of work.
Nice to be reminded someone, somewhere, makes this kind of effort.
ISSN 1813-3940
FAO/WFT Expert Workshop
24–28 April 2006
Vancouver, Canada
Comparative assessment of the environmental costs of aquaculture and other food production sectors
Methods for meaningful comparisons
RE #8 (Raven)
“Similarily, an aerospace engineer with experience in numerical modelling would be qualified on comment on whether the climate models are likely to provide useful information”
(Slightly off topic because the article was about the questions not the questioned)
I have strong doubts. An aeroplane is different from a climate system. The opinions of such people might be useful especially if they asked for plenty of time , first, in which to learn about climate models. The trouble is that people who over-estimate their own knowledge, are just the ones whose opinions are of the least value. I keep on reading comments from people who claim expertise in a general subject they call computer modelling. They probably belong to the latter category.
Returning to the topic of questions; how about another kind? i.e. questions designed to elucidate whether the person concerned has any knowledge of the subject? These should be taken under examination conditions. Why should we respect the opinions of any old scientist if (s)he is incapable of demonstrating that (s)he has read any of the leading papers?
A suggestion. Choose one well defined question such as the range of estimates and confidence for climate sensitivity i.e roughly the same as done by Morgan and Keith (mentioned above) but with a larger sample of experts. The survey being discussed is much too vague and dodgy.
Oddly enough, Bray and von Storch have responded to this post here but curiously did not see fit to let us know.
One minor typo in their post is worth pointing out. In quoting the post above they replace the word ‘corrupted’ (in reference to previous sampling problems, paragraph 4) with the word ‘corrupt’. This might seem minor, but it is the difference between an accidental problem that makes data untrustworthy and implying the researchers themselves are untrustworthy. This is certainly neither expressed nor was intended.
As I said in the post, I think that there could well be a great deal of community input that would be useful in improving these kinds of surveys, but avoiding talking to us – even to notify us of a response – does not bode well for that interaction. Shame.
Gavin-
Why not discuss the substance of their post rather than your indignation?
It is indeed ironic that you complain about not being notified when your original post was put up without notifying Bray or von Storch. You also posted their survey without asking.
Rather than the indignation, how about some substance from you?
[Response: Who’s indignant? I just clarified a possible misinterpretation. And as for complaining, I merely made an observation. The survey questionnaire was not described as confidential, and had B+vS asked me to take it down once I posted it, I would have. Instead, your post is the only communication they have made (albeit an indirect one and one I only found by accident), two weeks after the initial posting. I am really not that difficult to contact, even snail mail would have got to me faster. I may comment on their comment if I get some time later on – my comment above was merely a notification to the readers here that a response existed. I am not however blessed with instant judgment on posts I have yet to digest. – gavin]
[Further response: You should probably also correct the other misquotations. I did not say “These problems have meant that the results were less than useful.”, I said “These problems have meant that the results were less useful than they could have been…”. Neither did I say there were ‘too many questions’ (though others did in the comments), I said “There were too many questions … to go over each one in detail”. Both of these errors distort the meaning of what was said. And since the bulk of the my post was on the ambiguity of imprecise language, this is mildly ironic. – gavin]
Was it their typo, or Pielke mistyping?
The typo immediately following must be his, he writes:
> Real Climate had posted their survey on their site
> without first asking.
RC had not posted their survey. Posted _about_ it.
The previous survey was corrupted when people not intended as recipients were able to answer it.
Gavin-
FYI, the post appeared at Prometheus but was authored by Bray and von Storch. Telling me to correct their work is probably not too useful ;-)
[Response: Well, you appointed yourself go-between. But they can read this in any case, and so I don’t really care who fixes it. – gavin]
I am sure that they would be responsive to your complaints should you wish to engage their post. I have copied your comments over at Prometheus and notified Bray and von Storch of them by email. So now everyone is aware of everyone’s comments and we can get past who did or didn’t email whom about what.
FYI, here is the link to their piece:
http://sciencepolicy.colorado.edu/prometheus/a-response-to-realclimate-concerning-a-new-survey-of-climate-scientists-4635
Too often blog discussions devolve into accusations of “misrepresentation” and who said what really meaning something else, and whose feelings were hurt because their words were “distorted” (Shame!).
How about just engaging them on their survey? All surveys are imperfect and can be improved, but does their survey tell us anything at all about the views of climate scientists? What might be done specifically to improve it for the next iteration? Presumably these are the sorts of questions that would make for an interesting conversation.
[Response: What do you think the original post was about? – gavin]
The survey does appear on the website and as I read things Pielke did not write any of the post.
Clarify — the first survey had its response HTML link spread around and people filled it out who weren’t invited to as a result, corrupting the results. The second one — how are people’s responses to that being checked? Are they limiting responses to those they know are part of the population they intend to survey?
I think that’s the difference.
Gavin #89-
Thanks for asking . . . . I absolutely think that the first post was a preemptive hit job on bray and von Storch, designed to discredit their work in very public manner — and judging by the discussion that ensued on your site, it was very effective to that end.
Now, maybe I am very wrong in my interpretation of that first post and what you really wanted to do was start a productive, high-level conversation with Bray and von Storch about survey methods etc. If so, you are in luck as you now have their full attention as they have responded at some length to your criticisms with a post of their own.
So the question is, are you interested in spin, misdirection, and discrediting your peers? Or maybe instead you are interested in a substantive public discussion among experts on surveys of climate scientists? I have my views as to the answer, but feel free to prove me wrong.
[Response: Huh…… Well, I’m actually stunned that you wrote this. Until this comment, readers could have assumed that you were just helping improve the lines of communication, facilitating discussion and furthering goodwill amongst men and women. However, you appear to much more interested in stirring the pot, insulting people and generally being nasty and unpleasantly petty. You can therefore consider our interaction terminated. If Bray or von Storch want to communicate with me and readers here, they can do it directly, and frankly, it’s likely to go much more smoothly. – gavin]
As is all too typical, whenever Roger comes by to play games, he leaves mis-communication and rancour behind. Since he is someone who is frequently misquoted, you might think that he’d be sensitive to people incorrectly cutting-and-pasting statements… but let’s put that aside.
While, I have little interest in continuing to engage with him on this or any other topic, I do have an interest in improving the level of communications of scientists with the public and that includes the communication of what scientists think. Unlike Roger’s paranoid fantasies about what this post was about, I really do think that these surveys could be done much better – and the idea proposed above of having an open wiki where the questions could be hashed out ahead of time is still I think a good one. Bray and von Storch did not respond to that, nor to the issues of precision in the questions. These are important issues to get right.
The only one of the original points they did address was with regard to what I described as a leading question. I have to say, I am surprised at what the responses were. While the result was clearly skewed towards disapproval of the use catastrophic scenarios to alert the public, more people than I expected saw this favorably. Why this is so and what it means unfortunately can’t be easily discerned. A more specific question involving a real case would have been a much better thing to ask about.
As I said originally, this critique is too late for this survey, but I sincerely hope that the issues can be taken on board for any future attempts. There is a great deal of interest in what the community of climate scientists think outside of what we write in assessment reports and papers, and it will only take a few more poorly designed surveys before they’ll cease to respond to anything – however well crafted. Thus I urge Dennis Bray (who I think wrote the response) to engage more directly, and maybe ask himself (and the readers here) why the responses were as they were? There may well be lessons to learn there too.
I once set up a polling group and engaged experts to write the questions, a bit like engaging professional ststisticians to help with climate science (Wegman).
Take a question from Bray and von Storch to examine methodology.
“24. With how much certainty can we attribute recent climate related
disasters to climate change?”
This question is logically meaningless because it requires the respondent to agree that there have been recent climate related disasters. Some of us think climate is BAU, so any answer we give is, by definition, wrong. Extremely bad poll methodology to lead respondents to an answer they know has to be wrong. I could give a dozen other method errors in this amateur survey.
Try Sir Humphrey Appleby on compulsory national military service:
Humphrey: You know what happens: nice young lady comes up to you. Obviously you want to create a good impression, you don’t want to look a fool, do you? So she starts asking you some questions: ” Mr. Woolley, are you worried about the number of young people without jobs?”
Bernard: Yes
Humphrey: “Are you worried about the rise in crime among teenagers?”
Bernard: Yes
Humphrey: “Do you think there is a lack of discipline in our Comprehensive schools?”
Bernard: Yes
Humphrey: “Do you think young people welcome some authority and leadership in their lives?”
Bernard: Yes
Humphrey: “Do you think they respond to a challenge?”
Bernard: Yes
Humphrey: “Would you be in favour of reintroducing National Service?”
Bernard: Oh…well, I suppose I might be.
Humphrey: “Yes or no?”
Bernard: Yes
Humphrey: Of course you would, Bernard. After all you told her you can’t say no to that. So they don’t mention the first five questions and they publish the last one.
Bernard: Is that really what they do?
Humphrey: Well, not the reputable ones no, but there aren’t many of those. So alternatively the young lady can get the opposite result.
Bernard: How?
Humphrey: “Mr. Woolley, are you worried about the danger of war?”
Bernard: Yes
Humphrey: “Are you worried about the growth of armaments?”
Bernard: Yes
Humphrey: “Do you think there is a danger in giving young people guns and teaching them how to kill?”
Bernard: Yes
Humphrey: “Do you think it is wrong to force people to take up arms against their will?”
Bernard: Yes
Humphrey: “Would you oppose the reintroduction of National Service?”
Bernard: Yes
Humphrey: There you are, you see Bernard. The perfect balanced sample.
I left the following post over at Roger Pielke’s blog (sorry for length),
………
Dennis Bray and Hans von Storch,
Thank you for the time to post your critique of the RealClimate post. This seems to be turning into another episode of the “ha…I gotcha” game between blogs, with cheerleaders for either party, so I hope to provide a bit of focus. If you are new to the blogosphere, I’d recommend you ignore the cheerleading and simply respond to the primary post, since you’ll drive yourself mad going around various sources and reading each and every comment (and expecting them all to be rational). For instance the claim by Stef that this survey was “debunked” clearly makes no sense, as the RC post was not an attempt to “debunk” anything, nor did you make any positive scientific claim that could possibly be “right” or “wrong”…it was simply a survey, and the RC piece an opinion piece on the quality of questions asked.
For the most part I agree with your responses to the remarks made by RC commenters, but you said very little about what Dr. Schmidt actually wrote. Some of the objections raised by Gavin are probably too trivial, such as what you meant (Q16) concerning the ability of climate models to predict the next 10 years, etc. Gavin’s point is that a) there are many different meanings to “the models ability to reproduce temperature observations” and b) the ability cannot be determined, since it would require us to be around 10 years later to assess what actually happened to what was modeled…point b is kind of silly and nitpicking at words, since it doesn’t seem to be open to any other interpretation, however point a is very reasonable. He does make a good point however– models do projections, not predictions. This means an entirely different thing, since a projection assumes a given scenario (e.g., a rise of CO2 by 2% per year, no asteroid impacts, the sun doesn’t explode, etc) while a prediction is meant to forecast all possible socio-economic and physical conditions, so unlike a projection where you have “if x, then y” a prediction guarantees x, therefore y. We cannot predict volcanic eruptions or asteroid impacts, and scientists who answer this survey may be nitpickers and give an answer to that question which would have been different than had you said “projections.”
Clearly Gavin was incorrect in saying that no scientist would be in favor of “exaggerating” in a public setting, but he also makes good points on what “extreme events” are, etc. There were clearly very poorly phrased questions in the survey (e.g., what are things well enough for in Q15?). In my opinion, the effects of a doubling of CO2 are understood well enough for policy action on mitigation, but not good enough to stop research funding and going on to something else. So I’d have to assume what you mean. The essence of the RC post is this– The less assumptions your respondents need to make, the better. There is no way you can argue with that, or the fact that certain questions in this survey require assumptions. You need to see the forest a bit and not the trees in your response of their critique of your survey– Quibbling over who was not contacted or what he said, she said, etc is cute, but RC wanted to assure that questions are properly phrased, not open to question on meaning, and everyone here is hoping that they will not be misused by special interests (denialists or alarmists). I’d hope we all agree with this, so it’s not an attempt by them to discredit everyone else’s opinion as you say in your last paragraph.
By the way, Gavin is certainly right in that a few of his words were definitely misquoted, and those misquotes lead to a much different context. I hope that you can correct these mistakes (and why a [sic] next to “since?”). I do hope that you guys come over to RC and discuss this directly, so these types of backs and forth don’t get too out of hand. It seems Roger Pielke simply wants to insult Gavin which is non-productive, and certain RC commenters apparently have not understood the RC post, or have not bothered to read the survey. I doubt the aim was to “discredit your work in a public manner” as Roger Pielke thinks, but the public needs to be aware that polling can be a tricky business, and your piece had great intentions but a few of the questions they made examples out of. However, the overall procedure and intent deserves kudos, and I hope we can get reasonable assessments of what the mainstream community thinks. If anything, let’s hope respondents to this survey read the RC post to catch a few of the subtleties.
Chris
I would briefly comment on some of the cyber shenanigans of late concerning the survey of climate scientists.
First, to give Hank Roberts peace of mind: The first survey (1996) was a hard copy mailout in multiple languages. The second survey (2003) was on-line.
Gavin wrote: ‘I really do think that these surveys could be done much better – and the idea proposed above of having an open wiki where the questions could be hashed out ahead of time is still I think a good one. Bray and von Storch did not respond to that, nor to the issues of precision in the questions. These are important issues to get right.’
On the Wiki suggestion …
The idea of the Wiki proposal raises a couple of concerns which seem to predominate many of the comments on the survey. How would you control the respondents, i.e. would the questions be open for discussion by anyone so wishing to participate or would the participants have to be a practitioner of that particular specialty? Would there be community censorship on sensitive questions? (Just as an aside, haven’t some of the more damning criticisms – some of which might be valid – come from outside of the climate science community in the last couple of years?) What if questions were posed that could not be answered, would they simply be ignored? Who would be the intended audience of the responses? Would it be an in-house assessment with no room for critical comment? Might it turn out to be little more than asking the members of the Vatican to evaluate the significance of the Vatican?
[Response: (Thanks for stopping by). I don’t see this as a problem at all. You are free to ignore suggestions that aren’t constructive, but having pre-screening of questions would avoid many of the problems. Each question would have a discussion page where the purpose, clarity and appropriateness of the questions could be discussed prior to any edits for instance. Remember people aren’t discussing the answers, just the questions. – gavin]
So first I would suggest it is necessary to establish the PURPOSE of the survey. Ours was to get scientific OPINIONS concerning broad aspects of the climate sciences. We clearly stated the intention of the survey in the letter if invitation:
1 identifying areas in need of increased research and/or focus
2.suggesting funding priorities for research
3.providing the opportunity for the science community to express an opinion concerning the dissemination of scientific results
It will provide a collective perspective of the climate science community on matters of climate science and climate change from the perspective of climate scientists.
[Response: …but this a little of the problem. The questions were not directly addressed to these issues. No choices for funding priorities were suggested, and the spread of the questions dealt with what is a very limited set of climate science questions. Just compare the money that goes towards remote sensing vs. climate modelling and then the profile of each field in this survey. – gavin]
On precision …
The precision necessary to finely tune questions to each specialization in climate science would further result in a much lengthier questionnaire than any of the original surveys, with a much reduced sample size with the ability to respond to such specific questions, so what would be the representativeness of the survey?
[Response: That is a danger. But the alternative is to have people not really know what they are being asked about. What did you mean for people to think when you said ‘temperature’ for instance? It doesn’t take too much more text to say ‘global mean anomalies since 1900’ if that was indeed what you wanted. – gavin]
Still on the matter of precision …
Gavin also comments concerning the precision of a question asking as to the acceptance of presenting extreme scenarios. He says ‘A more specific question involving a real case would have been a much better thing to ask about.’
A ‘more specific real case’ might be fine for a more specific sample. Unfortunately, the global climate science community does not all read the same newspaper nor watch the same TV station. We simply wanted to know if climate scientists agreed with this practice and the question as posed captures this opinion, in my assessment, without any problem. Or, is it the case, that you feel some cases should be exaggerated and some not?
[Response: But without an example, how is one to judge whether it was exaggerated or not? One presumes that the author of any such piece didn’t think so when they wrote it. However, as I said above, I am surprised by the responses you received. That may well be worth following up in more detail. – gavin]
precision continued …
We had in mind to address a general broad spectrum of climate scientists and to do so the questions needed be be somewhat general.
question content and preemptive conclusions
Back to the general posting, Gavin stated ‘One set of questions (Q68+Q69) obviously come from a social rather than a climate scientist: Q68 asks whether science has as its main activity to falsify or verify existing hypothesis or something else [Gavin, you were very critical of my misquoting you in one of your comments. Please note that the question does NOT read ‘verify existing hypothesis or something else’, it reads ‘verify existing CONDITION and OTHER. A minor point to be sure and no less petty than your own comment]; and Q69 whether the role of science tends towards the deligitimization or the legitimization of existing ‘facts’ or something else. What is one to make of them? [I am a sociologist and oddly as it may seem, I have an interest in sociology] There are shades of Karl Popper and social constructivism in there, but we’d be very surprised if any working scientist answered anything other than ‘other’.
Well, once more your preemptive assumptions prove to be incorrect:
The frequencies of responses are as such:
68. Concerning what science is in general, what would you say is its main activity?
to falsify hypothesis – 88
to verify existing conditions – 100
other – 179
69. Concerning science in general, the role of science tends towards
deligitimization of existing ‘facts’ – 45
legitimization of existing ‘facts’ – 132
other – 179
Surprise (again), but that is what the data says.
[Response: Again, I am indeed surprised. But I’m still not any the wiser about either what the questions were trying to get at or what the scientists who answered them with other than ‘other’ were thinking. But given that ‘other’ was around 50% in both cases indicates that the question was not well framed. What did you hope to find out here? – gavin]
This part I do not understand:
Gavin concludes: ‘Thus I urge Dennis Bray (who I think wrote the response) to engage more directly, and maybe ask himself (and the readers here) why the responses were as they were? There may well be lessons to learn there too.’
Why the responses were as they were? Are the responses incorrect? Or is it perhaps they simply do not fit with preconceived notions and are therefore incorrect? Furthermore, little in the way of results have been made public, so how on earth do you know what the responses are?
[Response: I was referring the comments here, not the responses to the survey which of course I have no knowledge of. – gavin]
Well, I for one would be interested in the survey results.
Response to #94
G H Sherrington Says
‘I once set up a polling group and engaged experts to write the questions, a bit like engaging professional ststisticians to help with climate science (Wegman).’
Just out of interest and for the sake of clarity: How did it turn out? Are there references to the survey? Setting up a polling group means research group or respondent sample selection?
‘Take a question from Bray and von Storch to examine methodology.
“24. With how much certainty can we attribute recent climate related
disasters to climate change?”’
For the reader, the response options were 1 – none, 7 – very much. G H Sherrington, if you are going to cite the questions please be complete.
‘This question is logically meaningless because it requires the respondent to agree that there have been recent climate related disasters. Some of us think climate is BAU, so any answer we give is, by definition, wrong.’
Well I guess the hurricanes that were so debated and the evacuation of coastal regions this year (and the failure to do so in the past) might warrant the title of a climate disaster, but then, this is just an assumption. If respondents are likely to deny these things happened then the question might well be ill posed. As for BAU a response declaring that the (non?)climate disaster is not related to climate change would be an indication of the BAU position, would it not? Following Katrina, this was a major debate. Climate related disasters are somewhat different from climate-change related disasters. What I have asked is much different from what you imply the question says. To adhere to your logic the question would have to read ‘With how much certainty can we attribute climate change related disasters to climate change.
[Response: This is another example where precision would have helped. If you wanted results related to US landfalling hurricanes (or specifically Katrina) you should have said so. But the answer might have been different if you’d said the ongoing drought in the American Southwest, or the 2003 European heat wave. By grouping together all climate-related disasters you end up losing the granularity that would allow you to say something about how specifically the hurricane/climate issue is understood in the wider field. – gavin]
Thanks for the sanity check above on how access to the questionnaires was handled.
On the Q68-69 surprise, can you break the responses down by the actual kind of science the person practices? Field worker versus lab worker versus modeler? Or do you have that kind of information?
I doubt any individual scientist has an overall perspective of the totality of work done that’s called “science” — the answers may reflect what they experience personally.
Dennis Bray (#96):
>Gavin, you were very critical of my misquoting you in one of your comments
…
> A minor point to be sure and no less petty than your own comment
Just a note: the prometheus site has Gavin’s quotes clearly set aside in a format that suggests verbatim quoting. It is, of course, always possible to still be misleading by quoting out-of-context, but at the least it is expected that such quoted text be accurate (especially given the ease of cut-and-paste from one electronic document to another).
Gavin, in his post, probably should not have changed “condition” to “hypothesis”, but there were no quote marks about his restating of your question, and therefore his gaffe was not at the same level as yours. Nor did it really change the meaning or tone of your question nearly as much as your misquoting did of his, as far as I can see.
In a more substantial note: I do think the survey results will be of value, just not as much as it could have been with more outside input. It would be interesting to know how you pre-tested the survey questions, and whether any of the issues raised in this post were raised in the pre-testing? (in addition to Granger Morgan, Ansolabehere also does some interesting survey work in this area. They both had different foci than you, but there might be mutual learning in discussions with them)
And one more example of a specific question that is ambiguous: “Given our current state on knowledge, climate change is now mostly a…” (political issue or scientific issue): I’m not sure how I would answer that. I might say “well, I think the science is sufficiently resolved that we should be treating it as a political answer in order to work towards solutions”, or I might say “well, clearly studying climate change is a scientific issue! Despite all those deniers who say that the IPCC is political, it is scientific!”
-Marcus