A while ago, I received a request to publish a paper on a post that I had written here on RealClimate, exposing the flaws in the analysis of Humlum et al., (2011).
Instead of writing a comment to one paper, however, I thought it might be useful to collect a sample of papers that I found unconvincing (usual suspects), and that have had a fairly high public profile.
The first step was to examine the selection of papers and repeat their analysis. For this, I created an R-package in the vein of Pebesma et al., 2012 (‘replicationDemos‘). After having repeated the analysis in these papers, I felt that I had a good understanding of how their methods worked.
The main focus was on the methods, and I could test them to see how robust the results were and see if the conclusions depended on the exact set-up of the analysis.
This process also made a number of logical flaws visible, and showed that some of the papers had applied methods in a naïve fashion, neglecting tacit knowledge about their limitations.
There were other papers too, where the flaw was obvious, e.g. mixing up statistical concepts or inconsistent physics.
Finally, I wrote the paper together with some colleagues of mine, and we tried to publish the synthesis of this work together with an essay on the connection to public view on climate change and the peer review system.
An interesting and new dimension to this was to see the common denominator found in some of these papers. They did not appear to be just some random cases which happened to be in the lower part of a statistical distribution describing a hypothetical quality of papers. These papers came more in clusters – perhaps like minds attract each other?
Our efforts met strong resistance, with one reviewer dead against our paper (others more favourable), and after a couple of rejections, we proceeded to publish the work as a ‘discussion paper’ in Earth System Dynamics (European Geophysical Union). There it is open for comments (Agnotology: learning from mistakes).
The interesting part is to read the comments – all very negative in one way or the other.
One commenter revealed that he had been the negative reviewer in one of the previous submissions, but since our paper is criticizing his own work, one could hardly avoid conflict of interest. His comment fits nicely into our discussion, however, as it shows how little substance his arguments had in the first place (they were more or less repeated in the comment to the discussion paper).
There are quite a few gems in the comments, which I will not disclose here (just follow the link).
I don’t know if the paper will get published in the end, but this is nevertheless an interesting experiment. I don’t think there are many similar papers like our out there – if anyone knows, it would be nice to have a list in the commentary field below.
update: Dana Nuccitelli has also written a blog about our paper on SkepticalScience. Another thing I’d be interested to hear is if anybody find the cases discussed in the paper too superficial/not convincing. Or alternatively which is most persuasive.
References
- O. Humlum, J. Solheim, and K. Stordahl, "Identifying natural contributions to late Holocene climate change", Global and Planetary Change, vol. 79, pp. 145-156, 2011. http://dx.doi.org/10.1016/j.gloplacha.2011.09.005
An example that springs to min is “A critical look at solar-climate relationships from long temperature series”
B. Legras, O. Mestre, E. Bard, and P. Yiou, Clim. Past, 6, 745-758, 2010, http://www.clim-past.net/6/745/2010/, doi:10.5194/cp-6-745-2010 .
This one seems like a good parallel.
S. Lewandowsky, J. Cook, K. Oberauer and M. Hubble-Marriott (2013) “Recursive fury: Conspiracist ideation in the blogosphere in response to research on conspiracist ideation” Front. Psychology 4:73. doi:10.3389/fpsyg.2013.00073
I think the concept of “agnotology” is not very well explained in the paper and integrated into the ongoing analysis. Perhaps in consequence, when I read in the conclusion, “The concept of agnotology brings all these cases together by providing replication exercises that demonstrate the shortcomings of these analyses,” I wondered what that could mean. Obviously you don’t literally mean that the concept provides the replication exercises, but beyond that, it isn’t obvious what connection you intended to draw. What do all these examples have in common other than apparently being wrong? (You may think that you just answered this in preceding paragraphs, but again, I found it very hard to follow.)
So, maybe some reviewers are sincerely confused about how philosophical this argument is supposed to be. Probably there is a way to make it clearer.
[Response:Thanks for your response, Mark. Good and sharp question. I guess agnotology can imply different things depending on where you stand (climate scientist or general public), but I guess I need to rephrase some of the text. One common denominator is of course the issue of climate change/AGW, a presence in the public discourse on climate, and the public confusion about a number of relevant issues, and the promotion of some of these concepts through various organisations. From my point of view, the replication of these cases is a common denominator, and is an enlightening exercise from which I get an understanding of why these studies get conflicting results to the main stream. Hence, a common denominator is that I think I know why those studies are wrong (and a link to agnotology). -rasmus]
> If you can point to those “snarky comments”, please show us where
> they are – we do not want to be seen as being snarky.
Nicely said (from the responses to the draft)
“where the flaw was obsious”
Perhaps “obvious”?
“in one of the previous submission”
Do you mean “in the previous submission” or “submissions” meaning there have been more than one previous submissions?
[Response: fixed. thanks. – gavin]
The term “agnotology”, as I understand it, relates to the cultural and political forces that serve to create or maintain ignorance. There are undoubtedly cultural and political forces involved in the debate over climate change; however, nothing in your paper demonstrates those forces. Rather the your arguments in the cases chosen seem to be on statistical and scientific grounds. Your arguments may be perfectly valid in that context but they fail to lead us to any understanding of cultural and political forces which “agnotology” implies.
[Response: I see your point. In broad terms, ‘cultural’ may also involve science and the science community. One point is perhaps that the science community does not seem to value efforts going into replication as much as making new discoveries, and most such work tend to be comments to individual studies. I also think there is some merit in taking a step back and look at more than one study, to see if there is something in common and to provide an overview (albeit, there are review articles). I try to understand why the studies give conflicting results and learn from mistakes, and I guess it would be interesting to understand cultural and political forces which “agnotology” implies. “agnotology” by Proctor and Schiebinger does discuss cultural and political, however. -rasmus]
1) It’s nice when a bad paper is dissected alone, but sometimes resembles whack-a-mole without obvious progress.
2) Rasmus’ and coauthors’ study is very useful in extracting, categorizing problems and explaining patterns, somewhat akin to the list a SkS.
3) Just as the EPA is not required in every ruling to once again prove the existence of atoms, in a paper about technical errors, it seems unnecessary/irrelevant to also try to also have do the social science work to dig out the agnotology connections.
That’s for sociologists, who do studies like Dunlap&Jacques’ Climate Change Denial Books and Conservative Think Tanks
Exploring the Connection (free), or for those of us who track money flows and other connections, as in CCC or Fakery 2.
Dunlap and Jacques show most climate anti-science books (by “pros”) were once mostly associated with thinktanks, but later, with the rise of self-publishing, more (“amateurs”) have been writing. I own about 50 of these, pro+amateur, and the latter often draw heavily on the former, (plus blogs, and bad papers), sometimes copying errors, illustrating the way ignorance-creation can work to spread beyond a core group. Social Network Analysis (SNA) researchers might have fun analyzing the network of relationships among (many of) the authors, organizations, etc. I would speculate that many of these papers are positively cited from with a particular small group, and mostly ignored otherwise.
It was rather surprising to see the lack of sophistication of the arguments made against your paper, which in general either took the form of: 1) it’s off topic for an article in a scientific publication to investigate why certain researchers and organizations seem to be producing papers with systematic flaws biased in favor of certain beliefs, and the nature of which call into question the intellectual integrity with which the were written. or 2) the threshold of proof demonstrating such such behavior should be held to such an high level as to make publishing such criticisms, in practice, impossible.
None of them offer any solution to the question of: if there were to be a systematic attempt to publish papers promoting an agenda how could such an attempt ever even be discussed, let alone acknowledge and prevented from corrupting an entire body of work or even a scientific discipline, since all of the complaints made against this paper could be leveled against any paper discussing a set of research of any arbitrary level of bias, intellectual dishonesty or even out-right fraudulence.
That seems to be a severe indictment not just of criticism, but of the authors of it themselves. Indeed, it’s puzzling why none of them argued from the very reasonable perspective that published articles and their authors, should be treated as having earned the benefit of the the doubt and that in all questions it is the indictment that must be proved, with respect to intent (but not with respect of the science itself obviously where a paper’s novel assertions must be assumed false until proven true) and the evidence should be compelling not just a bare statistical majority, and then go on to show why your arguments weren’t compelling, and instead argued from positions that would not only block your paper and any conceivable future paper too.
> positively cited [by and to] a particular small group
Let me guess.
When you question, ‘What do all these have examples have in common’ ?
Do you mean they are all sceptical authors? perhaps you feel you in turn have been treated in a biased way rather than your paper being objectively critically appraised? as you have made the link yourself inferring more than agnotology
So is peer review too much consensus vetting?
What kind of journal would make some review an article whose own work (read: scientific skills and/or honesty) was criticized in that article? Were they hoping for an extra balanced point of view?
I think this paper was really necessary in pointing out some of the common scientific errors in climate analysis. Also, I do see why one would want to go in detail about this “agnotology” business, as you call it. Neither do I think that it’s misplaced in a scientific journal. On the other hand, that makes it just easier to dismiss it as an overly philosophical rambling without dealing with any of the hard facts; the introduction is the reader’s first impression, after all. So I guess to achieve maximum impact, cutting down on the philosophical part and concentrating on the hard facts might be necessary. The use of the word “agnotology” and this “big picture” thing might just be overly scholarly to some. Just my 2¢.
@ara – If a paper is critiquing work by other researchers then journals will often ask these researchers to comment. This isn’t to get extra balance (which should be provided by the editor, who should take into account the possible prejudices of the reviewers) but instead to make sure that the new research doesn’t either misunderstand, or misrepresent, the critiqued work. As long as the editor does their job right and takes into account the possible biases of each reviewer when considering their recommendations on the paper then this is not a problem.
Would be great to see these comments on PubPeer.com
A fascinating paper; It’s v useful to have critiques of multiple papers in a single article. It improves visibility of the fact that the same or similar flaws occur repeatedly.
I read Scafetta and West 2005 (SW05) a few years ago. I couldn’t follow the maths in the paper and didn’t really understand the published criticisms of it that I was able to find (Alas I never found Benestad and Schmidt 2009 at the time).
At that time I didn’t understand the context, i.e. occasionally papers have been published which seem to contradict the mainstream scientific view of AGW; such papers are spread all over the internet and other media by the purveyors of “anything-but-CO2”, but these papers are generally found to be flawed fairly quickly. Obviously Realclimate and Skeptical Science could have given me that context, but as I was new to climate science I was trying to rely entirely on reading peer reviewed literature for myself. I quickly realised that I couldn’t possibly put together a coherent picture of all the evidence without expert guidance, but it was some while before I worked out which websites can be trusted. (It’s obvious now; but those who disbelieve that AGW is a major problem are so adamant that they are right, and so certain that the IPCC is dominated by fraudsters, and their arguments are so numerous and often so complex that it’s utterly bewildering for someone new to it all).
Fortunately I found Realclimate and Skeptical Science and saw the integrity and consistency of the arguments on these sites, otherwise I doubt I would have ever made sense of any of the evidence. If I’d found a paper like “Agnotology: Learning from Mistakes” I think it would have helped me to reach a sensible conclusion about AGW more quickly.
I see Ross McKitrick accuses you of bait-and-switch by using a paper about a new methodology as a cover for making criticisms of papers you don’t like.
So a bit like “Panel and multivariate methods for tests of trend equivalence in climate data series” by…er…Ross McKitrick where his test for his methodology happened to involve disproving Santer and proving Douglass…
…which I see he references in support of his comment on this paper.
Your paper takes an interesting approach – the discussion has been particularly interesting to follow.
Ross Mcktrick launces some serious accusations regarding your (in his view) repeating of errors and misrperesentations. His comments layout in some detail his view of what is incorrect – much more than just your wrong I’m right. You indicate in this post that his arguments had “little substance” but only offer that there is a conflict of interest and that he has repeated his arguments as evidence. It would seem to be in your interest to provide a more substantial reply to demonstrate the lack of substance. In fact, the current minimal reply will only lead to speculation that the accusations are valid. I realize it can get tediuous, but accusations on a public stage should not go without a proper response.
To play devils advocate from the outside looking in, it seems reasonable that someone would want to defend their work, which does introduce a bias but not a conflict of interest requiring them to excuse themselves. Further, the fact that the arguments are repeated does not in itself make them invalid, particularly if (as alleged) they were ignored the first time. Would it be possible to provide a detailed response to some of the more pointed arguments? or perhaps this detailed response already exists and it can be referenced here?
[Response: Thanks for this comment! I will revise the text somewhat to explain more carefully the problems with the papers and provide a fuller response to his comment when I submit the revised version of the paper. In the meanwhile, there is some overlap with my response to Rypdal regarding long-term persistence and trends (link). -rasmus]
Just a quick impression after a quick scan of the early pages of the paper as well as the reponses here (sorry, short on time here).
I suggest that a different title and different approach may make the paper more readable and more appealing in the scientific sense to journals.
A better title?: “Is there systemic error in one genre of climate science?” Alternatively: “Is there systemic or organized error is one genre of climate science.”
Following from that, the approach would be: (1) Hypothesis — certain errors appear repeatedly in some climate science papers, and these appear to occur systemically. (2) Describe the general nature of these errors, as an simple introduction. (3) Describe how the case study papers were selected. (4) Results — analysis of each case study. (5) Discussion — summarize the systemic errors that were found, as opposed to those which were non-systemic and any findings in the papers that appear valid. (6) Further discussion — tell what metadata about the papers (e.g. organizations which supported the work) adds to understanding of the findings. And discuss paralells to any similar historic occurances of scientific error or subterfuge (e.g. tobacco industry misdeeds).
If you find subterfuge of science or the public’s understanding, use that word or some equivalent. Use of the word agnotology should be included, but shouldn’t be the main focus.
This would be a more traditional systematicly scientific approach, rather than having the tone of putting a conclusion first and then providing facts to support it, which has the appearance of advocacy. I think you took the requisite systematic approach; the problem is one of how it is presented in the paper. So, I urge a thorough reorganization and change in tone. If advocacy is merited (and it seems to be), that is for others to do. Your role (at least in this paper) is just the scientific pursuit of what the common thread is in this genre of papers.
I asked this question over at Skeptical Science; but it would have been better here… In the interactive comments for review of this paper, in a reply by Rasmus (“SC C292: ‘Reply to Ellestad/Solheim/”klimarealistene”‘, Rasmus Benestad, 25 Jun 2013”), there is a reference to a scanned copy of a letter:
[begin quote]
I will maintain that Ellestad and “klimarealistene” are engaged in a propaganda campain. On February 3rd 2012, Ellestad wrote a letter to the director at MET Norway in an attempt to gag me, after I had commented on the Humlum et al (2011) paper on the website of a Norwegian newspaper. (A scanned copy is available on-line: https://drive.google.com/?usp=chrome_app#folders/0B5ZHm1tjzEtDWjhWZmxIQzVVSWc)
[end quote]
The link provided does not work. Is the scanned copy still available anywhere? Thanks!
Chris @18
Is this maybe what you’re looking for?
http://www.dagsavisen.no/nyemeninger/alle_meninger/cat1003/subcat1018/thread260149/
Thanks mrlee! That looks like it. Doh; I forgot it would (of course) be in norwegian; and I can’t even use a translation package as the letter is a pdf that apparently is an image — no text copy. Ah well. The whole kerfuffle does look interesting; and I’ve been fascinated to see just how much climate science denial goes on in Norway. Didn’t expect that…
#20–“I’ve been fascinated to see just how much climate science denial goes on in Norway. Didn’t expect that…”
I know–but Norway sells a lot of North Sea oil, don’t forget, and that revenue has been good to the country over the last several decades.
#20: Norway likely has the highest amount of high profile deniers per capita in the world. Rather good understanding of the real science until 2009, but a string of cold winters and cold summers ever since that effectively put an end to it, and the country sunk into a bottomless pit of denial and deliberate misinformation. The professional disinformers have a free pass to the major newspapers, and are allowed to print pure disinformation without being questioned. For TV based “debates”, pro disinformers are invited, but no real climate scientists like Rasmus. This tactic has had a major negative impact on the public understanding of the scinece behind climate change. The situation borderlines ridiculous these days, with extreme events like 50 year floods wreaking havoc every two years now.