The publication ‘Learning from mistakes in climate research’ is the result of a long-winded story with a number of surprises. At least to me.
I have decided to share this story with our readers, since it in some aspects is closely linked with RealClimate.
The core of this story is the reproduction and assessment of controversial results, and it has unfolded with this publication.
Almost at the same time, discussions from the session reproducibility at the Royal Society meeting on the Future of Scholarly Scientific Communication were released. Similarities may suggest that my story is not unique.
The story I want to share started in 2012, after a response to my blog post here on RealClimate and a plea to publish formal rebuttal of Humlum et al. (2011).
Rather than assessing one specific paper, however, I wanted to know Why are there conflicting answers concerning climate change in the scientific literature?
So what is the best strategy for answering this question? I started off with replicating the past analyses, and both the results and the code (Mac/Linux & Windows) for doing the analysis have been made openly available on Figshare.com. It is important that the replication also is replicable.
I also managed to assemble a team for writing this paper, which also included people from SkepticalScience.
I was naive at first, thinking that we could persuade with the provision of open source code and detailed description of how a study is invalid. But it is not uncommon that the publication process is long-winded, as Bill Ruddiman explains in A Scientific Debate.
We first submitted our work to a journal called ‘Climate Research’.
The opinion of one of the reviewers on our manuscript was “profoundly negative”, with a recommendation to reject it (29 June 2012):
“The manuscript is not a scientific study. It is just a summary of purported errors in collection of papers, arbitrarily selected by the authors.”
But what does it mean being a “scientific study”? Perhaps not very well-defined, and google only returned one hit when I searched, with a vague description on Wikipedia.
A clue about the lack of science could be gleaned from the reviewers comment:
“It is also quite remarkable that all the papers selected by these authors can be qualified in some way or another as papers that express skepticism to anthropogenic climate change. I wonder why is this so?”
The same reviewer, also observed that “I guess that any one of us could collect their own favorite list of bad papers. My list would start with the present manuscript“, and remarked that “It may be published in a blog if the authors wish, but not in a scientific journal”.
That’s an opinion, and perhaps a reaction caused by expecting something different to what our paper had to offer. Apparently, our paper did not fit the traditional format:
“This manuscript itself is not a research paper but rather a review, more in the style that can be found nowadays in internet blogs, as the authors acknowledge.”
Because we disagreed with the view of the anonymous reviewers, we tried the journal ‘Climatic Change’ after some revisions (26 July 2012).
When the verdict came, we were informed that it had been very difficult to decide whether to accept or reject: “We have engaged our editorial team repeatedly (as well as a collection of referees), and the decision was not unanimous.”
The manuscript was rejected, and we were in for a surprise when we learned the reason the editor gave us:
“Nonetheless, we have agreed with reviewers who have offered some serious sources of concern and have not been persuaded (one way or the other) by blog conversations. Some of the issues revolve around the use of case studies; others focus on the appropriateness of criticizing others’ work in a different journal wherein response would not be expected.”
We were not entirely discouraged as the editor said the board was “intrigued” by our arguments, and suggested trying a different journal.
After rejection by ‘Climatic Change’, we decided to try an open discussion journal called ‘Earth System Dynamics Discussion’ (ESDD), where the manuscript and supporting material were openly accessible and anybody could post comments and criticism.
It was from the discussion on ESDD that I learned that the critical reviewer swaying the decision at ‘Climate Research’ or ‘Climatic Change’ had been Ross McKitrick (link). He was an author of one of the papers we included in our selection of contrarian papers that we had replicated.
McKitrick had apparently not declared any conflict of interest, but had taken on the role as a gatekeeper, after having accused climate scientists to do so himself in connection with the hack and the so-called “ClimateGate” incident:
“But academics reading the emails could see quite clearly the tribalism at work, and in comparison to other fields, climatology comes off looking juvenile, corrupt and in the grip of a handful of self-appointed gatekeepers and bullies”.
Nevertheless, ESDD (May 3 2013) turned down our manuscript for publication in the final and formal version ‘Earth System Dynamics’ (ESD), and the editor of ESD thought there were several problems with our manuscript: its “structure“ since there was “no paper in this paper”, “no actual science (hypothesis, testing of a hypothesis) in the main body”, that the case studies were “inflammatory and insufficiently supported fashion”, “authors’ stated opinion”, and that R-scripts do “not reveal mistakes”.
I guess we had not explained carefully enough the objective of our paper, and again, the editor expected a different type of paper. We also thought the verdict from ESDD was unfair, but it is not uncommon that authors and editors have different opinions during the reviewing process.
The paper was revised further according to many of the reviewers comments, explaining more carefully some of the critical points and taking into account the counter-arguments. Since it didn’t fit into the format of ESD, we thought it could be more suitable for the journal called ‘Nature Climate Change‘ (6 February 2014).
It was rejected, surprisingly despite quite positive reviews.
One reviewer thought it had “potential of being a very important publication”, and found the “information in the Supplementary Material to be important, compelling and well-presented”.
We were pleased by the view that it was “an important contribution to the climate science debate”, but was intrigued by the response that it was “unlike any other paper I have been asked to review in the past”.
Another comment also suggested that our work was fairly unique: “Reviewing the manuscript has been an unusual assignment”. Nevertheless, the tone was quite positive: “The manuscript is clearly written”, noting the effort going into reproducing the controversial analyses: “extensive discussion of the specific replication attempts in the supporting material, including computer code (written in R) that is available to the reader.”
But another reviewer took a different stance, thinking our manuscript was “poorly written” and that it failed “to adequately capture the importance of the project or convey its findings in an interesting and attractive manner”.
Since the reviews actually were quite encouraging in general, and we decided not to give up. We thought our paper could be suitable for another journal called ‘Environmental Research Letters’ (ERL).
The rejection from ERL came fairly promptly (21 February 2014) with the statement that our paper was an “intriguing” but “not sufficiently methodologically based for consideration as an ERL letter”.
The reviewer thought the manuscript was not suitable for the journal, as it was “more of an essay than a scientific study”, and recommended some kind of perspective-type outlet. Reason for rejection: (a) “not a research Article in the ERL style” and (b) “number of methodological concerns”.
It appeared that the greatest problem with our paper was again its structure and style, rather than the substance, however. I therefore contacted the editor of the journal ‘Theoretical and Applied climatology’ (TAAC) in May 2014 and asked if they would be interested in our manuscript.
The TAAC was interested. It has now been published (Benestad et al., 2015).
So what was special with our manuscript that was “intriguing”, “unlike any other paper“ but did not fit the profile of most of the journals we tried?
Not only was Humlum et. al. (2011) rebutted, but we had examined 38 contrarian papers to find out why different efforts give conflicting results. We drew a line at 38 papers, but could have included more papers too.
Many of these have been discussed here on RealClimate and on Skeptical Science, and are the basis for think tanks such as the Heartland Institute and their output such as the “NIPCC”. The relevance for our readers is that many of these have now formally been rebutted.
We had been up-front about our work not being a statistical study because it did not involve a random sample of papers. If we were to present it as a statistical study, then itself would be severely flawed as it would violate the requirement of random sampling.
Instead, we specifically chose a targeted selection to find out why they got different answers, and the easiest way to do so was to select the most visible contrarian papers.
Of course, we could have replicated papers following the mainstream, as pointed out in some comments, but that would not address the question why there are different answers.
The important point was also to learn from mistakes. Indeed, we should always try to learn from mistakes, as trial and error often is an effective way of learning. There must also be room for disagreement and scholarly dialogue.
Our selection suited this purpose as it would be harder to spot flaws in papers following the mainstream ideas. The chance of finding errors among the outliers is higher than from more mainstream papers.
Our hypothesis was that the chosen contrarian paper was valid, and our approach was to try to falsify this hypothesis by repeating the work with a critical eye.
Colleagues can know exactly what has been done in our analyses and how the results have been reached with open-access data code such as R-scripts that were provided. They provide the recipe behind the conclusions.
If we could find flaws or weaknesses, then we would be able to explain why the results were different from the mainstream. Otherwise, the differences would be a result of genuine uncertainty.
Everybody makes mistakes and errors some times, but progress is made when we learn from trial and error. A scientists job is to be to-the-point and clear as possible; not cosy up to colleagues. So, is is really “inflammatory” to point out the weakness in other analyses?
Hence, an emphasis on similarities and dissimilarities between the contrarian papers was a main subject in our study: Are there any similarities between these high-profile “contrarian” papers other than being contrarian?
So what were our main conclusions?
After all this, the conclusions were surprisingly unsurprising in my mind. The replication revealed a wide range of types of errors, shortcomings, and flaws involving both statistics and physics.
It turned out that most of the authors of the contrarian papers did not have a long career within climate research. Newcomers to a scientific discipline may easily err because they often lack tacit knowledge and do not have the comprehensive overview. Many of them had authored several of the papers and failed to cite relevant literature or include relevant and important information.
The motivation for the original plea for a formal rebuttal paper was that educators should be able to point to the peer-reviewed literature “to fight skepticism about the fundamentals on climate change”.
Now, educators can also teach their students to learn from mistakes through replication of case studies.
The important question to ask is where does the answer or information come from? If it’s a universally true result, then anybody should get similar answers. It is important to avoid being dogmatic in science.
References
- R.E. Benestad, D. Nuccitelli, S. Lewandowsky, K. Hayhoe, H.O. Hygen, R. van Dorland, and J. Cook, "Learning from mistakes in climate research", Theoretical and Applied Climatology, vol. 126, pp. 699-703, 2015. http://dx.doi.org/10.1007/s00704-015-1597-5
- O. Humlum, J. Solheim, and K. Stordahl, "Identifying natural contributions to late Holocene climate change", Global and Planetary Change, vol. 79, pp. 145-156, 2011. http://dx.doi.org/10.1016/j.gloplacha.2011.09.005
In a sincere quest for actual scientific truth, one tries to eliminate as much bias as possible prior to conducting research. Because the authors did not choose to replicate the results from EVERY paper, or at least a completely random sample from every sub-category in the overall category of papers titled “papers stating a position on AGW”, and offer up those codes and replications as well, their entire argument and conclusions are specious at best.
By conducting themselves in this manner, they can easily be accused of exhibiting the exact same behaviors that they are supposedly exposing. Such a paper is “missing contextual information” regarding how accurate papers in other sub categories were. If those papers were just as filled with mistakes as the 38 they chose, then the authors can be accused of “ignoring information that does not fit the conclusions”. They have zero proof that pro-AGW papers do NOT have “shortcomings [that] are due to insufficient model evaluation, leading to results that are not universally valid but rather are an artifact of a particular experimental setup.” Or “Other typical weaknesses include false dichotomies, inappropriate statistical methods, or basing conclusions on misconceived or incomplete physics.” (all quotes above taken from the abstract of “Learning from mistakes in climate research”)
And I find this behavior disturbing especially in light of the last sentence in the abstract- “We also argue that science is never settled and that both mainstream and contrarian papers must be subject to sustained scrutiny.”
Yet these authors consider those they do not agree with to be outside the borders of how they define mainstream science, and didn’t bother to subject the same number of “mainstream” science papers to the same continued scrutiny they reserve for “contrarian” ones.
I’m very grateful for all the work that has gone into this paper. For an amateur like me, it is very useful to have a peer-reviewed critique directly addressing flaws in these 38 papers.
Salamano says: “Climate scientists cannot claim the mantle of knowledge to all branches of education when publishing (including statistics, btw). You’re acting like economists can’t ‘dabble’ in the climate arena when the economic research direction enters it, yet climate scientists have carte blanche. – See more at: https://www.realclimate.org/index.php/archives/2015/08/lets-learn-from-mistakes/#sthash.oFWYfBwe.dpuf”
If the papers being debunked concern the socioeconomic impacts of global warming, sure, bring in the economists to review the debunking. If they’re about things like how solar cycles cause global warning…then an economist is an odd choice for reviewer, no? No experts in climate forcings were available?
Oddly missing so far in the thread is discussion of how these flawed papers got through peer review to begin with. I’m not suggesting mistakes don’t get published, simply that the mistakes in these papers seem to be, on balance, straightforward and not so difficult to catch.
Jim Baird #27. Rather long undersea electrical transmission lines have been constructed or under consideration. Thus conversion to hydrogen or another energy carrier is not necessary
http://rabett.blogspot.com/2012/04/facebook.html
IEHO there is nothing wrong with editors selecting Richard Tol or Ross McKitrick as referees. Eli would suggest that both should have sent a response asking if the editors knew of their relationship with the authors. The editors would have replied (most likely) yep, but we want your POV because you have skin in this game and it will be weighed in our decision.
Salamano:
Strawman #1
No one of has ever said or implied that.
Strawman #2
One can of course “dabble” in whatever they want to; again however when an expert in one field “dabbles” in other fields they are far more prone to make “beginner” mistakes.
Did I mention for example that per the supplementary material of the paper that this article is the subject of economist Ross McKitrick made the “beginner” mistake of incorrectly assuming independence of the data in his climate science paper?
I do believe that I did.
Strawman #3
“I’m not doing X because others are already doing X” is not valid logic.
Strawman #4
Creating a strawman as an excuse for “skepticism” isn’t real skepticism.
In some ways, I can understand why this paper was rejected for publication in journals concerning climate science investigations. It does not provide any information which advances the science of the earth’s climate and factors which influence the climate. It is more like Metadata in that it is a description of the data rather than the data itself. Submission to a different type of journal may have reduced the fishing for a publication venue. However, a publication in a non-climate science venue may not make the community aware of the potential to correct mistakes.
Given the stated acceptance percentage of papers recognizing the position that global warming is occurring, that the cause is increased concentrations of CO2 and that human actions are the cause of both the increased CO2 and thus the increase in global temperature, this paper may be like preaching to the choir. After all only 2% of the papers published in peer reviewed climate journals reject the hypothesis. It may be that like this paper, the authors fished around many journals until they found one where their effort could be published. Within the scientific community, the critical issue might be how to get peer reviewed journals to clearly state the bounds of topics which are suitable for their publication, verify that the editorial staff and peer reviewers have the proper expertise within the bound of topics covered in the journal and then for the journal to manage accepted papers to keep the content within the advertised bounds. Outside the scientific community, where there is a lack of consensus, the real issue may be how to get the editors and producers to provide a representative balance of information for public consumption. For example to replace the 50/50 balance with a proportion reflecting the consensus within the scientific community having the expertise in the area. When contrarian opinion is presented to the public, the editors should have a criterion for the selection of individuals with climate science expertise. While some of the non scientific media may have their own biases, those outlets which are attempting to provide balanced information will adjust this balance to reflect the scientific consensus. Perhaps this may be sufficient to shift the publics opinion more toward scientific consensus.
[edit – too much, stick to the substance]
Skeptics trolls might want to drive up to Washington State and hang around for awhile breathing deeply, refreshing smoke to the bottoms of their lungs like the firefighters fleeing for their lives have to – the smoke of a burning redwood is so intoxicating. The acrid air isn’t too bad coming back to Texas through Old Oregon Ponderosa and cedar forest valleys when you stay in that air-conditioned car. Armstrong Woods is quite pleasant behind glass. The high speed bumper traffic is quite the thrill as fires lick the sides of the highways and wild wires dot the horizon like mushroom clouds.
Gasp… wheeze.
Just another summer of business as usual in the West.
The Diners will still give you a glass of water if you ask for it.
“Because the authors did not choose to replicate the results from EVERY paper, or at least a completely random sample from every sub-category in the overall category of papers titled “papers stating a position on AGW”, and offer up those codes and replications as well, their entire argument and conclusions are specious at best.”
– See more at: https://www.realclimate.org/index.php/archives/2015/08/lets-learn-from-mistakes/comment-page-2/?wpmp_tp=1#sthash.u5lcxWi2.dpuf”
Complete and utter nonsense. If the critiques are valid, they stand–regardless of how much other crap is floating around. And if there are patterns of error that are discernible within the 38 papers critiqued, as claimed, why is it necessary to show that it applies to all bad papers? AFAICT, it isn’t.
Speaking as an economist:
I would not ask an economist to review a paper in the physical sciences. Certainly not a paper directed toward the scientific as opposed to economic aspects of climate change.
The typical economist does not understand the fundamental difference between the physical sciences and economics. It is this: There are no conservation laws in economics.
Economics really is as much of a muddle as it appears to the non-economist schooled in the hard sciences. Not (necessarily) because economists are especially stupid, but because the underlying phenomena do not obey any type of conservation laws. (Consider what you could(nt) do in physics if there were no conservation or mass or energy.) There are no necessary, quantitative, precise, stable relationships between economic variables. Economics tolerates (even requires) a broad tolerance for approximation (i.e., fuzzy thinking) when dealing with models and data. But that’s entirely inappropriate when applied to the physical sciences.
The only plausible scientific contribution that I could see from an economist in critique of statistical analysis. We deal almost exclusively in observational (non-experimental) data from complex systems (markets). That (ideally) leads to a certain healthy skepticism toward treating such data as if they were experimental data. In the best case, an economist might be able to spot issues that might not be obvious to someone who only routinely deals with experimental data. But in the worst case, the excusable imprecision from our own field could easily lead to inexcusable imprecision when applied in the hard sciences.
Eli Rabbett 56
Electrical transmission lines have been constructed and The Energy Island group also produced a map showing how the world could be supplied by an HVDC grid linking near shore OTEC facilities. The best ocean conditions for producing OTEC power however vary with the seasons and thermal conditions can be degraded by permanently located plants. It is best therefore to use moving plants that can graze the best conditions, which are also conducive to producing tropical storms, which would be degraded by these plants. It is pretty difficult to link a moving facility to a stationary grid. Also hydrolysis performed with the technique developed by a Lawrence Livermore team sequesters CO2 and neutralizes ocean acidity (https://www.llnl.gov/news/livermore-scientists-develop-co2-sequestration-technique-produces-supergreen-hydrogen-fuel ). Why would you give this up or the potential to deliver water and energy with a single grid? Also why forego the sea level benefit of moving liquid volume to the land?
As I wrote, I did not judge the paper on its merits as a contribution to economics. I judge the paper on its statistical merits. I feel qualified to do so. Indeed, I recall the day I mentioned Haavelmo (1944) to an author of a textbook on statistics for the climate sciences — and was met with a blank stare.
But, if you want to play the discipline game, have a look at the credentials of Rasmus’ co-authors.
Economist Richard Tol:
Good for you. Just as your economist brethren Ross McKitrick evidently felt qualified to author a climate science paper wherein, per this paper, McKitrick made the “beginner” mistake of incorrectly assuming independence of the data.
I am a scientist in a field closely related to climate science, and as such I doubt that I’d make the sort of “beginner” mistake that Ross McKitrick reportedly did there. I were to ever try write a climate science paper I have little doubt however that I’d make other “beginner” mistakes, especially in the highly unlikely event that the paper I was writing purported to undermine the scientific consensus on global warming.
Which is to say I do not suffer from the Dunning-Kruger effect, at least as much as some people do.
Yes – rather than play that “game” let’s instead assume that not only economists but also for example college dropout TV weathermen bloggers, fake members of the House of Lords who were trained in the classics, and lawyers at “free market” “institutes” understand climate science as much or better than actual climate scientists do.
Oh, wait…
If I were to ever author a climate science paper I’d have an actual climate scientist like Rasmus as a co-author, in an effort to ensure that any “beginner” mistakes that I might make would be caught before publication.
I sense an aversion to letting economists enter the climate forcing debate because climate scientists know more about climate. However, economists know more about how to analyze time-series data, a key research area in climate science. Time-series analysis is very tricky, and it is the rare climate scientist who has sufficient expertise. There are lots of good papers about climate trends by economists, especially in Climate Change.
Maybe one more aspect:
The paper at hand had a somewhat easy time finding mistakes for the reasons outlined in its summary and supplement. It is much harder usually to identify why a certain “outlier” study exists, and it usually takes much more dedication to do so. Unfortunately, as the authors found out the hard way, such dedication is not always “rewarded” in today’s fast-paced, “new-demanding” publishing world. And so, even poor studies enjoy a long shelf live, since many researchers do not have the time (or funding) to address why it/they might have been wrong. Kudos for persistence.
However, one does not always have to be a expert-in-the-field researcher to show flaws in other people’s (outlier) papers. So let me celebrate another paper here by a single, dedicated person, Touche Howard:
http://onlinelibrary.wiley.com/doi/10.1002/ese3.81/abstract
It reminds us that even if we (think we) do excellent work, Murphy’s law still applies … sometimes.
#61
http://www.historylink.org/index.cfm?DisplayPage=output.cfm&file_id=5496
Yes, it is just another forest fire summer as usual in Washington State. Western states that grow millions of acres of flammable vegetation have been catching on fire naturally, and UN-naturally for a very long time. Not sure what your point is exactly…
#62
“Complete and utter nonsense. If the critiques are valid, they stand–regardless of how much other crap is floating around. And if there are patterns of error that are discernible within the 38 papers critiqued, as claimed, why is it necessary to show that it applies to all bad papers? AFAICT, it isn’t”
The author’s stated inspiration was that “Rather than assessing one specific paper, however, I wanted to know Why are there conflicting answers concerning climate change in the scientific literature?”
In order to answer that question in an unbiased manner, one would first have to examine all papers in the scientific literature and determine which papers have answers that conflict with the answers of similar papers. Then all the methodologies, and all the data in those papers would have to be examined equally to determine where the differences in their conclusions originates from. That is how one answers the question ” Why are there conflicting answers concerning climate change in the scientific literature.”
Benestad et al didnt do that. They specifically “examined a selection of papers rejecting AGW.” and only a selection of papers that reject AGW. By doing so, they left the door wide open for this paper to be accused of the exact same weaknesses they outline in the papers they examined; “false dichotomies, inappropriate statistical methods” “starting with false
assumptions or executing an erroneous analysis.”
They chose to only examine papers that reject AGW for weaknesses… science demands to know why? Did the author’s personal opinions/beliefs on the topic affect how they set up their analysis? If so, it could be personal bias/congruence bias/observer-expectancy effect. Did they work from the assumption (as they provided no proof) that all papers that endorse AGW were done correctly and are free from weaknesses so the only reason why there are conflicting answers in climate science is due to non AGW papers being flawed, weak, incorrect? False dichotomy.
It is possible that AGW papers contain just as many flaws, and their conclusions may be just as incorrect. Proving that 38 papers “are crap” doesn’t prove that the papers they disagree with are NOT crap as well. If what we can “learn” from the mistakes in 38 papers is a noble and worthwhile objective, then certainly what we could “learn” from the mistakes in ALL of the paper would be far more noble and worthwhile, wouldn’t it?
I concur with Robert #55: how did the flawed papers pass peer review in the first place? Interesting question.
Also, since journals are ranked I’d like to see a comparison of the distribution of rankings between denial and consensus papers.
The methodological flaws in denial papers seem very similar to those we can identify with the material put out by denial “think tanks” – only with the think tanks we know cherry picking, manipulation of graphs, ignoring inconvenient data, & disregarding known physics are not mistakes but required tools to manufacture their “deliverables”, to use the term Dr. “Willie” Moon used to bill his paymasters.
Economist McKitrick wrote in 2003 about Mann’s climate hockeystick graph claims; Psycologist Lewandowsky started writing about climate and public opinion in 2012. Score: 1 – 1.
Better to concentrate on WHAT they wrote than on the who.
I was involved in the discussion in ESDD in 2013 when a previous version was posted as a discussion paper. As a result of that discussion I a wrote a comment paper with my colleague Martin Rypdal which we submitted to ESDD. The editor wouldn’t consider it as a discussion paper because “ESDD does not accept comments to discussion papers that are rejected.” But I sent a copy to Rasmus. I have not had time yet to study the version published in TAAC, so I don’t know if Benestad et. al. have “learned from their mistakes” back then (I have my doubts). There is much that I would write differently today (I learn from mistakes), but I think our comment is worth to add to Rasmus’ story.
Below is the abstract of the paper. The full paper can be downloaded from my Dropbox:
http://bit.ly/1EkkkDq
Abstract: “This is a critical comment to Benestad et al. (2013), with emphasis on their discussion on long-term persistence (LTP) and its implications for the significance of estimated trends. Their attempts to discredit papers by Cohn and Lins (2005) and Franzke (2013) reveal misconceptions of the principles of statistical hypothesis testing and ignorance about the large body of published literature on LTP. Their claim that it is impossible to distinguish LTP in the climate response from similar features induced by the anthropogenic forcing is incorrect; there exist at least four independent methodologies which all give similar results. Their assertion that LTP can not be a property of internal variability because it violates energy conservation and physical constraints on climate sensitivity is based on insufficient understanding of the dynamics of complex driven-dissipative systems. Their suggestion that LTP does not occur in climate models contradicts several published studies. Their “replication demo” applies an inadequate statistical measure of LTP to a cherry-picked, purely atmospheric, circulation model. The totality of the paper is a superficial blog-like list of recipes for rectification of the scientific process on climate-change issues, inconsistent with the virtues of openness and multidisciplinarity in science.”
[Response:There is a difference between ‘learning’ and ‘accepting’, Rypdal. Yes, I read your comment to the ESDD, but I was not very impressed. Your contribution was, to be frank, rather silly in parts and not convincing in general. Your assertions were then and are here misguided and based on misconceptions. I’m not suggesting that LTP does not occur in climate models. Nor am I denying the existence of LTP behaviour, but my main point is that your argument and the claims in the cited papers imply circular logic. This is carefully explained in the current SM, which was revised accounting for your previous comments. Your abstract cited above is erroneous too – we looked at CMIP simulations that involve coupled ocean-atmosphere models – not like you purport ‘purely atmospheric’. Another mistake you do is to put the blinkers on and just look at a very limited selection of data records – which happened to look like LTP. You should look at more robust indicators, such as the global mean sea level that provides an integrated picture of the bulk of planetary heat content. -rasmus]
Economist Richard Tol:
Your economist brethren Ross McKitrick evidently felt qualified to author a climate science paper wherein, per this paper, he made the “beginner” mistake of incorrectly assuming independence of the data in his statistical analysis.
I am a scientist in a field related to climate science; as such I doubt that I’d make the sort of “beginner” mistake that McKitrick reportedly did there. I were to ever try write a climate science paper however I have little doubt that I’d make other “beginner” mistakes.
Yes – rather than play that “game” let’s instead assume that not only economists but also for example college dropout TV weathermen, political advisers who were trained in the classics, and lawyers at “free market” “institutes” understand climate science as much or better than climate scientists do.
Oh, wait…
If I were to ever author a climate science paper I’d have a climate scientist like Rasmus as a co-author, in an effort to ensure that any “beginner” mistakes that I might make would be caught before publication.
#68–Still bollocks. “In order to answer that question in an unbiased manner, one would first have to examine all papers in the scientific literature…”
Clearly an impossible demand, and not obviously a correct one: you’ve provided no reason why some subset wouldn’t be perfectly adequate for addressing patterns of error. Sampling is, after all, hardly a new and radical analytic technique.
One can only conclude that you simply don’t want the question addressed–and, secondarily, that you favor the ‘if we don’t know everything we know nothing’ meme.
Susan Anderson said:
“For what it’s worth, I’ll be listening to the decorated doctor who specializes in this kind of treatment.”
Hope his decoration isn’t a Kinko’s-created Nobel Prize!
LB and others,
It is not that the papers disagree with the consensus. It is that the papers are wrong–in some cases egregiously so or even laughably so (yes, G&T, I’m talkin to you!). By all means there is room to examine many aspects to climate science. However, I think that if you are questioning the existence of the greenhouse effect or that CO2 is a greenhouse gas, then you are not a scientist. You are a joke. Jokes get laughed at.
Folks asking how flawed studies get through peer review don’t understand peer review.
Peer review is not a guarantee of correctness. All it means is that 3 or more of your colleagues thought the work was sufficiently interesting that a substantial subset of the community might like to read it. The peer reviewers may not even be experts in the subfield of the paper.
LB #52 – In a way, you miss the point. This paper, and the papers it criticizes, are rhetorical exercises, like lawyers conduct. That is, one advances one’s position, and it us up to opposing parties to advance theirs and find holes in yours, and so on. Supposedly, in long run this process arrives at the truth. It is not the ideal, pure system you apparently have in mind. In my experience, economists mostly use this rhetorical procedure, probably because there is so much ideological baggage.
Here’s a paper from psychology whose authors tried to replicate results: http://www.sciencemag.org/content/349/6251/aac4716
In most cases they were NOT able to replicate. Why? I think it has a lot to do with confirmation bias. It’s easier to see what you expect than what you do not expect. Also, negative results are often not published, e.g.: http://www.ncbi.nlm.nih.gov/pubmed/19160345?dopt=Abstract
Since climatology is not econometrics I see now reason whatsoever to consider the work of
http://www.econlib.org/library/Enc/bios/Haavelmo.html
From the article
(1) … educators should be able to … “to fight skepticism about the fundamentals on climate change”.
(2) …It is important to avoid being dogmatic in science.
Glaring contraction I’d say.
From the article
(1) … educators should be able … “to fight skepticism about the fundamentals on climate change”.
(2) …It is important to avoid being dogmatic in science.
Glaring contradiction I’d say. (typos corrected)
Ray: It is not that the papers disagree with the consensus. It is that the papers are wrong–in some cases egregiously so or even laughably so
RC: From the public’s standpoint this is a “he said, she said”. If the people who wrote the critiqued papers said, “Oops! I made a mistake.”, we’d have closure. Instead, we’ve got little expertise and since (I’m assuming) the critiqued papers are behind paywalls, we’ve got no access. Without either, we’re close to back to the same place we started, plus some ruffled feathers.
I would have liked to have seen enough of each paper to expose the details of each flaw, along with a to-a-layman explanation of the error.
You spoke about peer review’s limits. I thought post-publication comments provided a place to hash out errors and omissions. I’ve never seen them, but I would have guessed they’d have been a grand place to dig out previously found errors, as well as a way to take the community’s pulse. “I found an error”, followed by 197 “I agree it’s an error”s and two “Nope, the paper’s right”s would tell a story everyone could understand. Such a “system” wouldn’t require any changes to policy, just a change in behavioral norms.
One of this paper’s great features is that it exposes an embarrassingly bad error detection and correction system. There’s a similar current paper about behavioral papers. Turns out that many of those overstate the results.
Harkin @~75.
That’s pretty funny since you obviously don’t know what you’re talking about. Part of it is my fault for not including the link to the original article, which I meant to do. It’s a metaphor/parable but in a later comment I added this, being a little ashamed not to have noticed it was a real medical situation:
First, Sir Paul Nurse who visited James Delingpole and was accused of “intellectual rape” for doing what he did in that link (which is truly weird, given there is no attack involved in the video, plain for all to see) has credentials readily available in the public sphere.
Second, here’s the link I failed to provide earlier to the actual article I extracted:
https://andthentheresphysics.wordpress.com/2015/07/31/guest-post-what-would-you-do/
Third, before you start talking about Kinko Nobels, you might like to know this guy is my father so I’ve met a good few highly qualified scientists and know a lot about how they think and how trustworthy they are (very):
https://en.wikipedia.org/wiki/Philip_Warren_Anderson
One could hope that you will be embarrassed by your content-free display of splenetic unskeptical “skepticism” but given that Trump appears to have given us all permission to indulge our basest urges and say so publicly, perhaps not …
Shouldn’t you and your co-authors have also use your tool/program to attempt to replicate the analyses of a random sampling of research papers from the 97% consensus group? Wouldn’t the results of that inquiry prove, or disprove, the usefulness of your research and quite those reviewers who categorized your work as editorial rather than science?
Economics is institutionalized Mysticism. It should be treated as such: A belief system and not Science. Therefore, no matter how much “mathematics” and “equations” or “statistics” they put into it, it’s still a fraudulent Religion like all the others, and the most fanatical proponents of it no better than Witch doctors, Indian Guru, or your nearest Tarot card reader at the local New Age bookstore.
Wow. Thank you for this paper. It’s exactly what I requested here on this site, back in February:
Forget the naysayers, this is useful work.
Just as one can argue that an unexamined life is not worth living, one can argue that an unreflective field of study is deficient. Peer-review and the normal back-and-forth among authors are just the first two steps in that process. Every field needs occasional meta-studies to summarize what is working and what isn’t, and the reasons why. Expect to see more of this in every discipline.
@David Benson
Haavelmo, of course, laid the foundations for the statistical analysis of non-experimental data.
@LB
Exactly right. Benestad’s paper would have been more convincing if their sample had included “correct” papers using the same methods or “correct” papers by the same authors. Their exclusive focus on “wrong” papers does not tell us anything about why they are wrong. You need a control group to tell you anything about the treatment group.
Steinar,
Why should we care what you think is the reason for the lack of replication? And how is this relevant to the discussion here?
TO 85,
I think you’re too hard on economics. The analysis of markets and the development of marginal utility theory were real advances in understanding mass human behavior. If there’s a problem with economics, it’s that since it affects national policy-making directly, proponents tend to fall into politically-based, contending “schools.” Advancement in economics can come the same way it comes in other sciences: by observing and experimenting, recording, analyzing, and interpreting.
#85 you are an “economics denier”. I partly agree, but the fact remains that economists can do time series analysis, and climate scientists cannot.
What you say about economics is what the deniers say about climate science. Not helpful.
Comment by BFJ — 29 Aug 2015 @ 3:31 AM ~#82
Your comment- “Glaring contradiction I’d say” -is untrue. Consider that scientific claims that the earth is an oblate spheroid and the solar system is heliocentric are both examples of scientific dogma. You beg the question.
Steve
To Rasmus’ response to my comment:
I referred to a full paper I wrote, but never published, and gave the link:
http://bit.ly/1EkkkDq
The idea was to refer to a document that probes a little deeper on the subject of LTP and gives the context which is missing in your paper.
You write, “yes I read your comment,” and then you give a link to a different, very short comment I wrote to ESDD, and characterize it as “silly.”
You write in your post: “Is it really “inflammatory” to point out the weakness in other analyses? No it isn’t. But your style is.
Ray @89,
Please be less juvenile. Several of Rasmus’ findings (cherry picking, curve fitting, false assumptions) can be the result of confirmation bias. The polarisation of climate science increases the vulnerability to confirmation bias.
James,
Yes, that would’ve been quite worthwhile. Without that comparison, the work is rather useless.
“Rasmus & Co observe in the abstract :
A common denominator seems to be missing contextual information or ignoring information that does not fit the conclusions, be it other relevant work or related geophysical data. In many cases, shortcomings are due to insufficient model evaluation, leading to results that are not universally valid but rather are an artifact of a particular experimental setup. Other typical weaknesses include false dichotomies, inappropriate statistical methods, or basing conclusions on misconceived or incomplete physics. We also argue that science is never settled and that both mainstream and contrarian papers must be subject to sustained scrutiny. The merit of replication is highlighted and we discuss how the quality of the scientific literature may benefit from replication.”
Very true, but how do these lessons apply to earlier episodes of the polemic ( and philosophical) abuse of global systems models?
WAPO has article abt the cold blob in the Atlantic:
http://www.washingtonpost.com/news/energy-environment/wp/2015/09/24/why-some-scientists-are-worried-about-a-cold-blob-in-the-north-atlantic-ocean/?wpmm=1&wpisrc=nl_evening
in that piece, Michael Mann is quoted thusly: “I was formerly somewhat skeptical about the notion that the ocean “conveyor belt” circulation pattern could weaken abruptly in response to global warming. Yet this now appears to be underway, as we showed in a recent article, and as we now appear to be witnessing before our very eyes in the form of an anomalous blob of cold water in the sup-polar North Atlantic.”
if I can paraphrase, I think Mann just said: an abrupt weakening of AMOC appears to be underway. That is something he says he was formerly skeptical about that possibility. That is an example of the conservative, linear thinking that I think most climate scientists have taken regarding climate change and global warming. Some may cry bullshit, but the WAPO is quoting Mann in what appears to be a clear admission that his skepticism regarding the possibility of abrupt change appears to have been overtaken by the actual events in the Atlantic.
This article mentions that as the globe continues to warm we need to keep a close eye on the cold blob in the Atlantic. Sound advice! That’s how we observe climate change in all its permutations. I don’t think that WAPO or Mann could be accused on alarmism in this piece.
I continue to posit that scientists and the scientific community have not done a good job of projecting the changes that are happening and will happen and as a result, the scientists give policy makers wiggle room to avoid hard policy decisions. btw, the scientific community has many mouths, RC is one of those mouths.
#97 Mike:
A new article out from Jim Hansen and associate suggests that the weakening of AMOC is of concern but that of even greater concern is evidence of anomalous surface cold water in areas around Antarctica. They believe this is leading to processes which will accelerate ice loss and sea level rise well beyond IPCC predictions. See: http://www.columbia.edu/~jeh1/mailings/2015/20150921_IceMeltPredictions.pdf
The scientific community has made it clear that there are tremendous risks associated with climate change. It is the policy-makers who have proven ineffective in implementing the Precautionary Principle.
The link to your supplemental material at Springer is broken.