I said on Friday that I didn’t think that Lindzen and Choi (2009) was obviously nonsense. Well, a number of people have disagreed with me, and in doing so, have presented some of the back story on the how the response was handled. I think this deserves to be more widely known in the hope that it will generate some discussion in the community for how such situations might be dealt with in the future.
From Chris O’Dell:
Given the large number of comments on the peer-review process in general and in the LC09 case in particular, it is probably worthwhile to give a bit more backstory to our Trenberth et al. paper. On my first reading of LC09, I was quite amazed and thought if the results were true, it would be incredible (and, in fact, a good thing!) and hence warranted independent checking. Very simple attempts to reproduce the LC09 numbers simply didn’t work out and revealed some flaws in their process. To find out more, I contacted Dr. Takmeng Wong at NASA Langley, a member of the CERES and ERBE science teams (and major player in the ERBE data set) and found out to my surprise that no one on these teams was a reviewer of LC09. Dr. Wong was doing his own verification of LC09 and so we decided to team up.
After some further checking, I came across a paper very similar to LC09 but written 3 years earlier – Forster & Gregory (2006) , hereafter FG06. FG06, however, came to essentially opposite conclusions from LC09, namely that the data implied an overall positive feedback to the earth’s climate system, though the results were somewhat uncertain for various reasons as described in the paper (they attempted a proper error analysis). The big question of course was, how is it that LC09 did not even bother to reference FG06, let alone explain the major differences in their results? Maybe Lindzen & Choi didn’t know about the existence of FG06, but certainly at least one reviewer should have. And if they also didn’t, well then, a very poor choice of reviewers was made.
This became clear when Dr. Wong presented a joint analysis he & I made at the CERES science team meeting held in Fort Collins, Colorado in November. At this meeting, Drs. Trenberth and Fasullo approached us and said they had done much the same thing as we had, and had already submitted a paper to GRL, specifically a comment paper on LC09. This comment was rejected out of hand by GRL, with essentially no reason given. With some more inquiry, it was discovered that:
- The reviews of LC09 were “extremely favorable”
- GRL doesn’t like comments and is thinking of doing away with them altogether.
- GRL wouldn’t accept comments on LC09 (and certainly not multiple comments), and instead it was recommended that the four of us submit a stand-alone paper rather than a comment on LC09.
We all felt strongly that we simply wanted to publish a comment directly on LC09, but gave in to GRL and submitted a stand-alone paper. This is why, for instance, LC09 is not directly referenced in our paper abstract. The implication of statement (1) above is that LC09 basically skated through the peer-review process unchanged, and the selected reviewers had no problems with the paper. This, and for GRL to summarily reject all comments on LC09 appears extremely sketchy.
In my opinion, there is a case to be made on the peer-review process being flawed, at least for certain papers. Many commenters say the system isn’t perfect, but it in general works. I would counter that it certainly could be better. For AGU journals, authors are invited to give a list of proposed reviewers for their paper. When the editor is lazy or tight on time or whatever, they may just use the suggested reviewers, whether or not those reviewers are appropriate for the paper in question. Also, when a comment on a paper is submitted, the comment goes to the editor that accepted the original paper – a clear conflict of interest.
So yes, the system may work most of the time, but LC09 is a clear example that it doesn’t work all of the time. I’m not saying LC09 should have been rejected or wasn’t ultimately worthy of publication, but reviewers should have required major modifications before it was accepted for publication.
To me this raises a number of questions. Why are the editors at GRL apparently not following the published editorial policy on comments? The current policy might not be ideal, and perhaps should be changed, but surely not by fiat, and surely not without announcing that policy change? This particular example has ended up divorcing the response from the original paper and clearly makes it harder to follow the development of this analysis in the literature. Additionally, in cases where there appears to have been lapses in peer-review (for whatever reason), is there not an argument for having a different editor deal with the comment/response? Perhaps a new online journal which independently publishes peer-reviewed comments and responses is called for?
Everyone involved in the peer-review process knows full well the difficulty in finding suitable reviewers who have the time and inclination to do a good review. The pressures on editors both to be seen to be fair, and to actually be fair to the authors (and the readers!) are strong, and occasionally things will go wrong. The measure of such a system is not whether it is perfect, but whether it deals appropriately and quickly with problems when they (inevitably) arise.
NB. Comments on how to improve the situation are welcome, but please avoid simply criticising papers that you personally think shouldn’t have been published in the form they were.
Tilo:
““nor on the fragility of trends based on 12 year periods rather than 11 or 13 year periods. – gavin]”
I believe that I have already answered that one.”
Nobody with working neurons believes you have, Tilo.
Tilo stood on the burning deck
“It’s just a light show” he insists…
CTG: #250
“No, you didn’t.”
You don’t even know what my argument is. How do you know that I didn’t.
CTG:
“It’s ironic, because you have committed the same error as L&C.”
Again, showing that you don’t know what my argument is.
CTG:
“Your result is entirely dependent on using 1998 as the start year, as I and others have pointed out.”
Again, showing that you don’t know what my argument is. I don’t see any reason to answer the same points again and again. Until someone shows that they understand my argument, this is a waste of time. If someone shows that they understand my argument and then shows me why they think it’s wrong, then we will be able to move forward. So far you show that you don’t know what my argument is and you are making the same argument that I have already answered. We either need to drop this argument or move forward from a base of mutual understanding. I understand all your points because I specifically address them in my own argument. But then you continue using the same starting point that I already answered and you don’t address any of the points that I made in my own argument.
Rodney B: “True in the real complete world. But as I said you just killed any pragmatic Physics 101 course.”
Nope, physics 101 doesn’t demand a model of the atmosphere where you can change only one thing and still get something that is physical.
And there are plenty of “physics 101” models of the atmosphere. Arrhenius did one.
But it still didn’t let him change only one thing and keep EVERYTHING ELSE the same.
You could change CO2 concentrations, but you couldn’t, at the same time, keep the TOA output the same.
Without breaking the physicality anyway.
> Tilo Reber
> many other factors than solar involved
In paleo work? Volcanism’s documented. You got something else in mind that isn’t? Over the time scale they discuss?
> The Hoffert paper has to be paid for, and
> I am not going to do that.
A back issue of Nature should be easy to find; your local library will have it, or get it via Interlibrary L o a n.
You might want first to review the lists of citing papers so you can look them up on the same trip:
http://adsabs.harvard.edu/cgi-bin/nph-data_query?bibcode=1992Natur.360..573H&link_type=CITATIONS&db_key=PHY
http://scholar.google.com/scholar?cites=212553895015365246&hl=en&as_sdt=2000
Tilo, unless you can explain why 12 is a magic number, it’s kind of hard to have much confidence in a 12 year result that disappears when we look for it at 11 or 13 years.
Likewise, if you are going to reject any argument based on models, then you are rejecting science–all of it. And just how likely do you think it is that it is sheer coincidence that a model finds the most probable value to be the same when analyzing a dozen different lines of evidence?
Finally, I never saw the 2035 estimate. FWIW, having been to Nepal, I think I would have found it somewhat surprising, but it is not a significant factoid in the climate debate. And I have never made the sorts of arguments Ms. Stewart makes, but your umbrage at ideas of equality and justice does start to illuminate your true motivations.
Gee, any other logical fallacies you want to try to include?
“You don’t even know what my argument is.”
Not even Tilo.
Well, he knows “AGW is wrong” but doesn’t know why he thinks so.
Ray: #255
“Tilo, unless you can explain why 12 is a magic number,”
I did. You didn’t read it.
“Likewise, if you are going to reject any argument based on models,”
Climate models! There is not enough information about sources of variability to make them work.
“Finally, I never saw the 2035 estimate. ”
Doesn’t matter. It came from the IPCC. And you seem to think that all of their stuff is “science”.
“Gee, any other logical fallacies you want to try to include?”
No, I get my fill looking at yours.
Hank: #254
“You got something else in mind that isn’t?”
Many. But for one:
Briffa:
“I believe that the recent warmth was probably matched about 1000 years ago. I do not believe that global mean annual temperatures have simply cooled progressively over thousands of years as Mike appears to and I contend that that there is strong evidence for major changes in climate over the Holocene (not Milankovich) that require explanation and that could represent part of the current or future background variability of our climate.”
> I believe that the recent warmth was probably matched about 1000 years ago
September 22, 1999. MBH was barely out. It’s been a while…
Tilo, You don’t understand the first thing about science. The power of science comes not from being infallible, but rather from the fact that it is self-correcting. It was not an “auditor” who found the error in the WG report, but a scientist. And actually, I am not a big fan of the IPCC. It is really not part of the science, but rather a body that summarizes the state of the science. I think it does a reasonable job at this, but unfortunately, it gives denialists like you a target you can focus on. I prefer to get my science from the peer-reviewed literature. In fact, when I am reading an IPCC report and find something interesting, I usually check the reference and look it up.
So, Tilo, if you don’t like the IPCC, that’s fine. Look at the peer-reviewed literature. You will find plenty to be concerned about there–or at least you would if you would take off your ideological blinders.
TR: if you are basing a climate sensitivity number on the correlation between radiative forcing and temperature, I don’t see how this is possible.
BPL: dT = RF * lambda
where dT is the change in temperature, RF the radiative forcing, and lambda the climate sensitivity. Units are K, W/m^2, and K/W/m^2, respectively.
TR: Until someone shows that they understand my argument, this is a waste of time. If someone shows that they understand my argument and then shows me why they think it’s wrong, then we will be able to move forward. So far you show that you don’t know what my argument is and you are making the same argument that I have already answered.
BPL: If nobody can understand the argument you’re making, making you had better rephrase it–or check to see whether it’s logically coherent.
TR: Climate models! There is not enough information about sources of variability to make them work.
BPL: http://BartonPaulLevenson.com/ModelsReliable.html
Comment by Tilo Reber — 20 January 2010 @ 2:50 PM
“Until someone shows that they understand my argument, this is a waste of time.”
Circumstantially tautological.
Of COURSE!!! It’s not Tilo’s fault he’s clueless (or appearing so), it’s OUR fault he’s clueless (or appears so)!
It’s so simple!