I said on Friday that I didn’t think that Lindzen and Choi (2009) was obviously nonsense. Well, a number of people have disagreed with me, and in doing so, have presented some of the back story on the how the response was handled. I think this deserves to be more widely known in the hope that it will generate some discussion in the community for how such situations might be dealt with in the future.
From Chris O’Dell:
Given the large number of comments on the peer-review process in general and in the LC09 case in particular, it is probably worthwhile to give a bit more backstory to our Trenberth et al. paper. On my first reading of LC09, I was quite amazed and thought if the results were true, it would be incredible (and, in fact, a good thing!) and hence warranted independent checking. Very simple attempts to reproduce the LC09 numbers simply didn’t work out and revealed some flaws in their process. To find out more, I contacted Dr. Takmeng Wong at NASA Langley, a member of the CERES and ERBE science teams (and major player in the ERBE data set) and found out to my surprise that no one on these teams was a reviewer of LC09. Dr. Wong was doing his own verification of LC09 and so we decided to team up.
After some further checking, I came across a paper very similar to LC09 but written 3 years earlier – Forster & Gregory (2006) , hereafter FG06. FG06, however, came to essentially opposite conclusions from LC09, namely that the data implied an overall positive feedback to the earth’s climate system, though the results were somewhat uncertain for various reasons as described in the paper (they attempted a proper error analysis). The big question of course was, how is it that LC09 did not even bother to reference FG06, let alone explain the major differences in their results? Maybe Lindzen & Choi didn’t know about the existence of FG06, but certainly at least one reviewer should have. And if they also didn’t, well then, a very poor choice of reviewers was made.
This became clear when Dr. Wong presented a joint analysis he & I made at the CERES science team meeting held in Fort Collins, Colorado in November. At this meeting, Drs. Trenberth and Fasullo approached us and said they had done much the same thing as we had, and had already submitted a paper to GRL, specifically a comment paper on LC09. This comment was rejected out of hand by GRL, with essentially no reason given. With some more inquiry, it was discovered that:
- The reviews of LC09 were “extremely favorable”
- GRL doesn’t like comments and is thinking of doing away with them altogether.
- GRL wouldn’t accept comments on LC09 (and certainly not multiple comments), and instead it was recommended that the four of us submit a stand-alone paper rather than a comment on LC09.
We all felt strongly that we simply wanted to publish a comment directly on LC09, but gave in to GRL and submitted a stand-alone paper. This is why, for instance, LC09 is not directly referenced in our paper abstract. The implication of statement (1) above is that LC09 basically skated through the peer-review process unchanged, and the selected reviewers had no problems with the paper. This, and for GRL to summarily reject all comments on LC09 appears extremely sketchy.
In my opinion, there is a case to be made on the peer-review process being flawed, at least for certain papers. Many commenters say the system isn’t perfect, but it in general works. I would counter that it certainly could be better. For AGU journals, authors are invited to give a list of proposed reviewers for their paper. When the editor is lazy or tight on time or whatever, they may just use the suggested reviewers, whether or not those reviewers are appropriate for the paper in question. Also, when a comment on a paper is submitted, the comment goes to the editor that accepted the original paper – a clear conflict of interest.
So yes, the system may work most of the time, but LC09 is a clear example that it doesn’t work all of the time. I’m not saying LC09 should have been rejected or wasn’t ultimately worthy of publication, but reviewers should have required major modifications before it was accepted for publication.
To me this raises a number of questions. Why are the editors at GRL apparently not following the published editorial policy on comments? The current policy might not be ideal, and perhaps should be changed, but surely not by fiat, and surely not without announcing that policy change? This particular example has ended up divorcing the response from the original paper and clearly makes it harder to follow the development of this analysis in the literature. Additionally, in cases where there appears to have been lapses in peer-review (for whatever reason), is there not an argument for having a different editor deal with the comment/response? Perhaps a new online journal which independently publishes peer-reviewed comments and responses is called for?
Everyone involved in the peer-review process knows full well the difficulty in finding suitable reviewers who have the time and inclination to do a good review. The pressures on editors both to be seen to be fair, and to actually be fair to the authors (and the readers!) are strong, and occasionally things will go wrong. The measure of such a system is not whether it is perfect, but whether it deals appropriately and quickly with problems when they (inevitably) arise.
NB. Comments on how to improve the situation are welcome, but please avoid simply criticising papers that you personally think shouldn’t have been published in the form they were.
> 1) The reviews of LC09 were “extremely favorable”
Were they completely anonymous?
Have the reviewers any related publications themselves, if anyone knows?
@Hank Roberts, O’Dell: “The reviews of LC09 were “extremely favorable”
Were they completely anonymous?
Have the reviewers any related publications themselves, if anyone knows?”
The submitting editor knows, and we should know who that is. It doesn’t say on my copy of the paper but the editor who accepted the paper isn’t normally anonymous.
I suspect they are going to plead lack of resources – they are asking for volunteers to become editors on that journal’s home page (http://www.agu.org/pubs/editors/search/GRL_Editors.pdf)
@ 83 – Hank, yes, the reviewers on LC09 were anonymous.
Chris ODell,
Thanks for the additional information. It does indeed sound as if perhaps the reviewers may have let this one get past them. In my experience, this usually happens when a paper straddles lines between disciplines/expertise. Could this have been the case here? After all, GRL has a very broad audience, and not every reader/reviewer is going to appreciate the subtleties of an analysis.
Second, although I’ve been critical of Lindzen in the past, in this case, it sounds as if he might have been short-changed by the review process. After all, I’m sure he wanted to state his case as strongly as possible, and he certainly isn’t helped by the fundamental errors in the paper. After all, Lindzen was playing the game by the rules.
In fact, I wonder if people here really appreciate that the system worked pretty well. A paper was published that made some remarkable claims. Those claims and the methodology are examined by the community at large (not there’s no “AUDIT”, no congressional hearings…). The independent researchers discover shortcomings in the paper that invalidate the results. A couple of these independent groups get together and compare notes. They also publish–in a peer-reviewed paper rather than a comment–and the shortcomings of the LC’09 are manifest.
Note that while there may be some griping about the quality of the peer review and how flaws that should have been clear made it through into the final paper, there’s no need for any of the individuals to get personal. The treatment is factual and methodological. That’s how science works!
LC’09 stuck to evidence. Others point out that the evidence doesn’t say quite what they thought. No talk of conspiracies. No talk of “groupthink” or “confirmation bias” or whether the authors involved like each other or not, because such questions are irrelevant. A paper is published. Its conclusions are not independently confirmed. The original authors say they’ll go back and try again.
Contrast that to the blogosphere, with its insinuations, character assassination and general nastiness. The climate debate cannot be a civil scientific debate, because one side has all the evidence. All the other side can do is attack small pieces of that evidence or the character of the scientists who generate it. That’s science vs. anti-science. We see it in the creationists, among the anti-vaccine movement, the power-lines-cause-cancer conspiracy theorists.
If the denialists would simply stop concentrating on decade-old studies and start trying to generate understanding on their own, I think the debate would be a lot more civil, as the case of LC’09 demonstrates.
IMO the problem here for Lindzen is not that he’s corrupt, or that he’s saying what important people want to hear, but that he’s over-ready to avail himself of any limelight he can manage.
And that’s not necessarily *wrong* but now it comes back to bite him.
As Ray says above, Lindzen’s been shortchanged by peer review and may well want the best argument possible for his stance and his paper.
But his chasing the limelight here makes it hard to put much faith behind that reading of it.
It’s just as easy to read into it that he likes the poor review process undertaken because it get him in the spotlight again.
David Bellamy found just the same sort of limelight that was easy to garner, hence his changed story on how he has been done wrong (cue blues riff…).
Does he have the courage to risk losing the fame by going “my bad” and countering his paper? It depends on how much of a career he thinks he has left, I think.
But that’s not a nice way to think, is it.
“Perhaps a new online journal which independently publishes peer-reviewed comments and responses is called for?”
I couldn’t let that comment slip without hoisting the flag for Atmospheric Chemistry and Physics (http://www.atmospheric-chemistry-and-physics.net). This is an online, open-access initial submission, reviews and responses are all available online, although the official reviewers can still remain anonymous if they want. Additionally, third parties are free to put comments into papers at the discussion stage.
Personally I think this has indeed brought the overall standard of the science up, simply because the there is much more openness in both the initial submission and the review process. One might argue that this has been partly responsible for its very impressive impact factor (4.9, which is pretty good for a relatively new journal), although I’d admit not everyone sees it that way (some people attribute it to it attracting a large number of special issues due to its streamlined publishing). But whichever way, our group is currently sending most of our papers there now at the expense of journals like JGR.
Sorry, that should have been “This is an online, open-access journal, where the initial submission, reviews and responses are all available online.”
“The measure of such a system is not whether it is perfect, but whether it deals appropriately and quickly with problems when they (inevitably) arise.” Perfect! That sums the whole situation in one nice, neat package!!
I have to wonder how carefully GRL has been screening people who answer their call to volunteer to review submitted papers. Anyone know?
I am not a climate scientist, but I will say that I think that the peer-review process could be made better in most fields of science. one review to a paper I submitted was riffled with spelling mistakes, grossly erroneous suggestions for statistical analysis, and off-topic comments. What is clear is that people are submitting more and more papers, academics are getting busier and busier, and the supply of potential reviewers is not realy increasing fast enough.
Having said that, how can we improve peer-review? here are a few suggestions:
1. Make the process doubel blind, so that the authors as well as the reviewers are anonymous.
2. Insist that the selected reviewer is the person to perform the review, not his graduate students or research assistants.
3. If you are an editor, read and pass judgement on the damned review – spelling mistakes and ad-hominem attacks are clear indicators of a shoddy review.
4. Reward scientists for performing this vital service – either through points on their annual assessment, by weighting reviews highly in tenure applications or through other means.
I am sure there are many others
Best
Andy
[Response: Thanks. Double-blind reviewing doesn’t have much of a track record in Earth Sciences (I’ve never come across it), but where it might matter is pretty much the only examples in which you’d be able to guess correctly anyway. As for point 2), I think it should be the other way around. We need a bigger pool of reviewers, not just a cadre of big-name scientists doing it all. Passing reviews to more junior members of the team (with supervision if necessary) is both good for them (experience) and for the authors (people have more time to do a good job). With you on 3+4 though! – gavin]
Any chance of GRL volunteering who the reviewers were? I smell a rat as to how those reviewers were selected.
Were they the same reviewers that Lindzen recommended to the editor? While I understand that journals and editors find it difficult to find reviewers. Really, it does not take much tome to go through the literature to find the big names in that field or names associated with the data at the centre of the paper. It places authors in a very awkward position if they have to provide two or three names of people to review their paper. If it can be shown that Lindzen provided inappropriate names (e.g., names of people who have conflict of interest or favourable bias or we not qualified to review the material), then the editor and he have a lot to answer for.
I find it odd to hear people claim that Lindzen is a victim of a faulty peer-review process at GRL. He is not a novice researcher (not even close), he has been around long enough to know when he is overstating his case, or to know when he does or does not have a good paper. Lindzen is not naive nor is he stupid; he very likely submitting this paper to provide those in denial with more fodder. Given his track record and strong beliefs that AGW is not a concern, to dismiss that (fodder) idea outright would be naive and irresponsible.
Why did he choose GRL and not JAS, or MWR (i.e., AMS journal) or JGR-A? Maybe there was no particular reason, but just maybe there was. And this fiasco points to the latter.
This fiasco at GRL needs to be looked into closer by the journal and the findings made public. I sincerely hope that this is not another “Climate Research” incident.
MapleLeaf:
“While I understand that journals and editors find it difficult to find reviewers. Really, it does not take much tome to go through the literature to find the big names in that field or names associated with the data at the centre of the paper.”
Consider, if editors direct most papers to “big names”, how can those papers be given adequate scrutiny?
Truly it is too easy to underestimate the problems editors face in seeking reviewers, just as it is too easy to dismiss the amount of work involved in scrupulously reviewing a paper.
Review is a first pass. Completely validating a submission in detail would essentially require duplicating all of the work presented in a paper, an unreasonable request of reviewers who after all are doing this work gratis, using time borrowed from professional or personal duties.
Detailed validation happens after publication, as we see here. For a field with a high level of activity and interest coupled with professional egos at stake the system works very well.
Could you (anyone) just briefly review the main arguments for and against open review, by which I mean peer review in which (1) reviewers are not anonymous and (2) reviewers’ comments are openly available?
Or point me to a quick summary of these arguments, relevant to this field?
Thanks.
I agree with James #7 and not Andy #10. The more open the process the better. I have found the reviews of may papers where the reviews are published online along with the paper and the author responses at the draft stage has led to better and more on point comments from the reviewers. There needs to be more of an emphasis on reviewing, I enjoy reviewing papers, learn something everytime. Can each review be counted toward page charges for examples.
@MapleLeaf: “Any chance of GRL volunteering who the reviewers were?”
I most certainly hope not.
“I smell a rat as to how those reviewers were selected.”
The editor should be happy to explain that selection, up to the point where the referees might be identified.
Letting the occasional loose cannon paper into a Letters journal is a minor sin for the editorial board. Mishandling a duty of trust like anonymity of the referees? That one would be inexcusable under most circumstances I can imagine.
In a scientific sense there isn’t a problem – the problem is more of a political one.
So perhaps GRL might be thought of as a bit of a rag-bag journal where short papers are allowed through an overworked editorial/reviewing system, without the scrutiny that is appropriate for a journal that publishes more substantial analyses. So the community recognises that GRL papers might be outstanding, interesting or flawed, but individuals and the community can make a scientific judgement on the quality of any paper that they consider interesting enought to read. In terms of scientific progress, the liklihood of flawed papers isn’t so problematic and one might have to accept this in the context of an overworked editorial system (if the community really wants a very rapid turnaround “Letters” style journal of this form).
And perhaps it’s understandable that GRL, in such a circumstance, should find “Comments” problematic, since the system may promote exotic, controversial, and even “political” papers since these will be likely to be allowed through even if flawed. The journal presumably doesn’t want to be overloaded with “Comments”.
I don’t think the situation is satisfactory ‘though. In my field (Molecular Biology/Biophysics) there is at least one “Letters” journal, but this has a seemingly higher degree of quality control, even if this puts the editors and we reviewers under additional workload.
Essentially the journal is a representation of the community, and perhaps it’s up to the community to decide whether it wants a “Letters” journal that operates in such a manner as to allow flawed papers through. This doesn’t matter too much in terms of scientific progress, but in the current climate, it’s unfortunate that it appears slightly disreputable, and seems to promote a drip feed of papers that are used to support a non-scientific view….
@10: Point 3 has arrived in the theoretical computer science community in the past two years. Note that our idiom is slightly different: we publish at peer-reviewed conferences, so there’s a program committee rather than an editorial board; they fill the same function. The answer is accept / reject, the former meaning we should make slight revisions, the latter meaning we should make more important revisions and submit to another conference, or the same conference a year later (or simply give up).
Whereas previously the reviewer comments came back verbatim with a congratulations / sorry email, now we get the notification on time, but the program committee edit the reviewer comments and send the edited comments about a week later. Particularly egregious comments are excised, others come with a note from the PC.
I’m not sure it has much affected decisions — the committees already did this in their discussions — but it does improve submitters’ blood pressure.
James Allan wrote in 6, 7:
I have seen something along these lines before. For example, Eugene V. Koonin was the lead author of a paper on the coevolution of viruses and cells a while back and W. Ford Doolittle was one of the reviewers. The paper was quite illuminating, but so was the exchange between two individuals who have contributed so much to evolutionary biology, and I found the latter to be a rare treat — almost equal to the paper itself. In particular, Doolittle put forward a view of science that seemed at once Popperian and post-modernist which included the view that theories of early evolution are in essence a useful fiction and not really scientific since they lack falsifiability.
Eugene V. Koonin responded in part:
Personally, I found Koonin’s views quite similar to my own as well as those that light the path taken by climatology, or for that matter, modern science in general. Without the exchange being made public like that I believe the readers would have missed out on a great deal, but instead they were invited in. At a certain level it had much the same feel as attending a performance of one of Shakespeare’s plays.
The lack of any reference to Forster & Gregory (2006) was something that struck me very much on looking over the paper. It’s strange that the authors failed to include it… it is outright surreal that no reviewers noted the omission.
Mapleleaf says “…he very likely submitting this paper to provide those in denial with more fodder. Given his track record and strong beliefs that AGW is not a concern, to dismiss that (fodder) idea outright would be naive and irresponsible.”
I strongly disagreee. If all he’d been doing was attempting to give fodder to the denialists, he would not have published, but instead simply submitted the idea on various sympathetic blogs and lamented the inability of dissenting voices to be heard in the “mainstream scientific literature”. He did at least put the idea in front of his peers rather than scoring points ex cathedra.
And I agree that Lindzen is not naive–that is why I think he actually thought he was on to something rather than trying to fool peers he knows he can’t fool.
Well things could get very interesting with FOIA requests.
http://views.washingtonpost.com/climate-change/post-carbon/
“A case in point: Greenpeace has filed a Virginia Freedom of Information Request with both Virginia Gov. Tim Kaine’s (D) office and the University of Virginia demanding an array of correspondence concerning Patrick Michaels, a senior fellow at both the conservative Cato Institute and at George Mason University. Michaels also held the title of Virginia’s official climatologist starting in 1980 until Kaine announced in August 2006 he did not consider Michaels as holding that honorary post.
The FOIA to Kaine requests any “letters, email, faxes, reports, meeting and teleconference agendas, minutes, notes, transcripts, tape recordings and phone logs generated by or involving Dr. Patrick Michaels regarding global climate change (a.k.a. global warming).”
Since Chris O’Dell has come forth with some background on his experience at GRL, I’m going to add some details on mine.
In the fall of 2008 I submitted a Comment to GRL on a paper published the previous spring, which I took serious issue with. The response from GRL led to a very Rick Trebino-like episode that lasted over a year and went through several rounds of review by 5 different reviewers and 2 Editors-in-Chief; it should have taken a couple of months, and two rounds of review by two reviewers. There were a number of suspicious or unclear decisions made during the review of the original paper, and especially, in the handling of my Comment.
Although I muddied the waters some by including data from an unpublished analysis in the Comment, which is frowned upon, GRL did not really want to publish the Comment, as evidenced by long delays in the review process, and an attempt to reject it on procedural grounds, after a previous acceptance of an earlier version. It was only through the writing of several long letters, detailing both my scientific points and the details of how the review had been handled, and sending these to the highest level of the AGU hierarchy, that my Comment was published. I simply refused to budge from a position I felt very strongly about, and which was bolstered by support from someone who has published in GRL (and had some similar difficulties, unknown to me at the time), and with whom I shared the details as the process unfolded.
Now Chris says that GRL wants to do away with Comments altogether. I’m not surprised, and as Gavin mentions, this cannot be done as some sort of whim, just because they don’t want to deal with these issues, or to follow other journals (e.g. PNAS).
Very interesting story Jim – thanks for sharing it with us. It’s not clear to me though what are the underlying reasons driving GRL’s behavior. Part of me thinks it is simply driven by their incredibly quick timescale. For instance, if you submit a letter and it requires any revision of any kind, they just reject it, but tell you to make the revisions and resubmit. You do and most times it is accepted the 2nd time, now requiring no revisions. Then they get to say that they average such-and-such from submittal to publication. Comments sort of screw this up, because they go through more rounds and have to go back to the original author for his comments and the process is just lengthy. They seem to have no appetite for lengthy processes, however warranted they may be. This is my hope of what is going on.
One has to be extremely cautious about not publishing a paper. The “only” criteria for this might be that
1- it is repetition of previous
2- methodology is somehow fundamentally flawed
But basically the conclusions should not be touched by the reviewer.
For instance, In my area there are long-standing feuds about fundamental causes of swelling in gels. Are they due to eg ion-exchange, or the Donnan effect, or both, or….???? If the editor selects a reviewer from the “opposite camp”, the comments are always brutal. It is then up to the editor to see past the animosity and publish the paper anyway. Hopefully. Science wars should not be allowed to repress the other opinions.
But, of course, then there are papers that are nonsense… ;-)
Ray Ladbury, with much respect, I do not agree with your comments on my ‘fodder’ hypothesis. If the ‘skeptics’ can get a paper which strongly challenges the theory AGW into a mainstream journal (and not E&E), then that really bolsters the case for the denialist camp, while also giving them the credibility that they crave. LC09 is now cited widely by denilaists as evidence that 1) the models are wrong and 2) that the climate sensitivity is so small that there is no reason for concern. Sadly, this flawed paper will be cited by ‘skeptics’ ad nauseum.
Lindzen id fiercely loyal to the “skeptic’s’ camp, and nowadays I suspect the climate science he woks on has very little to do with advancing the science, but rather focusing on muddying the waters of climate science.
He almost got away with it. One can argue that he would not be so silly as to overreach, but then one would be assuming rational and reasonable thought. People do unreasonable things when they are overzealous. My suspicion is that they took a calculated risk, a risk which was minimized b/c they knew that they could specify some friendly reviewers to help rubber stamp the paper.
I could be wrong, and it could be that in his single minded obsession with his beloved iris hypothesis, he honestly thought that he had nailed it with this paper.
Only Lindzen knows why he did this, but we will probably never know the truth.
PS: I and others have tried to give him the benefit of the doubt (not me here, those thoughts were in my private musings), but in the end we must not be naive and must be pragmatic/realistic, and accept that Lindzen has an agenda. This concerns me, b/c people in that position are not really interested in advancing the science. We also have to keep in mind Lindzen’s less than stellar track record. Denying the link between smoking and lung cancer, for example. Why was he even involved in that debate? That and the fact that he has admitted to taking money from the FF in the past.
Ray Ladbury wrote in 20:
He doesn’t have to fool his peers — in order to create for non-academics the illusion that there is a controversy as to whether feedback is positive or negative. He doesn’t have to fool his peers — in order to add to an alternative literature that can be referenced by fellow “skeptic” academics.
As they approach a certain critical mass it becomes easier to have a list of references that entirely sidesteps the mainstream and become insular like certain “traditions” in the humanities. Hard deconstructionism would be one example of this but there have been others. It would be a bit like Pielke referencing Pielke referencing Pielke all the way down — only at the level of a “community.”
And he and other “skeptic” academics don’t have to convince any of the mainstream academics at any point that there actually is a controversy. They only have to create the appearance of a controversy by generating a list of “peer-reviewed” articles that question anthropogenic global warming, its seriousness, etc..
That is enough justification for politicians in Congress or the Senate to argue that there is no consensus regarding global warming — and that we should “wait and see” until there is. After all, “The problem can’t be that urgent if scientists are still debating whether or not the problem actually exists.”
Creating fodder for the sympathetic blogs is one thing — creating “peer-reviewed” alternative literature with which to confuse the newspapers or make further excuses for politicians is another. And you need to make excuses for the latter if you are to “bring onboard” those who are closer to the political center in a time when many of the more extreme have been driven from office.
Admittedly it is a lot of trouble to go to. But it is capital — an investment in the future, particularly as it adds to the alternative literature and helps to create a platform for the efforts of others. It is a puzzle with which to exercise the mind. And it is a chance to prove to yourself how much more intelligent you are than everyone else — assuming they can’t figure out what it is that you are doing and prove it. Certain minds would find that a reward in itself.
*
“Just because you’re paranoid doesn’t mean they aren’t out to get you…”
Lindzen has a near-religious belief in the “stable equilibrium” notion of climate – i.e. that the system is robust with respect to perturbations (like CO2 forcing), more or less. He doesn’t seem to have much rational evidence to base this on, and he is very testy when criticized on the basis of facts.
Searching around, I came across a pretty good analysis here:
http://julesandjames.blogspot.com/2009/08/quick-comment-on-lindzen-and-choi.html
The central gist of their argument is that the authors used a suite of atmospheric models that were forced by historical SST data for comparison purposes, which seems like a strange choice (unless, of course, those were the subset of models which gave them the results they wanted) – and they didn’t look at fully coupled AOGCM results – but I’d read the post yourself.
In fact, the best test of the models so far has been Pinatubo, which did indeed work out as predicted – despite the wildly erroneous and roundly refuted claims of Douglass & Knox on the issue…
Gosh, where did they publish? In fact, the whole episode seems like deja vu… with a rather important difference vis-a-vis comments and so on:
1) http://www.agu.org/pubs/crossref/2005/2004GL022119.shtml
“Douglass, D. H., and R. S. Knox (2005), Climate forcing by the volcanic eruption of Mount Pinatubo, Geophys. Res. Lett., 32”
2) http://www.agu.org/pubs/crossref/2005/2005GL023287.shtml
“Robock, A. (2005), Comment on “Climate forcing by the volcanic eruption of Mount Pinatubo” by David H. Douglass and Robert S. Knox, Geophys. Res. Lett., 32”
Douglass and Knox [2005, hereinafter referred to as DK] present a confusing and erroneous description of climate feedbacks and the climate response to the 1991 Mt. Pinatubo eruption. Their conclusions of a negative climate feedback and small climate sensitivity to volcanic forcing are not supported by their arguments or the observational evidence. As pointed out by Wigley et al. [2005a], this is the consequence of assuming a one-box representation for the climate system, and ignoring energy exchange with the deep ocean…
You see, the nice thing about comments is that if you search for, say, Douglas Knox Pinatubo GRL, on Google or other sites, you get the original paper plus the comments on the paper – each and every time. A separate paper (unless titled, “A comment on…” etc.) will not automatically show up.
Why does this matter? Well, the denialist crowd has an unfortunate habit of ignoring comments on and revisions to published papers that work in “their favor” – let’s see, for example, does Pielke Sr. still cite Lyman on the cooling of the oceans, even though it was retracted?
Well, gosh, yes he does!
http://pielkeclimatesci.wordpress.com/2006/09/29/the-lyman-et-al-paper-recent-cooling-in-the-upper-ocean-has-been-published/
“This mystery is a critical question, as it is not known if this is just a “speed bump”, or indicates that we have a poorer understanding of the climate system, even in terms of global average radiative heating, than has been advocated by the international climate assessments such as the IPCC.”
That’s how the tobacco science boys operate – just get the message out there as often as possible. First, of course, they have to get their work into a “reputable journal.”
However, the fundamental point here remains: why did GRL allow comments on the Douglas-Knox nonsense (they used ridiculously oversimplified 1D-“feedback models” lifted from electronic circuit analysis methods as well, I think) but not on Lindzen-Choi?
I know that this is somewhat off-topic, but I saw this preprint today and was very pleased: http://arxiv.org/abs/1001.0776 Apparently IceCube, a neutrino detector near the south pole, was able to detect temperature in the ozone layer, and it correlates well with the NOAA record.
God I love science.
Having worked in publishing, I can attest to the pressure to be “timely”. One year, the organization that published the magazine (call them the International House of Pompous Physicists–IHOPP) gave every staff member a clock to remind us to be “timely”. (Another year, it was an umbrella to remind us we were an “umbrella organization”. How fricking clever and creative!)
Most journals have you propose some reviewers, but some also ask you to propose an editor (PNAS). How about that? Is that not a dangerous practice?
@23 Chris ODell: As an author you ofcourse want as short a process as possible. So you want to avoid journals that have an ineffective process. Like you, I suspect that GRL is trying to beef up their statistics on this. I.e. they’d rather reject and have you re-submit than having than have a revision. Probably that is also why they dont want comments (if that is their new policy). I really think that is a shame. You are no longer adressing a specific review. Further, while it appears to be faster then i think it is actually a slower process (you might get assigned a new editor, and a new set of reviewers which obviously needs more time).
I always remember a piece of advice I had. “If the results are not as you expect check them carefully: if they are as you expect check them carefully twice.”
I don’t think that there is any need to assume that the LC09 paper was a deliberate attempt to muddy the evidence. The authors were doubtless happy with the results they got and did not check further. I am sure other researchers, in both climate ‘camps’, have been happy with their results and not gone looking for errors.
You recently had an article headed “Unsettled science”. Until it is settled there will continue to be (and should indeed be) a series of apparently contradictory articles published in reputable journals.
It’s a bit ironic for you people to be complaining about the flawed peer review process in a rather similar way to how the skeptics are complaining! With such a strong over-reaction to any such paper that doesn’t support your view, you merely give L&C more publicity while lending weight to the skeptics who say that you are biased and are not behaving like objective scientists. Anyway, you now have your response accepted so I don’t think there is much to complain about (with a comment, you would have got less space and lindzen would have been allowed a reply!)
The main conclusion of L&C seems to be that warmer SST leads to more radiation. This seems natural. I wonder why Chris O’dell finds it amazing and incredible?
BTW I have a suggestion of why they didn’t cite Forster and Gregory – editors often select referees from the authors in the reference list :)
Mapleleaf, In my opinion, the proper criticism of Lindzen and other dissenting scientists is not that about what they publish in peer-reviewed literature, but about what they publish for credulous lay audiences.
Science worked. LC’09 raised what the authors considered a serious challenge to the consensus. Knowledgeable people in the community took up the challenge, pointed out significant flaws that invalidated the results. L&C acknowledged the flaws and are back to the drawing board. Done!
Contrast this with the obvious dreck published by G&T, Miskolczi, which lives on and on because the authors published in obscure journals and refuse to engage seriously with the community. And worse, just try to get a denialist to admit there might be a mistake in a blog post on WUWT or CA.
In this case, they cite LC’09. We cite TFOW and mention that L&C acknowledged the flaws in their paper. No muss, no fuss and nobody gets hurt. It doesn’t have to get personal, because the argument is about evidence and methodology, not personalities. That is the way science is supposed to work.
If you make the debate about evidence, then the side that is most effective at explaining the phenomena under study and predicting future behavior wins. That is an objective standard, and it favors the experts who have devoted their lives to the study of the subject. If you make it about personalities and motivations, then everybody feels equally qualified, because everyone has their own theories about personalities and motivations.
By all means, I will be among the first to attack a scientist who takes a doubtful position in front of a lay audience, but I’ll be damned if I’m going to criticize them for playing within the rules of the game, whether or not they played the game particularly well.
“PaulM says:
11 January 2010 at 9:12 AM
It’s a bit ironic for you people to be complaining about the flawed peer review process in a rather similar way to how the skeptics are complaining”
Begging the question “are ‘we people’ complaining in the same way?”.
And the answer is “no”.
Denialists (your “skeptics”) argue that peer review process is inherently broken and coopted and should be scrapped for trial-by-jury.
Here we’re arguing that poor papers get past peer review.
NOT that this shows that the peer review process is inherently and unfixably flawed. Nor that the peer review process is biased to not showing opposing viewpoints.
The question was begged and the begging went unwarranted.
PaulM, Sorry, but I’m not sure that you get it. In this case peer review did let one get by them. The referees let a paper into the literature that had obvious flaws. This is a disservice to the authors of the paper as well as the community. L&C would undoubtedly have liked to present their case in the best possible light. The failure of the process deprived them of this opportunity.
I rather doubt that L&C would have been able to make their case even under tha best of circumstances. There are simply to many insurmountable flaws in their argument. By allowing them to publish such a flawed paper, the referees allowed them to score an “own goal” rather than taking the ball out of play.
You also seem confused about what the paper was attempting to show. Clearly a warmer world will radiate more. That is not the point. The question is whether the radiation that escapes from a warmer world with both more CO2 and more water vapor is sufficiently greater that it negates the greenhouse feedback even with a very small increased temperature. That is not at all obvious and is contra-indicated by the vast majority of the evidence.
What are we to think when you don’t even understand the arguments being made by papers that support your position?
> PaulM
> warmer SST leads to more radiation
Citation to some actual sentence in the text needed; do you pull this out as “the main conclusion” based on something you read somewhere? Or your own analysis of the text?
re: 33
“If you make the debate about evidence, then the side that is most effective at explaining the phenomena under study and predicting future behavior wins.”
As long as it’s a “debate”, denialists win because the denialists win by delay. There’s an article on the blog Desdemona Despair about the collapse of pine forests in British Columbia due to the advance of the pine bark beetle. (A billion dead trees in 2 years!) The pine bark beetle isn’t a climate forcing and figures in no GCM, but it’s advance is a result of warming. Undoubtedly there will be others. The process reminds me of lines from the final stanza of “The Hill Wife” by Robert Frost:
Sudden and swift and light as that
The ties gave,
“Debate” is a quaint term for what’s going on.
Link:
http://www.desdemonadespair.net/2010/01/pine-beetles-transform-bc-forests-into.html
climatedata.info have a quite thoughful but rather biased opinion piece on peer review at:
http://www.climatedata.info/Discussions/Discussions/opinions.php?id=4079823894944676119
PaulM: The main conclusion of L&C seems to be that warmer SST leads to more radiation. This seems natural. I wonder why Chris O’dell finds it amazing and incredible?
BPL: Because the vacuum of space doesn’t begin directly above the sea?
We had double-blind reviews for Weather and Forecasting back when I was Co-Chief Editor. The review process worked well and most people liked according to the surveying we did. Younger reviewers seemed to be particularly enthusiastic, with comments along the lines of not being intimidated by big-name authors. (After a paper finally appeared, much improved over the original, one sent me an e-mail expressing shock at who the author was that he had bashed in the original review.) It was also amusing when some of the more senior reviewers would erroneously guess who the authors were. One told me, “I know why you asked me to review this paper” and I couldn’t for the life of me think of any reason other than I thought he was interested in the topic and a good reviewer until I realized he thought that one of the authors was someone he had had “vigorous” discussions with in the past. He was wrong.
The experiment ended when we started having more and more problems with the initial manuscript processing system and the people at AMS headquarters repeatedly accidentally revealing the authors’ identities to reviewers. Despite our requests to take care of the problems, there seemed to be no interest at AMS.
Personally, I don’t like open reviews. I’m not sure I’d be as aggressive as I might be. I’m perfectly willing to go out on a limb occasionally in a review when I’m not completely confident (something related to, but just far enough out of strengths that I might not know the subtleties) and have the authors respond strongly in the review and explain their points.
Gavin (re:11),
As I wrote over a MasterReource.org a few weeks back, I think that a double-blind system is probably better than the current single-blind system, and that perhaps a more open system is preferable than either. However, as you point out, there are very few journals covering general climate topics that offer either of these review methods. I suppose it is the case that the latter (open-review) has on really become possible in recent years as the use of the internet as expanded, but the former (double-blind) has always been possible. I am not sure why it has not been the standard. Perhaps others can offer some insight.
I think the Association of American Geographer publications are double-blind (which publishes some climate stuff) and also is the International Journal of Forecasting (which publishes very limited climate stuff). There are perhaps a few others. The choice of open-review journals is equally limited, mostly, as far as I know, confined to some selects EGU journals (e.g. The Cryosphere, Climate of the Past) that are niche journals rather than a place for general climate findings.
Hopefully, the future will see more options become available. As recent examples suggest, the current system, at least as applied by and to particular editors/reviewers/authors could use a little tweaking.
-Chip
Suggestions for double blind reviewing often fail to recognize how easy it often is to infer the authors from a blinded manuscript. LC2009’s heavy citation of previous Lindzen work (see paragraphs 3-5 and 18 in particular) would tip referees off to the high probability that it was a Lindzen paper even with authorship blinded. So might the discussion of the iris hypothesis in the conclusions.
People don’t write papers in a vacuum; a new paper connects to an author’s previous work and connecting a new paper to that work is often not so hard, especially in those cases where an author has a high profile (which is exactly the case where many folks want doubly-blinded review to protect a controversial author from prejudice by referees).
Moreover, since it’s so common to give talks on work in progress or circulate preprints, there’s a good chance a referee would connect the manuscript with its author, again this would matter particularly with high-profile authors.
Double-blind review might help some problems, but it would be no panacea and would be counterproductive if its effect was more to produce the illusion of anonymity than actually to address problems with review.
Chip – the sociopolitical issue here is more related to how journals deal with comments – as in “a timely response to published work” – consider how it works in Science or Nature – go read the letters sections sometimes. Again, why would GRL allow comments on Douglas-Knox but not on Lindzen-Choi?
PaulM says – “The main conclusion of L&C seems to be that warmer SST leads to more radiation. This seems natural.”
Seems natural? What does that mean with respect to the planetary temperatures?
The surface temperature also reflects warming of the ocean, doesn’t it? It also means that the CO2 blanket in the atmosphere is going to be absorbing more radiation from the surface, yes?
So, Lindzen and Choi neglect the transfer of heat to the deep ocean, on one hand, and also neglect the inevitable re-radiation from the mid/upper-tropospheric CO2 blanket.
Sounds like a re-hash of their old debunked theory about “the sky opening up to release infrared radiation to space”, doesn’t it?
It’s not too hard to see what is really going in the American-British science communities, if you bother to open your eyes – corporate fossil fuel interests (among others) and associated financial interests are exerting a great deal of pressure within the confines of the “university-industrial complex” in order to get bogus nonsense published, while also suppressing lines of research that are contrary to their interests – and clearly, the leading scientific journals (as well as the most prestigious universities) are participating in this to varying extents – a trend that has been building for several decades, at least:
http://theuniversityfaculty.cornell.edu/forums/pdfs/complex102203.pdf
The issues raised in that 2001 article in Nature, titled “Is the University-Industrial Complex Out of Control?” have not been addressed, and indeed the situation has worsened.
I agree with you Chris(23): they’re simply trying to pump too many articles through the system, too fast, for the number of available reviewers. This policy generates the problem, and is why I commented elsewhere that this high volume, “late-breaking” mentality is a serious problem. The rejection/acceptance practice you mention is disturbing and there is no better explanation for its existence than the one you’ve given. The idea that they’re considering dropping Comments altogether is a logical extension of that, and it burns me. And there is no better explanation for why some journals don’t allow them, then Rick Trebino’s, which is that they simply don’t want to admit that they published bad stuff. Having said that, the next time I dispute something, it will be as a stand-alone paper, given the amount of time and frustration I invested, only to have the original authors dismiss my points and get the last word on top of it all.
Any chance of GRL volunteering who the reviewers were? I smell a rat as to how those reviewers were selected.
Okay, first of all a demand for GRL to out anonymous reviewers coming from a person who signs their post “MapleLeaf” is a bit ironic. Most of us understand that anonymity allows people to speak openly on a potentially controversial subject. This is doubly true when the subject is a famous MIT scientist and the reviewer may be a post-doc at Random State University. We need to keep the debate on the science, not the personalities. My opinion is that double-blind reviews (mentioned above) are the best solution to this problem.
Second, some of the reactions to peer review and GRL in particular are over the top. Peer review typically involves 2 or 3 reviewers plus the editor. It’s meant as a first order check on the paper presented and will never be as good at spotting errors as the entire community can provide once the paper is published. Rebuttal articles like Trenberth et al. are the next, more rigorous stage in getting the science right. GRL published LC09 in June and it accepted a rebuttal by December. That hardly seems like they’re actively blocking criticism. Whether this should have been published as a Comment to the paper or as a standalone article is a separate argument.
Jonathan Gilligan@42 : Suggestions for double blind reviewing often fail to recognize how easy it often is to infer the authors from a blinded manuscript.
I don’t think anyone is arguing that a double-blind system would be perfectly blinded, but even in your example, you don’t know the author’s identity until you’ve read a significant portion of the paper and presumably begun critiquing the science. In the current system, the author’s identity is the first thing you really know. My guess is that if most of us commenting here received a request to review a paper by Richard Lindzen, we’d start under the assumption that he did something wrong. I don’t think the same would be true if we received a request to review a paper titled On the determination of climate feedbacks from ERBE data. In the second case, we begin with our focus on the science. Our biases would be reversed if we were asked to review a paper by Michael Mann.
And as Harold Brooks points out @40, people often think they can guess an author even when they can’t.
completely fed up: the point is the skeptics are saying that pro-AGW papers get waved through peer review by their friends, so poor papers get published, the same thing as you guys are complaining about.
Ray and Hank: the paper does not have obvious flaws. I wonder if you have even read L&C. I was referring to the 2nd sentence in the abstract.
“completely fed up: the point is the skeptics are saying that pro-AGW papers get waved through peer review by their friends,”
That’s only the veriest fraction of what the denialists say, PaulM.
Go have a look here. See anyone saying that this proves peer review is broken?
No?
Then you’re wrong: they are different.
That you have to ignore what you said and try to rewrite the past is merely icing on the cake.
A final irony here – I was asked to review a comment of McLean et al. from JGR (one of the papers Gavin has cited as being particularly egregious) at the same time I received the “we’re not fielding comments” response from GRL on LC09. The paper sent for my review was not even a comment but a purported corroboration of McLean et al. via an independent analysis and should have been rejected for this reason alone. Nonetheless it did get sent out for review while ours did not!
PaulM, you’re making statements without giving your qualifications–and you’re stating as facts what are only your opinions. You’re qualified to determine these things?
> “the paper does not have obvious flaws.”
Obvious to whom?
Some guy on a blog?
Someone who knows and understands the physics?