I said on Friday that I didn’t think that Lindzen and Choi (2009) was obviously nonsense. Well, a number of people have disagreed with me, and in doing so, have presented some of the back story on the how the response was handled. I think this deserves to be more widely known in the hope that it will generate some discussion in the community for how such situations might be dealt with in the future.
From Chris O’Dell:
Given the large number of comments on the peer-review process in general and in the LC09 case in particular, it is probably worthwhile to give a bit more backstory to our Trenberth et al. paper. On my first reading of LC09, I was quite amazed and thought if the results were true, it would be incredible (and, in fact, a good thing!) and hence warranted independent checking. Very simple attempts to reproduce the LC09 numbers simply didn’t work out and revealed some flaws in their process. To find out more, I contacted Dr. Takmeng Wong at NASA Langley, a member of the CERES and ERBE science teams (and major player in the ERBE data set) and found out to my surprise that no one on these teams was a reviewer of LC09. Dr. Wong was doing his own verification of LC09 and so we decided to team up.
After some further checking, I came across a paper very similar to LC09 but written 3 years earlier – Forster & Gregory (2006) , hereafter FG06. FG06, however, came to essentially opposite conclusions from LC09, namely that the data implied an overall positive feedback to the earth’s climate system, though the results were somewhat uncertain for various reasons as described in the paper (they attempted a proper error analysis). The big question of course was, how is it that LC09 did not even bother to reference FG06, let alone explain the major differences in their results? Maybe Lindzen & Choi didn’t know about the existence of FG06, but certainly at least one reviewer should have. And if they also didn’t, well then, a very poor choice of reviewers was made.
This became clear when Dr. Wong presented a joint analysis he & I made at the CERES science team meeting held in Fort Collins, Colorado in November. At this meeting, Drs. Trenberth and Fasullo approached us and said they had done much the same thing as we had, and had already submitted a paper to GRL, specifically a comment paper on LC09. This comment was rejected out of hand by GRL, with essentially no reason given. With some more inquiry, it was discovered that:
- The reviews of LC09 were “extremely favorable”
- GRL doesn’t like comments and is thinking of doing away with them altogether.
- GRL wouldn’t accept comments on LC09 (and certainly not multiple comments), and instead it was recommended that the four of us submit a stand-alone paper rather than a comment on LC09.
We all felt strongly that we simply wanted to publish a comment directly on LC09, but gave in to GRL and submitted a stand-alone paper. This is why, for instance, LC09 is not directly referenced in our paper abstract. The implication of statement (1) above is that LC09 basically skated through the peer-review process unchanged, and the selected reviewers had no problems with the paper. This, and for GRL to summarily reject all comments on LC09 appears extremely sketchy.
In my opinion, there is a case to be made on the peer-review process being flawed, at least for certain papers. Many commenters say the system isn’t perfect, but it in general works. I would counter that it certainly could be better. For AGU journals, authors are invited to give a list of proposed reviewers for their paper. When the editor is lazy or tight on time or whatever, they may just use the suggested reviewers, whether or not those reviewers are appropriate for the paper in question. Also, when a comment on a paper is submitted, the comment goes to the editor that accepted the original paper – a clear conflict of interest.
So yes, the system may work most of the time, but LC09 is a clear example that it doesn’t work all of the time. I’m not saying LC09 should have been rejected or wasn’t ultimately worthy of publication, but reviewers should have required major modifications before it was accepted for publication.
To me this raises a number of questions. Why are the editors at GRL apparently not following the published editorial policy on comments? The current policy might not be ideal, and perhaps should be changed, but surely not by fiat, and surely not without announcing that policy change? This particular example has ended up divorcing the response from the original paper and clearly makes it harder to follow the development of this analysis in the literature. Additionally, in cases where there appears to have been lapses in peer-review (for whatever reason), is there not an argument for having a different editor deal with the comment/response? Perhaps a new online journal which independently publishes peer-reviewed comments and responses is called for?
Everyone involved in the peer-review process knows full well the difficulty in finding suitable reviewers who have the time and inclination to do a good review. The pressures on editors both to be seen to be fair, and to actually be fair to the authors (and the readers!) are strong, and occasionally things will go wrong. The measure of such a system is not whether it is perfect, but whether it deals appropriately and quickly with problems when they (inevitably) arise.
NB. Comments on how to improve the situation are welcome, but please avoid simply criticising papers that you personally think shouldn’t have been published in the form they were.
PaulM, So you don’t consider obvious such flaws as:
1)bizarre selection of intervals that just happen to be the only ones that show the desired effect
2)the use of AMIP simulations
3)incorrect computation of the sensitivity
4)failure to cite previous work (e.g. Forster and Gregory)
just to name a few? Of course then there’s the rather extraordinary nature of the claims, which contradict every other study to date. That alone ought to have merited some attention.
Look, I think L&C were more ill served by the process than the community. I’d be willing to bet they wish they had this one back.
Jinchi,
In a small community, you get to know the players. I can often even identify my reviewers.
Me asking if they would reveal the reviewers seems to have raised some ire. Jinchi, I use a moniker for very special reasons and b/c of threats made against me in the past, but I see the irony from your perspective.
There are a few open review journals out there on the internet where the names of the authors and reviewers are known. There is no simple solution here. What I do not like and think needs to be stopped is asking authors to provide two or more names of potential reviewers before one can even submit a paper– as I was recently asked to do when publishing in an Elsevier journal.
The reason for me asking whether or not GRL would volunteer the names (note I did not say I demanded that GRL do that), is b/c I am curious who would have rubber stamped this paper. It strikes me as odd that someone in the know in this field would have done that. So either the reviewers were not suitably qualified to critique the work and/or they were not clearly being objective. I would not have been so curious had Lindzen not provide names of prospective reviewers. Maybe I am wrong and he wasn’t, but is sounds like standard procedure at GRL.
The reviewers are as much to blame as are the authors and the editors. Would it not be intriguing if Lindzen had specified sympathetic and less than objective reviewers? If he did, why did the editors not practice due diligence, see this and then see out more suitable candidates. Should the reviewer’s in question (of LC09) be asked to review more papers in view of their questionable performance on the LC09 paper? Is nobody here concerned as to who rubber stamped questionable work in question?
PaulM, had you read the paper by Fasullo et al., you would know that the LC paper has numerous and quite serious flaws, some of which are very obvious.
How does one get around the problem that people tend to cite their own work quite frequently? For example, I have reviewed a few papers in which although it was a blind review (i.e., I did not know who the authors were), I had a pretty good idea who they were upon completion of reading the manuscript. That is not an issue, unless the reviewer has an ax to grind or a personal agenda, or is competing to with the author to make a key break through in a particular field.
I guess more resources are required to help editors identify suitable reviewers. And I understand that is easier said than done.
I like points #3 and #4 @10 by Andy Park
PS: PaulM is this your blog?
http://sites.google.com/site/globalwarmingquestions/ipcc
If so, you really do need to do some fact checking on your attempt to refute statements made by the IPCC.
If I may, I think that many of the issues being brought up in the publication process are only issues in the area of AGW, and only because it has become such a heavily politicized debate. You won’t often see Fox News pundits and hordes of blogs attacking some physicist’s revolutionary position on String Theory and the existence of D-Branes. In GRL’s defense, their reason for reticently not wanting to publish comments on LC09 may well have been a result of the number of and angry content in comments that had been submitted, if the paper was that flawed, and if the primary author is as much of a lightning rod as he seems. I don’t know that that’s true, but it’s a possibility.
My point is, I don’t think that either science or the peer review system is suddenly broken. It works fine. It’s just not well suited to a topic that has become a global, bubbling cauldron of multiple branches of science, politics and economics, and one that’s motivated by its own implications to advance as rapidly as any area of science in history, and one that provokes an emotional instead of intellectual response in too many people.
Quite honestly, too, I’m partly grateful for the emotional response. I was the son of two teachers who would come home and roll their eyes at all of the kids in school that raised their hands to say “why do I have to learn this? I’ll never need this.” Well, now all those kids are adults, and they learned 1/100th of the science they needed to understand the climate “debate” and weigh in. They all have the arrogance to think they know the answers and that their opinions matter, while at the same time their understanding of the problem and the science and the mathematics barely scratches the surface. The end result is that you go to comment boards like this one and hear arguments like “CO2 can’t be bad, it’s plant food” and “don’t these scientists know that the sun controls the climate?” and “but it’s cold out now, so where’s this supposed global warming?”
My point is… the current process works well enough, AGW is going to remain a supercharged issue (and it should be) for a long time, and maybe that’s good, because maybe this generation of school children will realize that learning science, even if you don’t intend to become a scientist, is the right, human thing to do. The only measure of the worth of knowledge should not be how much more money you’ll make if you know it.
And maybe if this generation is better educated than ours, they’ll know when to listen to themselves, when to listen to the scientists, and when to ignore the pundits.
I agree with 43 Ike Solem that you have to check very often on whether or not your favorite journal has come under the influence of the fossil fuel industry. They certainly have the money to do so with a cash flow of $1 TRillion per year. Have you done that check recently with this journal? You have to get financial disclosures from the editors and the publisher as well as from the ownership on perhaps a weekly basis.
RE #10, & “4. Reward scientists for performing this vital service – either through points on their annual assessment, by weighting reviews highly in tenure applications or through other means.”
Some journals as a reward are giving access for 30-90 days to all the journals their parent company publishes, like ALL the Sage articles or ALL Elsevier articles (or Science Direct) or ALL Springer articles. This is really great for social science scholars, though it might not be much of a perk for those in the physical sciences who have free access through their university library to all the journal articles they need.
James Annan has had some excellent thoughts on the peer review and editorial standards at AGU journals, most recently inspired by L&C2009 , as well as by McLean et al 2008 and Klotzbach et al 2008 (both in JGR). Annan’s thoughts are well summarized in his advice to the AGU post.
Here are two of the eminently sensible, EGU-inspired suggestions from Annan:
a) Identifcation of the responsible editor for each paper
b) Obligation of the editor to choose at least one reviewer outside the list suggested by the authors
In my opinion, the case of McLean, de Freitas and Carter 2008 was especially problematic, as the paper was so obviously flawed, and the authors co-operated in a highly misleading PR campaign upon its release, as I explained in great detail at the time.
I do not believe that LC2009 rises to that level of abuse of the peer-review system. However, I agree with Annan in his statement on the matter:
As for the paper itself, it seems hard to defend it as merely honestly mistaken, given the errors identified. However, I haven’t seen LC’s defence…
Journal review policy should always include sending the paper out to those most likely to criticize it – and pretending that scientists are unemotional objective calculating machines is ludicrous.
The ideal peer review team might include a passionate supporter, a harsh critic, and a neutral outsider. If the reviewer’s comments are then not addressed by the authors – if the paper is not revised to take them into account – then the paper should be rejected by that journal.
If published, the journal then has a responsibility to publish comments that address specific issues within the paper, particularly if they refute the overall conclusions of said paper.
If you don’t take such steps, you’ve simply abandoned the peer review process (which by the way, tends to work similarly in the even more contentious issue of grant applications to federal financing agencies – although apparently the DOE doesn’t use peer review at all in that process, preferring to give grants to private contractors and corporate-public “partnerships” – talk about conflict of interest and gross negligence in dispersal of taxpayer funds, as compared to the NSF or NIH).
Even with good peer review, erroneous papers still can make it through, and second-rate research proposals can still get financed – but usually they’re exposed as such over time – or are simply ignored and never referred to again.
The final stage of peer review is actually whether or not other people build on the work – the citation count – and heavily cited papers make it into the reviews, and hence into the textbooks, and hence into the public perception of the scientific enterprise, as “commonly accepted knowledge” – but even that can be overturned by some startling new scientific discovery, rare though that is.
That’s the real difference between science and ideology/religion – prejudices are discouraged, an open mind is encouraged, and there are no shibboleths or sacred cows that cannot be discussed or challenged. Of course, as long as religion and ideology stay out of scientific questions, there is no real conflict between the two – but historically, that has not been the case, has it? Communist and fascist governments of the 20th century clearly tried to make scientists conform to their various ideological views, as did the Church in centuries past… but the new point of ideological conformity in scientific institutions now seems to revolve around, for lack of a better phrase, “capitalist ideology” – dogmatic alignment with the so-called “profit motive” is now becoming a condition of employment in our public and private universities (look at the growing administrative obsession with patents if you doubt this).
This even extends to universities supporting blatantly nonsensical notions like “clean coal” and “zero-emission combustion” and “carbon offset trading” – all of which which pleases their corporate partners to no end (witness the Stanford-Exxon-Schlumberger-GE-Toyota GCEP program).
Regardless, pursuing science is a lot more fun and interesting than conforming to dogma, isn’t it?
Mapleleaf,
OK, let’s say you are an editor or reviewer and get a paper like LC’09 from a prominent “skeptic”. The paper has some serious flaws, but it has already made the rounds of the blogs where they are saying denialists can’t get published.
Do you:
1)Take it on yourself to recommend rejection, knowing the howls of righteous indignation that will echo through the denialosphere?
2)Attempt to reject the most egregious of the errors, put in a boatload of work doing so, but ultimately recommend publication.
3)Let it be published as is, knowing the community will see through the errors and show the work for the flawed example it is.
Me, I would choose 2, but 3 would be a lot less work and the end result is the same in terms of the science. Not saying this happened, but the current outcome is a lot cleaner than it is with many pieces that are much more flawed.
My take on the source of the problem at GRL is that they made a decision to speed up the turnaround time on articles for the sake of “rapid communication.” They only give reviewers two weeks to complete a review. Since many of us can’t fit a review into our schedule on that time scale, more people say no, so it’s hard to find reviewers. And then when you do find reviewers, often that two weeks can get squeezed into a rather perfunctory review on the last day if something unexpected comes up. Short papers should be fast to review, but the creeping growth of supplementary material offsets that. In addition, even a short paper can have techniques in it that are time-consuming to check (though the disturbing thing about Lindzen and Choi is that many of the things wrong with the paper can be spotted with even a cursory review, which evidently didn’t happen).
MapleLeaf @53 Me asking if they would reveal the reviewers seems to have raised some ire.
That wasn’t ire. I was simply pointing out that there are perfectly good reasons for anonymity and your use of a pseudonym demonstrated that you were well aware of them. GRL can’t simply volunteer the names of the reviewers, now.
So either the reviewers were not suitably qualified to critique the work and/or they were not clearly being objective.
And there is a perfect example of one of those reasons. You’re making a judgment on both the character and competence of people about whom you know nothing more than that they reviewed this paper. Reviewers are unpaid, volunteers. Nobody wants to be held responsible for the flaws of someone else’s research. If you want to guarantee that nobody again agrees to review potentially controversial papers, then by all means, renege on a pledge of confidentiality.
Ray @52 In a small community, you get to know the players. I can often even identify my reviewers.
But the community in this case isn’t nearly so small. Otherwise, we’d all know who reviewed this paper, wouldn’t we.
OK wrt James Annan’s comment, “As for the paper itself, it seems hard to defend it as merely honestly mistaken, given the errors identified. However, I haven’t seen LC’s defence…”
Why does the paper need defending at all? It was flawed, but it was put before the community. The motivations do not matter. What is more, Lindzen has no need to “establish himself” in the community, so while I think you could question the authors’ judgment somewhat, he will continue to receive attention.
There are worse things than being wrong in the published literature–one of them is never publishing in the first place.
RE, #43 & “the university-industrial complex.” I’ve seen the impact at my campus, in a situation of ever decreasing state funding (it’s now at 12%) at this state university, and ever increasing industry funding.
A year or so ago I suggested that our campus environmental club show WHO KILLED THE ELECTRIC CAR? (mainly bec of the technology it discusses, not even thinking about the political issues), and a science prof objected, saying “We can’t show that; we’re funded by Exxon, GM, [and so on].” I responded that maybe YOU (the physical sciences) are so funded, but the social sciences are not so funded. But I didn’t pursue my suggestion at that time, so shooked was I about the REAL SITUTATION of things.
But I did run into a poli sci prof, who was heavily involved in trying to stop CIA funding in the soc sciences, and I told him “The CIA is the least of our worries; it’s big oil and industry that are jerking us around, creating a chilling effect re showing such films as ELECTRIC CAR.” He was surprised, then enflamed, and guaranteed that film would be shown on campus that year. And it was — in the social science building to soc sci students.
But really it is the industrial-military-government-media-university complex.
Oh, I forgot religion. For example, there is the Acton Institute for the Study of Religion and Liberty (www.acton.org), which supplies GW denialist statements and speakers for religious media programs — heavily funded by Exxon (http://www.exxonsecrets.org/html/orgfactsheet.php?id=5 ). Also the Interfaith Stewardship Alliance (http://www.exxonsecrets.org/html/orgfactsheet.php?id=142 )
I’ve been listening to religious radio, and “Faith 2 Action” and “CrossTalk” are both heavily denialist “Christian” media orgs. When I wrote to CrossTalk, they sent me back this: “We recommend ClimateDepot.com for a more scientific assessment of the fraud called ‘climate change.'” I haven’t found their connection to Exxon, so maybe it’s to some other big biz interest.
So it’s the industrial-military-government-media-university-church complex. About the only social instition not involved in the greatest denialist hoax that ever threatened life on planet earth is the family.
But this is not saying that the journal in question or the editor’s university is involved. There are still good folks out there, like the RC people. And I guess big oil/coal doesn’t have enough money to buy off everyone, though they can do slights of hand and make it seem so — like getting big subsidies and tax-breaks from government (a good return on their campaign contributions to the Repubican and Democrat politicos), which, of course, we the people have to pay for April 15th, but are too not-smart to realize that, happy as we are with lower gas & electricity costs, not realizing the true costs or the really true costs.
Jinchi: “Reviewers are unpaid, volunteers. Nobody wants to be held responsible for the flaws of someone else’s research. If you want to guarantee that nobody again agrees to review potentially controversial papers, then by all means, renege on a pledge of confidentiality.”
I am occasionally an “unpaid volunteering” reviewer, and I definitely hold myself responsible for potential flaws in someone else’s research, if I were support its publication … and I wouldn’t be shocked if others were holding me responsible too (in the present case of anonymity, that would be the editor only). I can see a few good reasons for the anonymity of reviewers, but allowing them to not be held responsible for what they declare worth publishing is not one of them.
@59: “OK, let’s say you are an editor or reviewer and get a paper like LC’09 from a prominent “skeptic”. The paper has some serious flaws, but it has already made the rounds of the blogs where they are saying denialists can’t get published.
Do you:
1)Take it on yourself to recommend rejection…”
We can stop right there. If I don’t think the paper should be published, I’m going to say so. Full stop.
I do not believe in allowing any non-scientific considerations into the question of what should be in the scientific literature.
Frankly, I was occasionally a reviewer for a Letters journal, and I did once have a paper that I recommended rejection – it was clearly a Ph.D. thesis, joint with an extremely highly regarded adviser. It was a really really hard experiment, but they had overlaid some mathematics over their results, and pretty badly. I was able to infer that their incorrect mathematical interpretation had led them to suppress observations which I was sure they must have made which would have suggested to them the correct theory. Presenting only the data which supports a particular theory? Well that one won’t wash with me.
So I let ’em have it with both barrels. Yes I did understand this was a kid’s thesis and that he had apparently spent many years doing this painstaking experiment. Did I mention this was the first paper I ever reviewed? And that I was an unknown first semester assistant professor? I still am unknown but that’s another story. Thumbs down, said I.
OK so what happened is that the editor took the time to reconcile my brutal, but detailed and clearly stated criticisms, with the two other completely positive reviews. I think the other referees stopped reading when they saw who’s lab it was. Well, the editor decided to require substantial revisions, which boiled down to incorporating my criticisms. When this was done, the famous great scientist author actually asked if I would like to be a coauthor, which I declined on grounds of my idea of anonymous review. The end result? A paper well worth publishing.
So call ’em as you see ’em and let the freaking chips fall. Just stick to what you know (that’s whey they asked you to review) and make sure you’re right. Science is supposed to be about getting it right.
Peer review won’t improve much if we don’t stick to the truth as much as possible.
Jinchi,
Before I decide to review a paper amongst the questions that I ask myself are 1) Am I qualified to do this (I have received requests to review manuscripts in the past on a subject that I was not 100% comfortable, and believe that I would be doing the science and authors a disservice by agreeing to review the paper) 2) Do I have a conflict of interest?
It is an immense responsibility, in my opinion, to review a paper. For me, my reputation is also on the line when I review I paper; even if the review is anonymous, the editor still knows who I am and may not invite me back or may caution others about using me as a reviewer in the future.
So the fact that the editors, and two (?) reviewers missed so many serious flaws is of concern to me. This strikes me as more than a coincidence. Maybe it was a fluke, but I am not going to be naive and assume that it was not just b/c that would raise uncomfortable questions as to possible motive.
Maybe part of the solution is to have editors more involved in the review process and to have at least three solid reviewers chosen by the journal, not the author/s.
If you want to see some psychological wreckage emanating from the peer review process, take a look at this:
“Innocuous enough on the surface. What makes this sentence interesting (and I noticed it because I looked for something like this) is that, in my opinion, the sentence is sufficient to identify the paper in question. Further, there is convincing evidence that Jones did in fact carry out the requested review (after May, as he says here) and, even though the review is not in the Climategate documents, it is nonetheless accessible and, together with other Climategate Letters, leads on to many backstories.
Heike of the Climategate Letter 1080257056 can conclusively be identified as Heike Langenberg of Nature – enabling us to conclude that, around March 9, 2004, Jones was asked to review a submission to Nature.
…
There’s another strand of evidence suggesting that Jones was the added reviewer. Elsewhere, we’ve seen Jones’ tendency in reviews to self-cite. The added reviewer cited Jones and Mann (2004) on matters M&M – an article that was not even published until May 6, 2004 – after our re-submission to Nature in late March 2004.
Right now the evidence is circumstantial. (The question could be easily settled by either the University of East Anglia or Nature.) I suppose that it is remotely possible that, in March 2004, Nature asked Jones to review another paper and asked someone else to review our submission. But that seems a bit farfetched. For now, let’s work with the assumption that Jones was the added reviewer (and I’ll refer to the review by the added reviewer for the rest of the post as the “Jones Review” ).
Amusingly, the “Jones Review” used the word “tricky” – a word that Jones notoriously used elsewhere (as “trick”) in his es’ email about a “trick… to hide the decline”.”
He rambles on like this, for a long time.
http://climateaudit.org/2010/01/05/climategatekeeping-the-nature-intervention/#more-9706
Disturbing, or just disturbed?
An interesting alternative can be found at “The Cryosphere” an on-line journal by the European Geosciences Union at: http://www.the-cryosphere.net
Take some time to look around the site and especially the TCD section where submitted papers are open for comments — anonymous and otherwise — by editors, reviewers, and interested parties.
Ray,
“The climate debate cannot be a civil scientific debate, because one side has all the evidence.”
That’s certainly a point of view, not too far removed from:
“The climate debate cannot be a civil scientific debate, because one side refuses to concede it does not yet have all the evidence.”
You go one to make your political fears plain enough, (“enough justification for politicians in Congress or the Senate to argue that there is no consensus regarding global warming”), and why you fear any concession to there being any debate will undermine the cause. It is easy to see why you are so torn between acknowledging Lindzen’s legitimacy on the one hand, and sneering at his obvious incompetance on the other. It’s simply too hard for you to imagine he might have some contribution or other to make if it does not fit with your pre-determined view of AGW science. It’s ‘Alice in Wonderland’s” sentence first, trial second with you: A) Lindzen has to be wrong. B)Lindzen has published. C)Let’s find out what he did wrong.
The problem with that tautological approach, though, even if Lindzen proves to be wrong when he resubmits, is that it gives the impression of a closed-mindedness that so many scientists would prefer not to have ascribed to their profession.
That said, what is written by the regulars here is beside the point. Lindzen, as you say, is the published climate scientist in the arena, not Hank and Ray and the gang. It will be interesting to see what develops, and I note that Lindzen sounds eerily like Mann (and thousands of others, I’m sure) when he comments that despite the criticisms, which he and Choi have addressed, the results remain, especially the discrepancy between the models and observations.
The worst-case situation is where the reviewers chosen are exclusively the ones recommended by the submitters. Annan appears to be an advocate of the open review system employed by other journals (EGU), which would eliminate such problems.
http://julesandjames.blogspot.com/2010/01/open-review-process.html
@54:
Well, so much for my html skills (preview would be a big help). In #71 I had intended to reproduce this passage from Bob’s (#54) post:
I was the son of two teachers who would come home and roll their eyes at all of the kids in school that raised their hands to say “why do I have to learn this? I’ll never need this.” Well, now all those kids are adults, and they learned 1/100th of the science they needed to understand the climate “debate” and weigh in. They all have the arrogance to think they know the answers and that their opinions matter, while at the same time their understanding of the problem and the science and the mathematics barely scratches the surface. The end result is that you go to comment boards like this one and hear arguments like “CO2 can’t be bad, it’s plant food” and “don’t these scientists know that the sun controls the climate?” and “but it’s cold out now, so where’s this supposed global warming?”
It is an immense responsibility, in my opinion, to review a paper.
Again, you’re making a huge assumption about the character of the people who reviewed this paper. Note that we don’t expect every published paper to be free from error. Neither do we expect every reviewer to catch every flaw, no matter how seriously they take the responsibility.
Remember that this is what gavin wrote about this paper just yesterday having read the paper, the critique, and the RC post specifically written to rebut the paper:
He may have changed his mind after convincing argument in discussions here and elsewhere, but to claim that LC09 had obvious flaws simply doesn’t hold water. Or do you think gavin’s reputation suffers for having missed the “obvious flaws” before?
Walter Manny, Well, when I say one side has all the data, the wonderful thing about science is that you can prove me wrong in an instant–just publish your data. Failing that, after awhile assertions that the denialists have a scientific case being to sound like a third grader saying “Oh yeah, well my dad’s a professional wrestler and he’ll rip out your dad’s spleen!”
And nowhere did I ever disparage the abilities of either Lindzen or Choi. I think they are quite bright. The problem is that their ideas don’t have much explanatory power because they reject a crucial aspect of the consensus model.
In this particular case, he did at least publish–after first blogging about it on WUWT, but he did publish. Now, what happened next what his colleagues–you know, scientists–tried to reproduce his results and found problems. How is this outside the norm of science? It is how the game is played. Would you prefer they let a flawed analysis stand?
And as to the models and observations, there is absolutely no reason why the observations should have agreed with these particular models–that wasn’t what the models were constructed to do.
Walter, even Roy Spencer said this analysis was flawed. I think you really should look at your reaction to this. In this case, science worked as it should.
I wrote about the “GRL problem” a bit more than a year ago in my blog in the context of a different paper:
http://tinyurl.com/yjd2p64
My main complaint was that, because of space restrictions, articles in GRL (and Science and Nature) are unlikely to contain enough information to be reproduceable, or properly evaluated, or even useful. My proposed solution: Any such short publication ought to be followed up with a long-form publication (a manuscript of regular length) [in an appropriate journal], so that the methods and results can be fully explored by the authors and by the readers.
I would propose that climate papers in GRL, Science, and Nature should only be cited within the first two years of publication. By the end of that time, the corresponding full-length paper should have been published (or rejected), and the early-release short-form paper should be ignored in either case.
Ray,
I agree with most of what you just wrote other than the “denialists” having no case. For the sake of argument, though, let’s stipulate that they have no case, but that they do have a place. I, for one, am glad they are out there kicking the tires given the consensus’ reluctance to close the book on anything (sorry, metaphor too many) other than scary long-term forecasts. I don’t see much evidence of the Jones and Manns of the world aggressively seeking holes in their models, searching for negative feedbacks that might damage their political cause. Perhaps you can cite some cases where that is not true.
To be sure, Lindzen is predictable and is looking for results that show less warming, just as the AGWers predictably look for results that show the opposite. You would say the latter find warming all the time because that’s where the evidence, peer-reviewed, takes them, and fair enough. I would say that agendas that have long since polluted this scientific field are at least partly responsible, as many of the UEA emails show no matter how hard folks try to parse them into oblivion.
I read the Spencer piece when it came out (November?) and was impressed that there was debate in the “denialist” camp (or that the “conspiracy” had come off the rails for a moment :). I am equally impressed that there seems to be the opening of a debate here about the legitimacy of Lindzen’s [erroneous] publication, perhaps as a concession to the bad PR from UEA, and that there might even ensue an open debate about Lindzen and Choi’s alleged fixes when they are published. So, yes, the science worked, and perhaps it will keep working.
many of the contributors and their quals are well known to me after now having spent some time here, not so the people that i ask to visit
“who the hell is Ladbury, etc,
would it be possible to have a post listing such ,
like ” who the hell is gavin?
he is a NASA climatologist cloth head ,
Which models are “theirs”? Mann’s not one of the GISS modeling team, and Jones isn’t part of the Hadley Centre modeling team. Kinda hard to seek holes in their models when they’re not working on models. They both work on paleoclimate, i.e. reconstructions of what climate was like in the past.
You really need to bone up more on what’s what, and what’s not.
Walter Manny says, “I don’t see much evidence of the Jones and Manns of the world aggressively seeking holes in their models, searching for negative feedbacks that might damage their political cause.”
OK, two things wrong with this. First, we are talking about a community of a few thousand researchers. Is it your contention that NONE of the 97% of actively publishing climate scientists on the consensus side are actively looking to overturn the current model? Do you really think that they would say no to the fame and adulation they would receive if they overturned the current theory and made the threat of climate change disappear? Do you think that if there were an adverse outcome of an investigation that any real scientist would sweep it under the rug? If you really think this, then you don’t know jack about scientists!
Look, Walter, climate scientists don’t become climate scientists to save the world. Their motivation is to understand Earth’s climate. Period. If they wanted to save the world, they could have joined the frigging EPA or Greenpeace. If they wanted money, they could have used their skills to much greater advantage on Wall Street and bought the frigging world. There is simply no way that they would sacrifice that goal for any political expedient. It would mean dedicating their lives to a lie.
And what is more, even in the unlikely event that you could get some sort of confirmation bias among a couple of thousand climate scientists and a further 20000-30000 physicists and chemists, etc., why can’t the dissenting scientists find evidence themselves that supports their position? These people aren’t dumb. You know they are looking for evidence. Why can’t they find any?
LC’09 is a deeply flawed paper. The fact that they the wrong calculation of sensitivity is good for about a factor of 2 alone. And it is difficult to see how the signal they are cliaming persists over different intervals that are not cherry picked. Now maybe they will surprise us. But Walter, there’s a mountain of evidence weighing in against it.
42, Jonathan Gilligan:
“Suggestions for double blind reviewing often fail to recognize how easy it often is to infer the authors from a blinded manuscript.”
I don’t see why that’s a problem. In the worst case, the reviewer guesses the author and they’re in the same boat as a normal, single-blind review. Moreover, as Harold Brooks said, sometimes they guess wrong.
I like double-blind reviewing — having the author’s name there is a distraction (it’s hard not to think about) and so not having it allows you to focus on the science. So what if you guess who it might be by the time you get to the end of the paper? Do you go back and change your comments and re-evaluate your thinking? That would be a conscious decision to be unscientific.
“People don’t write papers in a vacuum; a new paper connects to an author’s previous work and connecting a new paper to that work is often not so hard, especially in those cases where an author has a high profile (which is exactly the case where many folks want doubly-blinded review to protect a controversial author from prejudice by referees).”
That’s not why I favour it. I favour it for the same reason that the prosecution isn’t allowed to bring up past convictions during a trial to assess a defendant’s guilt or innocence — the case (or, in this case, paper) has to stand on its own merits. We recognise that it’s hard for a jury to objectively decide whether the person is guilty in this case if they’ve been told that the person has committed similar crimes in the past, so we should recognise that it’s hard to decide if this paper is good science or not if the person who wrote it is (in)famous in that particular field.
It cuts both ways — we want to filter out poor papers by well-respected people just as much as we want to ensure good papers by controversial authors get published. A reviewer might find it hard to give honest critical feedback to a well-regarded figure in their field.
“Double-blind review might help some problems, but it would be no panacea”
Nothing ever is, so being a panacea is an unreasonably high hurdle; simply being better than the current system should be sufficient. I’m surprised to see Gavin say that he’s never come across double-blind reviewing in the Earth Sciences; I thought it was pretty normal and uncontroversial.
John N-G:
I thought the part of your blog piece on problems in publication research was dead on. The thing I really can’t stand, being a sort of methods freak, is the way complex methods are short-changed. The reader who doesn’t have the methods is the reader that can’t truly judge validity. A couple of points however: (1) it seems that at least at Science and Nature, there are more papers providing lengthy supplemental material (doesn’t help those without electronic access though) and (2) many biology articles need length just as much as climate science articles do, as many are just as complex.
The link to the letter from the GRL Editor in Chief is broken. Do you have a valid one?
Sorry to post what may be off-topic, but a search of RealClimate has not turned up what I’m seeking, which is a rebuttal to a comment by David Rose in the Daily Mail saying that we are entering a new ice age, as evidenced by INCREASING Arctic ice. Is this the area vs. extent thing? Thanks.
“According to the US National Snow and Ice Data Centre in Colorado, Arctic summer sea ice has increased by 409,000 square miles, or 26 per cent, since 2007 – and even the most committed global warming activists do not dispute this.”
This is certainly off topic – but can someone help?
The UK and Europe are suffering a serious cold snap, as, I gather, is the USA and the south of Canada. (OK, it is warmer elsewhere, like central America and the Sahara, but for a British denialist that is quite irrelevant – if it is cold in London, that’s global AWG finished forever :-))
Europe suffered serious cold periods in the winter of 1947 and 1963, both of which I remember.
What is known about the global weather pattern for 1947 and 1963? Were they similar to 2009/2010?
These winters were before satellite measurements, etc, of course.
The denialists are wetting themselves with excitement at the present cold snap, (including the Lord Monkton clan).
My main complaint was that, because of space restrictions, articles in GRL (and Science and Nature) are unlikely to contain enough information to be reproduceable, or properly evaluated, or even useful
I recently downloaded a paper from Nature (4 pages) plus Supplementary Methods. Thought nothing of quickly printing the Supplementary section to read at home and discovered when I went to pick it up that it was 34 pages long.
Is that long enough?
BBC- “Climategate” is now being investigated by a “National Domestic Extremism Unit” (police unit).
“A police unit set up to support forces dealing with extremism in the UK is helping investigate the leaking of climate change data in Norfolk.”
“Now it has been revealed the force is getting help from the National Domestic Extremism Unit, based in Huntingdon.”
It’s probably best not to add comments about this here, so as not to hijack this thread. Jan. 11, 2010
http://news.bbc.co.uk/2/hi/uk_news/england/norfolk/8453117.stm
John Mayer …
Natural variation not affecting the trend. Weather happens.
You’ll never get him to believe it, so I suggest inviting him to a high-stakes p-oker game, instead. These are the kind of people that professional g-amblers and c-asinos prey on.
As an aside, last year’s summer minimum was *still* more than two sigmas from the average 1979-2000 …
And, twice this fall/early winter new minimum records for a stretch of dates has been set.
And, ice volume is steadily dropping.
The old sports canard, “just wait ’til next year!”, is probably useful in this context.
John Mayer @82: an answer can be found here.
Why bother rebutting nonsense? It only serves to switch the focus of the debate away from the core issues.
Ray Ladbury – you should consider what it means to use sea surface temperatures as forcing for climate models – how is that any different from using the day-to-day water vapor content of the atmosphere as a forcing in climate models? It’s a ridiculously backwards approach – SSTs vary seasonally and with ENSO, but any long-term trend is a response to forcing, not a forcing itself, within the climate model context:
Watch this site for one year and you’ll understand why driving climate models with SSTs is nonsense – it’s the other way around, as usual with Lindzen:
http://www.nhc.noaa.gov/aboutsst.shtml
SSTs are used as forcings in short-term weather models, but those rapidly diverge from reality in a week or two. Why on earth would Lindzen use models forced with SSTs unless he was simply out to spin the research for his personal agenda? Why would any rational journal not use “skeptical” reviewers to vet such research? Who cares what some flea-bitten fossil fuel blogs say about it? That’s like catering to the Flat Earth Society, isn’t it?
Are you really saying that GRL should cater to the fossil fuel peanut gallery on this? And what about the refusal to publish comments?
You’re looking at something a bit worse, I’m afraid – the wholesale breakdown of peer review at a leading journal is nothing to gloss over. Science has to have standards or it becomes nothing but propaganda – your defense of the GRL editors is thus unfortunately a bit misguided, in my opinion.
MapleLeaf, I’m sorry you’re being “threatened”, but if you don’t want to identify yourself out of some ingrown paranoia, then don’t post comments on websites – lock yourself in your house, close the curtains, and it’ll all be okay. Or maybe you think that this is the Soviet Union, and dissent will land you in the gulag, on a Siberian train detail? Things aren’t quite that bad, yet… so please, grow up. Reviewers are kept confidential so that they can speak freely without “causing bad feelings” (yes, scientists are an emotionally sensitive lot, and do tend to hold grudges, etc. The phrase, “prima donna” has even been bandied about…).
Peer review isn’t perfect, but it’s the best system anyone has come up with – and the problem with GRL is that they apparently didn’t follow the standard peer review steps. Why didn’t they?
Maybe the AGU itself needs to revise the following policy:
For AGU journals, authors are invited to give a list of proposed reviewers for their paper.
Ha ha ha! I like to pick my critics, too – let me give you a list to choose from, okay?
American science is in deep trouble, if policies like this are the norm.
The way internet standards are developed is something the scientific community could learn from. The first stage is an Internet Draft available for general review but not formally considered to be published. An I-D can migrate to a Request for Comment RFC with further review and editorial correction, before it becomes a standard. Unlike with academic papers, any interested party can comment on an I-D. An I-D is withdrawn from the I-D site, but an RFC is an archival publication, and not altered once published.
This process ensures wide checking before a document becomes a standard. The downside is that embarrassing errors may see the light of day in an I-D. An advantage is that you can get your ideas out quickly long before they have been formally reviewed, to discourage others from scooping you. And of course you are not relying on a selected and limited pool of reviewers.
I agree that revealing the reviewers’ names is not an option. However, publishing the text of the reviews in a case where there is some controversy about the process should be considered.
It’s informative to read all the examples here of how soft a ride contrarian papers are getting. The evidence for the big conspiracy points more at them than the other way — even without anyone stealing their emails.
Theo, yeah, they are certainly wetting themselves, while ignoring the unprecedented heat waves in southern AU (Melbourne was almost 100 degrees at the night time low temperature the other night).
Here in Colorado we had extremely low temperatures during that cold snap that covered the whole US (in the mid teens at the highest), but it was almost 60 today, will hit 60 tomorrow for sure. I was wearing no coat at all. In the middle of winter.
I tend to avoid these weather pattern discussions and tell denialists up front that I don’t believe they represent climate. It makes me look like I’m conceding the point, but I’m not really.
Jeffery @ 37 – Some things about the pine beetle and the destruction that sometimes are missed:
Due to our extensive forest fire fighting efforts, a lot of the pine trees are older then normal. If nature had taken its course, we probably would not be seeing this devastation that we do today.
Older trees, from what I have read, are generally considered to be poor CO2 uptakers. New growth absorbs more at a higher rate.
The CO2 being released from the dead trees is considered to be nuetral, as most of the carbon that is bonded within the woody structure would have been from “natural” sources as opposed to GHG emmisions.
The bigger problem lies with the bacteria and fungus that emit methane as they consume the trees.
On a bit of a pendantic note, the pine beetle does not kill the trees directly. They harbour a bacteria that actualy does the dirty deed by growing dense enough to stop the upwelling of water and nutriants from the roots.
On a convergence note, on Pielke Jr.’s site today, there is a blog about how the tax regime in the USA is diverting much needed wood product from pulp and fibre board mills to bio-fuel. My solution, suspend the softwood lumber tarrif and let Canada ship all of the needed product from these trees!
Theo Hopkins #83: the low temps in northern countries are accompanied by unusually high temperatures in the Arctic and Greenland, with a peak anomaly of 8K.
John Mayer #82: you can find data on the same page at NSIDC on sea ice. Yes, 2007 was a very low year for summer sea ice extent, but we are still below the historical average and most of 2009 was much closer to the 2007 figures than this isolated period. In any case sea ice extent is not the whole picture: the longer-term concern is the decline in multi-seasonal ice.
Theo #83 said, “The denialists are wetting themselves with excitement at the present cold snap, (including the Lord Monkton clan).”
Errr, quickly, not all areas on Earth are currently getting colder if only the contrarians would take the time to read it… especially Canada, Alaska and the Mediterranean which has currently been warmer than average.
Meanwhile, the WMO reports that parts of the southern hemisphere are currently having record highs (remember, it is summer there!). Bur this doesn’t mean much as it is averages, averages, averages that count now and pretty much have since the defintion of climate change started even in 1824 with Fourier.
NOAA also reports that the ocean temps are some of the warmest ever recorded (not that this matters much for 30 year averages), but it shows that it is the 30 year averages that mainstream scientists are measuring not natural variations such as the warmest oceans right now, the warmest parts of the globe which are breaking records or the coldest records.
2009 was also the hottest year in history in most parts of South Asia and Central Africa. This doesn’t really mean anything for human-caused climate change except to show that were getting record highs along with other parts of the Earth getting record lows…and were still getting global warming which is averages, averages, averages.
This also means you shouldn’t concentrate on individual years or extremes which only ignorant people dealing with climate change would do.
Secondly, the NAO (North Atlantic Oscillation) is (was as of last week) in a sharp negative phase which is weakening the polar jet stream (westerly winds and storm tracks) and letting the polar air come far south in parts of the USA and parts of Europe.
Forgive me for absolutely scientifically butchering the following definition, but it gives a general feel in case someone has no idea). For understanding the NAO a little, think of kind of-like-an El Nino-like effect of two intertwined semi-permanent counterpoised high and low pressure areas, except it is north and south near Iceland and the Azores instead of East West between the coast of Australia and the coast of South America.
Certain people are on purposely ignoring this highly visible and understandable information although it is publicly available. It is unbelievable that people are falling for this.
http://www.noaanews.noaa.gov/stories2009/20090916_globalstats.html
http://www.cpc.noaa.gov/products/precip/CWlink/daily_ao_index/ao_index.html
http://en.wikipedia.org/wiki/North_atlantic_oscillation
WMO global 2009 temperature analysis:
http://www.wmo.int/pages/mediacentre/press_releases/pr_869_en.html
Visbeck, 2001
http://www.pnas.org/content/98/23/12876.full
83 Theo Hopkins says:
The UK and Europe are suffering a serious cold snap, as, I gather, is the USA and the south of Canada.
Well, except for the parts of the USA and Canada that are west of the Mississippi….
John Mayer #82: David Rose of the Daily mail is hoping you don’t go and look at what the National Snow and Ice Centre actually says about the last Arctic sea ice minimum extent. Yes it is above the 2007 record low, but it is dead on the trend line for a continuing decline of 11% per decade.
And as for the current extent:
“December 2009 had the fourth-lowest average ice extent for the month since the beginning of satellite records, falling just above the extent for 2007. The linear rate of decline for December [since 1979] is now 3.3% per decade.”
Anyone with an eloquent tongue care to respond to Terry Arthur’s letter on http://www.the-actuary.org.uk/872456 ? Someone must have a stock reply to this widespread misinformation?
John Mayer (#82)
No this is a cherry-picking your reference point thing, similar to the temperatures have been cooling since 1998 arguement.
2007 is the lowest recorded sea ice extent and things have “improved” since then – but predicting a new ice age from a 2 year trend ain’t valid.
2008 and 2009 September ice extent were still well below the long term average.
Can someone at RC do a short post on the present “cold” snap? OK, off topic :-(
If this could be related to the winters of ’47 (much worse, and the war-torn infra-structure of Europe nearly collapsed) and ’63 (not so bad but fresh in many peoples’ memories) it would help this side of the ditch. Indeed, was there a big cold snap in USA those two years?
John Mayer “Is this the area vs. extent thing? Thanks.”
No.
It’s just lying by omission.
A record low will ALWAYS have a subsequent higher low.
Theo colorfully said: “The denialists are wetting themselves with excitement at the present cold snap” — Well, isn’t that the proverbial way short-term thinkers keep their pants warm?
:)