I said on Friday that I didn’t think that Lindzen and Choi (2009) was obviously nonsense. Well, a number of people have disagreed with me, and in doing so, have presented some of the back story on the how the response was handled. I think this deserves to be more widely known in the hope that it will generate some discussion in the community for how such situations might be dealt with in the future.
From Chris O’Dell:
Given the large number of comments on the peer-review process in general and in the LC09 case in particular, it is probably worthwhile to give a bit more backstory to our Trenberth et al. paper. On my first reading of LC09, I was quite amazed and thought if the results were true, it would be incredible (and, in fact, a good thing!) and hence warranted independent checking. Very simple attempts to reproduce the LC09 numbers simply didn’t work out and revealed some flaws in their process. To find out more, I contacted Dr. Takmeng Wong at NASA Langley, a member of the CERES and ERBE science teams (and major player in the ERBE data set) and found out to my surprise that no one on these teams was a reviewer of LC09. Dr. Wong was doing his own verification of LC09 and so we decided to team up.
After some further checking, I came across a paper very similar to LC09 but written 3 years earlier – Forster & Gregory (2006) , hereafter FG06. FG06, however, came to essentially opposite conclusions from LC09, namely that the data implied an overall positive feedback to the earth’s climate system, though the results were somewhat uncertain for various reasons as described in the paper (they attempted a proper error analysis). The big question of course was, how is it that LC09 did not even bother to reference FG06, let alone explain the major differences in their results? Maybe Lindzen & Choi didn’t know about the existence of FG06, but certainly at least one reviewer should have. And if they also didn’t, well then, a very poor choice of reviewers was made.
This became clear when Dr. Wong presented a joint analysis he & I made at the CERES science team meeting held in Fort Collins, Colorado in November. At this meeting, Drs. Trenberth and Fasullo approached us and said they had done much the same thing as we had, and had already submitted a paper to GRL, specifically a comment paper on LC09. This comment was rejected out of hand by GRL, with essentially no reason given. With some more inquiry, it was discovered that:
- The reviews of LC09 were “extremely favorable”
- GRL doesn’t like comments and is thinking of doing away with them altogether.
- GRL wouldn’t accept comments on LC09 (and certainly not multiple comments), and instead it was recommended that the four of us submit a stand-alone paper rather than a comment on LC09.
We all felt strongly that we simply wanted to publish a comment directly on LC09, but gave in to GRL and submitted a stand-alone paper. This is why, for instance, LC09 is not directly referenced in our paper abstract. The implication of statement (1) above is that LC09 basically skated through the peer-review process unchanged, and the selected reviewers had no problems with the paper. This, and for GRL to summarily reject all comments on LC09 appears extremely sketchy.
In my opinion, there is a case to be made on the peer-review process being flawed, at least for certain papers. Many commenters say the system isn’t perfect, but it in general works. I would counter that it certainly could be better. For AGU journals, authors are invited to give a list of proposed reviewers for their paper. When the editor is lazy or tight on time or whatever, they may just use the suggested reviewers, whether or not those reviewers are appropriate for the paper in question. Also, when a comment on a paper is submitted, the comment goes to the editor that accepted the original paper – a clear conflict of interest.
So yes, the system may work most of the time, but LC09 is a clear example that it doesn’t work all of the time. I’m not saying LC09 should have been rejected or wasn’t ultimately worthy of publication, but reviewers should have required major modifications before it was accepted for publication.
To me this raises a number of questions. Why are the editors at GRL apparently not following the published editorial policy on comments? The current policy might not be ideal, and perhaps should be changed, but surely not by fiat, and surely not without announcing that policy change? This particular example has ended up divorcing the response from the original paper and clearly makes it harder to follow the development of this analysis in the literature. Additionally, in cases where there appears to have been lapses in peer-review (for whatever reason), is there not an argument for having a different editor deal with the comment/response? Perhaps a new online journal which independently publishes peer-reviewed comments and responses is called for?
Everyone involved in the peer-review process knows full well the difficulty in finding suitable reviewers who have the time and inclination to do a good review. The pressures on editors both to be seen to be fair, and to actually be fair to the authors (and the readers!) are strong, and occasionally things will go wrong. The measure of such a system is not whether it is perfect, but whether it deals appropriately and quickly with problems when they (inevitably) arise.
NB. Comments on how to improve the situation are welcome, but please avoid simply criticising papers that you personally think shouldn’t have been published in the form they were.
Ike: “MapleLeaf, I’m sorry you’re being “threatened”, but if you don’t want to identify yourself out of some ingrown paranoia, then don’t post comments on websites”
Nope, that’s completely fair to post on websites with a pseudonym.
What’s not fair is to do that and complain about other people’s anonymity.
John Mayer @82 “According to the US National Snow and Ice Data Centre in Colorado, Arctic summer sea ice has increased by 409,000 square miles, or 26 per cent, since 2007 – and even the most committed global warming activists do not dispute this.”
The short answer is that the Daily Mail is cherry picking. 2007 was a year of massive ice loss, down about 30% from the 2006 (strangely, the skeptics don’t consider that evidence of global warming). I think the figure that best clarifies what they’re doing is here: http://nsidc.org/images/arcticseaicenews/20091005_Figure3_thumb.png. It’s akin to the argument that there has been no global warming since 1998.
There’s some discussion of this in the comments here https://www.realclimate.org/?comments_popup=2625 (search on Arctic ice, and multi-year ice).
Ike, I agree that using SST as a forcing is a bit bass-ackward, but in the case of LC’09, where what they are really trying to do is demonstrate a particular negative feedback, it makes some sense. I think it is all to easy–especially wrt a subject that arouses passions to resort to allegations of incompetence or fraud. Well, in this case, we know with certainty that Lindzen is not incompetent. And misconduct is not an allegation to be thrown around lightly. Any decent scientist knows if they fudge the data they will get caught. The ease with which climate scientists discovered the serious flaws of this piece is an indication of the ruthless efficiency of the process. Lindzen and Choi are both smart enough and experienced enough to know they couldn’t fool the community. A GRL faces a much tougher crowd than that found at WUWT.
At most, we might think that L&C were perhaps overeager. They saw a hint of what they were looking for in the data and teased (or toutured) it out. That they could have been more diligent in quality checks is beyond doubt. I also think that the piece could have benefited from a third set of critical eyes. However, I see nothing that rises to a level even approaching misconduct, nor do I see any advantage in making such an allegation.
Lindzen and Choi deserve credit for publishing and putting their ideas in front of a critical and expert audience. We all count on our peers to correct us if we are going off into the weeds. All scientists make mistakes, and it is up to the collective of scientists to correct them. There are far worse things than being wrong. One of them is not publishing. And worse still is presenting a complicated analysis to a credulous lay audience who have no hope of spotting the errors. Give the authors credit for making the attempt and give them the opportunity to correct the errors in the analysis if they can. I am sure they (and the reviewers) will be much more circumspect with any new analysis.
One additional criticism might be made of the authors–the decision to publish in GRL. GRL is really intended for short, straightforward, newsworthy developments. As a Letters publication, peer review will always be more cursory than it will be for more deliberate publications. As L&C were arguing for some rather revolutionary ideas, it would have been more appropriate to publish in a journal where the peer-review process is more rigorous. Perhaps the authors felt that they needed to get the paper published before Copenhagen. If so, they did themselves and the community a disservice.
Sorry to be OT, but I’d probably forget if I waited until the next appropriate thread.
http://news.slashdot.org/story/10/01/11/2054237/Another-Crumbling-Reactor-Springs-a-Tritium-Leak?art_pos=8
WM: Lindzen, as you say, is the published climate scientist in the arena, not Hank and Ray and the gang.
BPL: Ray is not only published in the field, he has done famous work in the field. It was Forget and Pierrehumbert 1997 that finally nailed down the fact of the outer edge of the ecosphere being way out at 1.7-2.4 AUs, not close in as Hart and even Kasting had thought. Ray has a long, long list of publications on planetary astronomy and climatology. You know not whereof you speak.
[Response: Thanks for the advert, Bart, but I think that the “Ray” being referred to there was Ray Ladbury, not me; I adopted raypierre as a nom de plume here to avoid confusion of that sort, most particularly with Ray Bradley. –raypierre]
WM: Lindzen is predictable and is looking for results that show less warming, just as the AGWers predictably look for results that show the opposite.
BPL: Psychological projection. Professional scientists do no such thing. They may have a hypothesis in mind, but they look at ALL the evidence, WHATEVER it says. They don’t set out to prove their point by ignoring anything to the contrary.
You sound like a creationist who says evolutionary biologists do all their research “to prove evolution,” and are therefore too biased to trust.
On cold smaps – and the end of global warming.
One of the great wonders of the web is when one hits the wrong button.
Last year there was a cool snap in UK.
Lord Monkton,our potty peer, wrote that this disproved AWG.
AT the time, I was looking up the present temperature for Launceston, Cornwall, England, where I live, on the Met Office site. Indeed it was an unusually cold -4C.
But then I accidentally hit the button and got Launceston, Tasmania, Australia. Just so happened that Launceston, Tasmania, had recorded its highest ever temperature the day before.
dcomerf,
OK, this dude cites WUWT and the OISM petition. That is self-refuting. My question would be why would you pick a nonscientist, undegreed TV weatherman or your climate information over actively publishing climate scientists?
And on OISM:
http://www.sourcewatch.org/index.php?title=Oregon_Institute_of_Science_and_Medicine
Nuff said.
re: 98
The current cold snap is notable basically in how far South the cold went.
Here in the middle we got down to around 0 a couple of nights, but that’s nowhere near our record cold. (-21F)
20 years ago, on Christmas Eve, I remember bundling the kids in blankets for the trip to the family farm because we were facing -15F temps and -45F wind chills. And 20 years before that, my room mate and I were driven from our house for a month because the cold (-20F) had frozen the ground 4 ft deep and snapped about some megaboss percent of the city’s water mains and pipes.
0 is cold in the face of recent years, but it’s more of a case of cold air going where cold air doesn’t usually go than exceptionally cold air generally. It’s a disruption in wind patterns.
Also on the OISM:
http://www.youtube.com/watch?v=Py2XVILHUjQ
Looking back, I recommend Ike Solem’s comment above (11 January 2010 at 1:31 AM) and examples about how important it is (or should be!) for a journal to facilitate finding comments and subsequent references to papers.
This is fundamental to library research AKA “how to look things up” — and how a journal or ‘review letters’ publication can make this easier or harder for people to do.
> … let’s see, for example, does Pielke Sr. still cite Lyman on the cooling of the
> oceans, even though it was retracted?
>
> Well, gosh, yes he does!
Zing!
@dcomerf: “96.Anyone with an eloquent tongue care to respond to Terry Arthur’s letter on http://www.the-actuary.org.uk/872456 ? Someone must have a stock reply to this widespread misinformation?”
There he mentions http://www.petitionproject.org claiming that “31,486 American scientists have signed this petition, including 9,029 with PhDs”
Sounds big, right? It’s tiny. For example they break out their signatures by qualifications, in which they claim 693 mathematicians. OK the AMS combined membership is 32,000. OK there will be some membership in AMS by people who describe themselves as qualified in other fields (e.g. physics, etc.) but I suspect not all that much.They claim 3,128 chemists, but the membership of the American Chemical Society is about 160,000. Yes, again there could be some overlaps, but in both these cases the rate of signing their denialist petition is about 2%.
If the actuaries there are any good, they shouldn’t need too much help figuring out how strong the denialist case is when that sort of petition subscription is a major exhibit in the case.
I found some of the RC story fairly shocking (e.g., “This comment was rejected out of hand by GRL, with essentially no reason given.”). Does GRL have some apologizing to do? I don’t think this is complicated:
* Reviewers should remain anonymous — always.
* Comments should NEVER be rejected by editorial fiat: Comments are part of the process of science and should be accepted if they pass peer review.
* There is no basis for rejecting any manuscript because it is critical of a previously published one.
* Authors should NEVER be allowed to suggest reviewers (which idiot dreamed that one up?).
* Editors should pull their fingers out and find appropriate reviewers — or resign.
* Editors should follow published editorial policy — or resign.
* Editors hold enormous power but that comes with enormous responsibility.
* Peer review is the least bad of all the alternatives.
What did I miss?
Jim Bouldin (#81): Here’s an updated link to the letter in EOS (no subscription required) from the then-editor-in-chief of GRL explaining its purpose and philosophy:
http://www.agu.org/pubs/pdf/Editorial_GRL.pdf
Jinchi (#84) – Maybe it’s just me, but if a paper is 4 pages long and has 34 pages of supplemetary material, I’d much rather read a coherently-written 38-page paper than constantly skip back and forth between two separate documents. In my experience, “supplementary information” is more often essential than supplementary and should be integrated into the paper.
Fox News gets it wrong. This from the Washington Monthly:
January 12, 2010
QUOTE OF THE DAY…. The British Daily Mail ran a report yesterday with the headline, “Could we be in for 30 years of global COOLING?” The piece told readers, “According to the U.S. National Snow and Ice Data Centre in Colorado, the warming of the Earth since 1900 is due to natural oceanic cycles, and not man-made greenhouse gases.”
It led Fox News to report, “30 Years of Global Cooling Are Coming, Leading Scientist Says.”
There are, of course, two small problems. First, the National Snow and Ice Data Center said no such thing. The director of the NSIDC said, “This is completely false. NSIDC has never made such a statement and we were never contacted by anyone from the Daily Mail.”
Second, the Fox News report cites the research of IPCC scientist Mojib Latif, one of the world’s leading climate modelers. The story completely mischaracterizes his work, and gets the story largely backwards.
Latif told Dr. Joseph Romm:
“I don’t know what to do. They just make these things up.”
Yes, they do. And as long as there are news consumers who prefer the alternative universe these outlets provide, they’ll keep making these things up.
http://www.washingtonmonthly.com/
interesting comments there.
I think GRL’s always been poorer than average for peer review, and that should be borne in mind.
Other than that, I do have a suggestion generally:
Recently, checklists have become an issue for hospitals/surgery. The analogy made is that pilots need checklists to prevent crashes.
Peer reviewed journals should work up a checklist for peer reviewers – steps that you ALWAYS take when reviewing a submission.
That eliminates the carelessness and oversight problem. The problem of bias in finessing peer review, e.g. Climate Research, or the worse problem of being a fraudulent publication, e.g.Energy and Environment, at least then become your main problems and are easier to set out.
Checklists:
http://www.cwunbound.org/2009/01/surgeon-hospital-checklist-pilots-preflight.html
Melty, Regarding your points
* Reviewers should remain anonymous — always.
Agreed.
* Comments should NEVER be rejected by editorial fiat: Comments are part of the process of science and should be accepted if they pass peer review.
However if you get 10 pages of comments on a 4-page letter, it kind of clutters up the works. Comments require a response by the original author and so on. In part this is just a limitation of the “Letters” format.
* There is no basis for rejecting any manuscript because it is critical of a previously published one.
Agreed.
* Authors should NEVER be allowed to suggest reviewers (which idiot dreamed that one up?).
OK, here’s the thing. In a broad field, there may not be very many potential reviewers who have expertise in a particular subfield. This often leads to a few reviewers being inundated with each others papers over and over. An author who is applying a new tecnhique might want to recommend a reviewer who is expert in that technique.
Yes, there is the potential for abuse, but it’s not as idiotic as you make out.
* Editors should pull their fingers out and find appropriate reviewers — or resign.
Not always easy–particularly with 2 week turn-around.
* Editors should follow published editorial policy — or resign.
Agreed, but there will always be situations the policy doesn’t cover.
* Editors hold enormous power but that comes with enormous responsibility.
Yes they have enourmous responsibilities, but very little power–especially if they are doing their job right.
* Peer review is the least bad of all the alternatives.
Agreed.
However, you are deluding yourself if you think things are as clear and easy as you imply.
John N-G says:
In my experience, “supplementary information” is more often essential than supplementary and should be integrated into the paper.
I agree. Often times I find the methodological details in there that I’m looking for. I never cease to be amazed at how crucial methods descriptions are omitted or castrated in papers. You can’t really evaluate them. Waste of everyone’s time and resources.
Not to mention the uselessness to those with only a paper copy.
Ray Ladbury says:
Not always easy–particularly with 2 week turn-around.
Then change the rush-to-publication-at-any-cost policy for godsake.
#82 John Mayer, you must be reading the same denialist sites as others, ’cause I debunked that Daily Mail article yesterday on another site I frequent. Everyone loves to take Mojib Latif out of context and misstate what he says. Here’s him complaining about the Daily Mail incident: http://www.guardian.co.uk/environment/2010/jan/11/climate-change-global-warming-mojib-latif
He is no stranger to theses types of intentional misinterpretations: http://www.youtube.com/watch?v=khikoh3sJg8
Permanent link to the Washington Monthly article on Latif that Catman306 describes above (since WM changes fast):
http://www.washingtonmonthly.com/archives/individual/2010_01/021879.php
Comments there include this observation:
—- The quotes attributed to “U.S. National Snow and Ice Data Centre” have now been replaced by “some scientists”—-
Jim Bouldin, the letters are not meant to be full-on papers.
I do not think that what is being done requires changing. All that needs to change is the idea that gentlemen will be gentlemen when it comes to climate studies (because there’s a lot of money to be lost and therefore a lot of pressure). Therefore the what would be ACCEPTED as a GRL as opposed to a full paper needs to be tightened up with that in mind.
But not how a full paper or a letter is reviewed.
In this case, the expectation of the paper (AGW isn’t real, here’s an alternative that gets us off the hook) is far too extraordinary for a mere letter.
If this had been written in the 1950’s-60’s, then probably would: there was a lot less evidence and testing on whether GW is anthropogenic in origin and a paper now purporting to throw it all away is an extraordinary claim requiring extraordinary proof.
Something this paper wasn’t put through the mill to ascertain.
If editors had so much power, why did three have to resign because they couldn’t get a clarification of why an earlier (terrible) paper was printed?
Recommended: Joe Romm’s piece, which the Washington Monthly points to, about the original Latif figure and interpretation.
http://climateprogress.org/2010/01/11/foxnews-wattsupwiththat-climatedepot-daily-mail-article-on-global-cooling-mojib-latif/
He really did a good job on this one:
—-excerpt follows—–
“What they mean is what the lead author, Dr. Noel Keenlyside, wrote me [in 2008] when I asked for a clarification:
Thus, based on our results we don’t expect an increase in the mean temperature of the next decade (2005-2015).
They are predicting no increase in average temperature of the “next decade” (2005 to 2015) over the previous decade, which, for them, is 2000 to 2010! And that’s in fact precisely what the figure shows — that the 10-year mean global temperature centered around 2010 is the roughly the same as the mean global temperature centered around 2005.
@101 “What’s not fair is to do that and complain about other people’s anonymity.”
Point taken. Thanks “fed up”.
That said, one of the many factors which compel me to do a decent job when I review a paper is that I have never assumed that I would remain anonymous, especially if I happened to screw up badly. I operate on the assumption that if I screw up, then the editor will hold me accountable, and my reputation will take a hit. I do not wish for that to happen, so I do the very best that I can, and do not review papers which I am not qualified to critique, or if I have a conflict of interest.
What ‘m trying to say is that I personally do not believe that anonymity when reviewing papers is a fundamental right, especially if one compromises the science or someone’s work in a big way. There are also journals out there which use an open review process quite effectively.
@melty: “* Authors should NEVER be allowed to suggest reviewers (which idiot dreamed that one up?).”
No, asking the authors to suggest referees is not a bad idea, editors just have to avoid using it as a crutch.
There are lots of papers that come in where it can honestly be hard to figure out who can review it well from among the pool of referees for a journal. There’s usually nothing wrong with the editor having the option of choosing one referee from among the authors’ suggestions.
It’s where the other two referees come from that makes the difference.
Yes, we need RC to do a post on this current cold snap, so all I have to do it give the link when I’m blogging elsewhere.
Right now I’m using the links and info Andreas provided — see https://www.realclimate.org/index.php/archives/2010/01/unforced-variations-2/comment-page-15/#comment-153894 — to address the denialist sites all over the web ranting we’re in a global cooling period & GW has been definitively disproved. See: http://www.dailymail.co.uk/news/article-1242202/Could-30-years-global-COOLING.html#addComment
It doesn’t hurt to again (and again and again and again) draw the distinction between weather and climate, and between local & regional weather (and even climate) and global climate and global warming. The reason I even asked about whether there were also areas right now warmer than usual, is bec all I get here is local/regional and perhaps national weather…..not world weather. Before I got that reply from Andreas, I had been thinking 1st law of thermodynamics (it must be hotter elsewhere), also short-term downs (& ups) during a longer-term upward trend, solar minimum irradiance, perhaps the thermohaline slowdown or stoppage, etc., but “strongly negative phase of the arctic oscillation” also works. Enquiring minds need to know. And I’m impressed that the TV weather news programs I’ve been watching have been referring to this cold snap as an “ARCTIC BLAST,” which at the least indicates this cold snap is coming from up north (for us north latitude people), with the implication at least that is not due to a general global cooling. I remember when a certain weatherman (with a relative in some biz) used to deny GW, and would have used this cold snap to deny it.
There are new people tuning in all the time to the global warming issue, whenever the weather becomes the front page story or gets their attention. They need to learn some basics about GW, just like a new batch of school children are entering the educational system each year. At least some of them will be amenable to bona fide science on the topic.
Hank Roberts says: 12 January 2010 at 1:50 PM
At Deep Climate David Rose puts in an appearance to defend his presentation:
http://deepclimate.org/2010/01/11/mojib-latif-slams-daily-mail/#comment-1996
He claims to be outraged by “intellectual bullying”, but if true he’s strangely selective about rising to the defense of hapless victims. So far McIntyre has been the only beneficiary of his ire, even while we see myriad baseless charges leveled against actual practicing climate scientists.
Jim Bouldin and John,
Having been on both sides of these issues, I think there are a couple of things going on. First, LC’09 was not an appropriate candidate for a “Letters” format. Even if their analysis had been correct, it had a number of novel components that should have been fleshed out, and they were making some rather astounding claims. However, there are some results that are pretty straightforward and are “newsworthy” enough that the community would benefit from rapid publication. I think the question is whether there are enough such results to sustain a dedicated publication like GRL.
Also, in regard to supplementary material:
In some particularly broad fields, it can be difficult to define the technical savvy of the audience. It may be that techniques introduced in a paper are quite familiar to the author and a subset of the community, but unheard of in the rest of the community. In that case, it can be a judgment call whether to treat the novel techniques in the body of the text or in a supplementary section. The latter strategy often leads to a more readable paper as it allows a narrative flow to be maintained.
Ray: I’m firmly on your side but I have to disagree on these points:
* if a 10-page comment is *required* to rebut a 4-page letter, so what? An editor or reviewer can suggest shortening it but the author is free to argue that all the material is necessary to make the point. I’m not aware that comments *require* a response from the authors of the original (or are you discussing specific GRL policies?).
* “This often leads to a few reviewers being inundated” …I am one of those inundated reviewers. Sure it’s tough work and stretches my schedule to breaking point (ask my wife) — but s/one has to do it. Peer review is an under-rated part of our profession (but we knew this already). So I still contend that allowing authors to suggest reviewers is a really bad idea.
* a 2-week turn-around? Isn’t this rushing things just a tad? It takes me 6-12 months to get from first submission to print but I’m not complaining. Let’s not go for the 24-hour news cycle model of science publishing — please!
* well according to Chris O’Dell’s post, they can block submission of a relevant comment submission prior to peer review. That is not acceptable.
Btw, I’m hardly naive (deluded was a little OTT, no?). I publish and review regularly , and I’m on the editorial board of a science journal (not climatology). Oh — and my first publication was a comment.
Newsflash: Sen. Lisa Murkowski of Alaska is launching a potentially devastating attack on the Clean Air Act. Majority Leader Harry Reid has granted her a vote for January 20 that would block the EPA from regulating greenhouse gas emissions from coal plants and other polluters in 2010.
Signatures needed on petition:
http://act.credoaction.com/campaign/cleanairactvote/?r=5161&id=7318-2354132-WrCfMvx
126: So when I submit a ms it’s OK I for me to simply suggest all my friends (aka colleagues who are sympathetic to my views)? No. I actually like tough manuscript reviews because it makes me think harder — and I have to demonstrate to the reviewer(s) and editor that my ms holds up. Please, guys, I’m a reasonable person (go look for melty on Dot Earth sometime).
dcomerf wrote: Anyone care to respond to Terry Arthur’s letter on http://www.the-actuary.org.uk/872456
Why do I feel we’re being pulled off topic?
“Of Mice and Models: Response to S. Green, December 2009
It is easy to forget that all sciences rely on models although they prefer to call them laws and theories. They are widely accepted until they are proved wrong, which most of them are, given time.”
Tell them most of us would be dead and hideously scarred of diseases, dead from a single cut on our fingers, having almost all our kids die in infancy, be dead by age 50, walking in our own poop in the streets, and murdering ourelves in Salem witch trials (my family was from there, oooops)without mainstream science to methodically work things out with journals and the written scientific process. Yeah, it has problems, but it works over time. There is no alternative to replace it except anarchy…and humans wisely gave that up for good reason.
Melty says, “So when I submit a ms it’s OK I for me to simply suggest all my friends (aka colleagues who are sympathetic to my views)? ”
Well, it would be OK if you were a moron who didn’t care about the quality of your work (and no, I don’t think you are). Moreover, what kind of “friend” allows you to publish crap? Remember the reviewer’s reputation is on the line here as well. Oftentimes the only people who are really expert in a technique will be people who have published together in the past. That is one reason why you want multiple reviewers.
Ask yourself: If the choice is having a reviewer who is knowledgeable about the subject matter but who is a “friend” of the author or having a reviewer who knows bupkis about the subject matter, which do you take?
Lynn, you can collect enough links on the weather stuff; it’s not climate (unless someone wants to claim the extremely rare high pressure in the Arctic will persist).
Besides, you can refute it thus:
http://www.weatherzone.com.au/
http://www.weatherzone.com.au/news/victoria-heat-ends-with-sweaty-night/13577
(PS, if you use those, a way to check your readers’ level of sense is to see if they think this proves global cooling)
“…even after sunset, temperatures refused to drop below 40. Melbourne finally dropped below 40 around 8pm, was still above 35 after midnight and did not fall below 30.6 degrees all night.”
Ray (#127) – It would be great if supplementary material would be restricted to that purpose: a convenient repository for (mostly methodological) information that readers would otherwise have to dig for in other referenced papers.
To demonstrate that this is not the norm, I decided randomly to check out the most recent climate-related paper in Nature Geosciences: it turns out to be http://www.nature.com/ngeo/journal/v3/n1/abs/ngeo706.html Lunt et al. 2010. Gavin Schmidt turns out to be a co-author, so maybe we can get an author’s perspective too!
The paper is five pages long. The supplementary information (SI) is 18 pages doublespaced, which would probably come out to about four pages in a Nature-format journal.
The caption to Figure 1 of the article (there are three figures in the article and four in the SI) is:
“Figure 1: Charney sensitivity and Earth system sensitivity. a, Cs = delta Tc (C). b, ESS (C) calculated from Supplementary Equations S16 and S17.”
This illustrates my objection to the format of short papers plus supplementary information perfectly. According to the text of the paper, the term “Earth system sensitivity” is introduced for the first time here: “We term this temperature response the ‘Earth system sensitivity'” (no references cited). Furthermore, equations S16 and S17 are derived in the Supplementary Information, not obtained from another cited reference. So S16 and S17 are apparently novel equations, fundamentally part of this work itself and inseparable from its results.
It seems to me that any logically defensible presentation of this material would have Figure 1 and equations S16 and S17 in the same document, not two separate documents. The choice to include S16 and S17 in the supplementary information rather than the main text was made, I would hazard, not to improve readability but rather to meet the paper length requirements for publishing in a prestigious journal.
Gavin, please correct me if I’m wrong. For better readability and scientific presentation, absent Nature-imposed restrictions, would you prefer to have arranged your text differently? Do you regard S16 and S17 as important mathematical statements essential to your paper or as merely supplemental information whose absence would not weaken your analysis?
Interesting paper, by the way.
[Response: Thanks! (I think) The concept of Earth System Sensitivity is not difficult to grasp – it’s what you get when you allow everything to vary and come into equilibrium rather than just the subset of processes that were included in GCMs when Charney wrote his 1979 report. Estimating this from a finite set of experiments with an imperfect model combined with uncertain Pliocene boundary conditions is a little trickier and can get technical. How a paper is organised is ultimately concerned with what it is you want to communicate though. In this case, I think the concept and the approach are more important than the technical details and that is reflected in the organisation of the paper. The SI was part of the peer review and we had a lot of issues that we went back and forth on with the reviewers with respect to those details and we ended putting in more explanation and alternatives there to satisfy everyone without it making any substantial difference to what we considered the bottom line result. If the paper were written as a single narrative, I don’t think we would have included so much detail because it would have unbalanced the presentation – but I don’t really know, because the paper as published is a fair bit different from the initial paper we submitted. The process of getting a paper through Nature or Science or Nature Geoscience or PNAS is very much a factor in determining what the end product looks like. – gavin]
Hi Ray, You wrote “If the choice is having a reviewer who is knowledgeable about the subject matter but who is a “friend” of the author or having a reviewer who knows bupkis about the subject matter, which do you take?”
Your question is a good one but here’s the answer: THAT is why we have editors. The good ones spend time on finding appropriate reviewers. If my paper is reviewed by s/one who knows bupkis (zip?) about my field, it’s easy (oh so easy) to shoot the review down and hence increase the chances of my ms being accepted. Of course, I would prefer a reviewer who knew something — criticism helps to strengthen the ms.
On your other question: “what kind of “friend” allows you to publish crap?” You are missing that fact that there is more than one valid point of view, or approach, to many problems. In my field, people tend to push particular interpretation techniques or instrumentation types: most are useful to some extent but I’m not going to hope that I’m able to push my particular approach by having “friendly” reviews.
Call me an altruist — and bring on the tough reviews.
> merely supplemental information whose absence would
> not weaken your analysis
Wait, that amounts to a definition of “supplemental information” — is this a correct description?
I thought “supplemental information” is comparable to extended footnotes with figures — further information useful to anyone wanting support for the statements in the simple condensed article text.
CM #100:
Try that in freezing-cold weather. Oh, wait, that’s exactly what they are doing.
> Doug Bostrom says: 12 January 2010 at 2:25 PM
> At Deep Climate David Rose puts in an appearance to
> defend his presentation:
> http://deepclimate.org/2010/01/11/mojib-latif-slams-daily-mail/#comment-1996
Owie, that’s very sad.
“… without Steve’s brilliant work and this magnificent website, it could not have been written. May I also pay tribute to Ross McKitrick….
I am not a scientist, but an open minded investigative journalist. I have not written on climate before. …”
— David Rose, after an instant education at Climateaudit.
Reflecting a bit more on this I’m giving Lindzen and GRL some second thoughts.
I’ve been hesitant to give some of my views on the peer-review process, because the social sciences, especially anthropology, might be different from the physical sciences. Clifford Geertz commented on this distinction in his essay “Thick Description”: “Anthropology, or at least interpretive anthropology, is a science whose progress is marked less by a perfection of consensus than by a refinement of debate. What gets better is the precision with which we vex each other.”
The peer-review process in such a field, it seems to me, involves more subjective considerations; it pays to speak to “the old boys club” of a particular journal; and there are some tricks to getting an article accepted; it helps to know the publisher; at least he/she will make sure your ideological enemies are not the reviewers — a Marxist “critical school” reviewer would never tick-mark “accept for publication” a structural-functionalist framed paper, or even a symbolic interactionist framed paper, no matter how good; and if it’s an anthropology paper, be sure to include all the latest fadish jargon (a physical scientist actually did this trick — wrote a bogus anthropology paper imbuing it with plenty of postmodernist jargon and got it published, then revealed the hoax).
That’s not to say that mainly crud gets published in cultural anthropology (the above being a rare case), or that empirical information and data count for nothing, or that great ideas and analyses are constantly chucked overboard. I’ve found the peer-review process greatly improves my mediocre and even good papers.
What I’m thinking here is that in the physical sciences things are not completely nailed down either, and there is room for debate, though it is not as subjective/ideological as in the social sciences, and eventually the (as close as we can get) objective truth wins out and there is a movement toward “perfection of consensus.” And if this Lindzen/GRL type of thing had occurred in some other field, one that was not impacting our lives (at least not in ways that we people would be called on to act and mitigate), such as the debate between continental drift v. stationary continents, or big bang v. cyclical theory, it would be just as acrimonious within those narrow areas, and some scientists may privately grouse about some less worthy articles passing peer-review or their own being rejected, but no one outside the confines of those subfields would really care much about it. Science buffs might think, oh well, it will eventually be sorted out; meanwhile I’ll stick with Dr. XYZ’s theory.
For the past 2 decades I’ve had a fairly negative view of the skeptics and contrarians within the climate science community (who are now just a handful), especially when the media trot them out to create a balanced format (as if AGW were a matter of opinion rather than scientific fact), but now I’m not as negatively disposed toward either the contrarian scientists or the peer-review process flaws (whether they be accidental, structural, human bias, or special interest flaws). This is how science bumbles through to the truth, to the empirical, objective scientific facts. (And let the Piltdown Man type hoaxes be revealed.)
What I’m most opposed to is how the non-scientists are responding, the general public and policy-makers. If they can say that pro-AGW climate scientists are wrong, then they should also realize that the anti-AGW climate scientists could be wrong, and that we need to use Pascal’s wager in making our own decisions, apart from scientific standards. That’s the real problem here.
But since people are refusing to act sensibly to avoid possible great harm from AGW, and are assuming mitigation will cause great harm, without even giving it a try, the whole burden has fallen back on scientists and the peer-review process, making inhuman demands on them both.
And worst of all, even if climate scientists were perfect and knew everything, and even if the peer-review process were perfect, it wouldn’t do any good at all, it seems to me. The non-scientist denialists are simply not going to accept that AGW could be real and dangerous, no matter what. The last ones to “convert” and come on board were Pat Robertson and William F. Buckley, and I think there will be no more conversions — maybe just a trend nearly as slow as AGW itself in which denialists eventually die out, and not as many proportionately are born or inducted into such denialism. Maybe that will happen, but it will end up being much too little much too late.
I’m also thinking, what do the scientists know, maybe it already is much too little much too late — not that we should ever give up mitigating and making the total catastrophic disaster on into the future into a not-as-bad disaster.
Hank Roberts says: 12 January 2010 at 4:58 PM
And don’t lets forget Rose’s paean to McIntyre, “Sir, I salute you. Bravo!”
Mind you, this is the same McIntyre who has apparently organized some kind of Dewey decimal-like indexing system for a file of pilfered email obtained by hook or crook and is currently preoccupied not with science, instead is obsessively stirring the ashes of a long-dead article review process, babbling about “evidence” of a editor/reviewer plot against him.
Pretty sorry situation, when journalists attached to a paper with circulations of millions are besotted with crackpots.
Hi. The ‘Wheres The Data’ post of November was a good move, but given it’s importance I think that it deserves to be more obvious than the current link on the side menu. Do you guys have any plans to promote it to a top level tab?
RE #135, thanks, Hank. I still would like to see an RC post on the cold snap. And yes, the whole issue is to get people to understand the difference between weather and climate, and that one cold snap in some regions a global cooling trend make (as in one bee does not a hive make).
And it is also helpful (since many people can’t see the forest for the trees), if it can be shown that this cold snap is not just (or even) a brief dip in global average temp, but also due to the arctic blast, the weather pattern shifting from a more typical west-east thing to a north-south swing.
melty @132: So when I submit a ms it’s OK I for me to simply suggest all my friends (aka colleagues who are sympathetic to my views)? No. I actually like tough manuscript reviews because it makes me think harder — and I have to demonstrate to the reviewer(s) and editor that my ms holds up.
Let’s take a step back for a second and remember that we expect scientists to behave ethically in suggesting a list of reviewers for their work, just as we expect them not to doctor the data to fit their conclusions. When a journal asks authors to submit a list of potential reviewers, it’s because they are more likely than the editor to know of people who are experts with a specific data set or a numerical technique. I don’t know of any professional who simply suggests all his friends to get through peer review. Often I’ve never even met the reviewers I suggest. On the other hand, authors are under no obligation to recommend reviewers who are openly hostile, and the editor is not required to take their suggestions.
Authors should NEVER be allowed to suggest reviewers (which idiot dreamed that one up?)
That comment suggests that you think most scientists will cheat whenever possible. Honestly, it would be much easier to tweak results, throw out inconvenient data and modify model output in ways that would never be spotted in peer review. But science is self correcting. If you’re wrong, nobody will be able to work with your results and all your hard work gets tossed into the garbage bin.
It just struck me, there’s a kind of beautiful, weirdly twisted mirror symmetry here, staring us in the face.
Woven into this thread dissecting the demanding peer review process are a few snippets regarding a journalist who is writing Dadaist syntheses of reality and fiction regarding climate change, at the behest of editors beholden to a billionaire climate change doubter. This journalist has declared his admiration and respect for a person who is inventing cabalistic conspiracies to explain why he has had so much trouble getting his climate change doubting research published in peer-reviewed journals.
Opposites, linked by McIntyre. It’s almost poetic.
Oh dear, so much to comment on!
First, stop with the cold snap stuff, or at least have the intellectual honesty to admit that using weather in a climate debate is, well, intellectually dishonest. During the late drought in Georgia (USA), we heard lots about how it was proof of AGW. When the drought broke, credibility for AGW went with it for many. Using a cold snap to “prove” anything related to either side of climate political debate is short-sighted and counter productive.
Second, I’m really shocked at the number of “Big Oil/Coal” conspiracy posts. If a journal can be “taken over” by oil industry interests it can be taken over by ANY interest. The only difference is who’s signing the check if the journal is up for bid in the first place. While meant as a slur for big corporations, it was really a slur against the folks who put out journals and do peer reviews.
While I consider myself a skeptic, I have nothing but respect for the researcher’s professionalism doing the work. Saying they can be co-opted by one entity is saying they can be co-opted by any entity.
Third, we used to have a saying about predictive models (I did statistical analysis very unrelated to climate change): all models are wrong; some are better than others. Applying a zero defect criteria for models is incorrect on both ends of the spectrum. Saying a model is imperfect is mutually exclusive from saying it is invalid for use, unless one can show that the model’s imperfections amount to baseline errors.
Lastly, the publishing of a paper that goes against the grain (and the subsequent criticism of it based on facts) makes skeptics like me take a second look. I have no qualms when a majority of scientists agree on a general body of work; I have bells and whistles going off in my head when only papers which agree with it are allowed to be published. Proper consensus is formed when dissenting views (when, like the LC09, they are put into the formal debate) are allowed to be taken into consideration.
Melty,
Perhaps it is the difference in our professions. In my profession, there are people whose work is sufficiently specialized that they have published with everyone who does anything similar. And in my own case, I often bring in statistical techniques that are well known in other fields but unfamiliar in my own–I’ve actually had editors come back and ask me for potential reviewers.
I think that the thing you are neglecting is that the reputation of the referee or editor suffers if he does a poor job or shows favoritism or prejudice. If it happens enough, the reputation of the journal suffers. This is a situation where all the incentives line up on the side of proper behavior.
You need to read more carefully. The problem here isn’t that LC09 puts forward a dissenting view, it’s that it’s stunningly wrong.
Completely Fed Up says:
Jim Bouldin, the letters are not meant to be full-on papers.
What do you mean by “full-on papers”? Did you read the statement by Famiglietti, linked to by John N-G? GRL papers are designed to be cutting edge, newsworthy stuff that supposedly needs to get out right away. It’s a very high profile journal. I don’t see how Lindzen and Choi’s paper doesn’t qualify for submission.
I do not think that what is being done requires changing.
Have you read any of the comments here at all? Or the post itself?