or…Some thoughts on Personal Responsibility and the Peer Review Process
Eric Steig
Ryan O’Donnell made a series of serious of allegations against me at ClimateAudit, in the context of our friendly dispute about whether his new paper in the Journal of Climate supports or ‘refutes’ my own results, published in Nature.
To his credit, Ryan has offered to retract these allegations, now that he is a little better acquainted with the facts. However, it is still important, I think, to set the record straight from my point of view. There were such a great number of claims about my “dishonesty,” “duplicity” and [implied] stupidity, all of which are untrue, that it really isn’t worth trying to respond in any detail. Just responding to the main two ought to suffice to make the point.
“Eric recommends that we replace our TTLS results with the ridge regression ones (which required a major rewrite of both the paper and the SI) and then agrees with us that the iRidge results are likely to be better . . . and promptly attempts to turn his own recommendation against us.”
“[in his RealClimate post]…he tries to … misrepresent the Mann article to support his claim [about the iridge routine] when he already knew otherwise. How do I know he knew otherwise? Because I told him so in the review response.”
While it is quite possible that O’Donnell believes both of these claims, they are both false, as it is rather easy to demonstrate.
First, I never suggested to the authors that they use ‘iridge’. This was an innovation of O’Donnell and his co-authors, and I merely stated that it ‘seems’ reasonable. As O’Donnell’s co-authors are fond of pointing out, I am not a statistician, and I did not try to argue with them on this point. I did, however, note that previously published work had shown this method to be problematic:
“The use of the ‘iridge’ procedure makes sense to me, and I suspect it really does give the best results. But O’Donnell et al. do not address the issue with this procedure raised by Mann et al., 2008, which Steig et al. cite as being the reason for using ttls in the regem algorithm. The reason given in Mann et al., is not computational efficiency — as O’Donnell et al state — but rather a bias that results when extrapolating (‘reconstruction’) rather than infilling is done.
Second, I was the reviewer of the first three drafts of O’Donnell et al submission. However, I did not the review draft four, which was the published one. , and which is markedly different from draft 3 [note correction: it has been pointed out that it’s not really very different; in other words, my criticisms of draft 3 were ignored]. Nor was I ever shown their response to my comments on draft 3, so I did not in fact ‘already know’ what O’Donnell claims I did. It appears that the editor was swayed by the arguments that I was not a helpful reviewer. In other words, even if one believes that I was “bullying” them into showing particular results, they still had the last word (as any author should).
The fact of the matter is that my reviews of O’Donnell’s paper were on balance quite positive. I wrote in the confidential comments to the editor in my very first review that
I emphasize that I think that a fundamentally reworked version of this manuscript could potentially provide a useful scientific contribution, and many of the points made do indeed have scientific merit. Indeed, the authors have done a very thorough analysis, and are to be congratulated on this.
In my second review, I wrote that “O’Donnell et al. have substantially improved their manuscript … and clarified a series of items that led to some confusion on my part.”
With respect to O’Donnell’s lengthy discussion of the technical aspects of the difference between our papers, I’m not complaining. It is possible to have a disagreement — or even to be wrong — about the technical aspects of a paper without being ‘duplicitous’. The dependence of any analysis on the technical aspects of the methodology are completely legitimate subjects of discussion, and it is important to be clear about what does and what does not depend on those choices. People who want to see what the data are saying about the real world will focus on the similarities, people who are focussed on proving people wrong will focus on the differences. This is how O’Donnell and I can (legitimately) disagree about what their results mean.
The reality is that editors, not reviewers, make decisions about what is acceptable and what is not. Any comments I made as a reviewer of O’Donnell et al.’s work would have been weighed against what other reviewers said (and obviously were, since the main criticism I had of the paper was not ever addressed), not to mention the responses of the authors themselves. And the decision about what content eventually winds up published is still ultimately up to the authors. If the authors feel that they are being bullied into presenting their results in a particular way (as is the allegation here), then they have the choice to withdraw the paper and submit it elsewhere, or complain to the editor. But once they have signed off on the paper, it is their paper, and blaming someone else — reviewer or editor — for its content is simply passing the buck.
It’s perhaps also worth pointing out that the *main* criticism I had of O’Donnell’s paper was never addressed. If you’re interested in this detail, it has to do with the choice of the parameter ‘k_gnd’, which I wrote about in my last post. In my very first review, I pointed out that as shown in their Table S3, using k_gnd = 7,
“results in estimates of the missing data in West Antarctica stations that is further from climatology (which would result, for example, from an artificial negative trend) than using lower values of k_gnd.”
Mysteriously, this table is now absent in the final paper (which I was not given a chance to review).
Some months ago, O’Donnell cordially (though quite inappropriately) asked me if I was one of the reviewers, and also promised not to reveal it publicly if I didn’t want him to. I told him I was, but that I would prefer this not be public since the ‘opportunity for abuse’ was simply too great. Talk about prescience!
Many of my colleagues have warned me many times not to trust the good intentions of O’Donnell, Condon, and McIntyre. I have ignored them, evidently to my peril. But you know what has given me the most pause? The fact that a number of my colleagues and many otherwise intelligent-seeming people still seem to treat these guys as legitimate, honest commenters, whose words have equal weight with, say, those of Susan Solomon or J. Michael Wallace, or, for that matter, Gavin Schmidt or Mike Mann or myself. As a reporter wrote to me today “it’s simply impossible for a lay observer to make a judgement on his/her own.” Really?!
Perhaps there is a silver lining here. Perhaps the utter silliness of the shrill accusations that O’Donnell made against me — based on a version of the facts, in his head, that are demonstrably and unequivocally false, coupled with the fact that he then retracted them (or at least has promised to do so), will help more people see what the steadily growing list of other scientists who’ve been accused by McIntyre and his associates of plagiarism, dishonesty, data manipulation, fraud, deceit, and duplicity have been telling me for years: these people are willing to say anything, regardless of the cost to others’ reputations and to the progress of legitimate science, to advance their paranoid worldview.
I’ve even got a name for the clarity this affair would seem to offer: O’Donnellgate.
Sadly, attacking climate scientists by mis-quoting and mis-representing private correspondences or confidential materials appears now to be the primary modus operandi of climate change deniers. To those that still don’t get this — and who continue to believe that these people can be trusted to present their scientific results honestly, and who continue to speculate that their may be truth in the allegations made over the years against Mike Mann, Ben Santer, Phil Jones, Stephen Schneider, Andrew Weaver, Kevin Trenberth, Keith Briffa, Gavin Schmidt, Darrell Kaufmann, and many many others, just because they ‘read it on a blog somewhere’ — I’d be happy to share with you some of the more, err, ‘colorful’, emails I’ve gotten from O’Donnell and his coauthors.
If you still don’t get it, then I have a suggestion for a classic short story you should read. It’s called, The Lottery, by Shirley Jackson.
Seems like there is too much distrust on both sides of this debate. Having read both your comments here, and the ones from O’Donnell, I can’t see that you are guilty of the most serious allegations, Eric. Some seem to think the duplicity charge stands as they interpret you as having recommended a statistical method you later turned around and criticized.
However, I do find it strange that you were reviewing his paper in the first place. It just seems like a bad idea since the paper could be critical of your own findings. Anyway, I think the dialogue would be better if more open discussion was permitted, and that the borehole was a place for ad hominems and other destructive posts.
> in my experience
Which is?
quoteIn my experience, both you and Dr. O’Donnell bear responsibility for where this matter has gotten to. If I assessed responsibility, I’d say Dr. O’Donnell is 60-70% at fault, and you are 40-30% at fault. But the percentages are immaterial.
Interpreting the interpreters: my guy lost badly and took down his garbage, so I’m trying to salvage something, anything, so I’ll try to claim 30 to 40%. Sorry, no way.
I see two really true things: one, Antarctica is warming; two, one of these two had the scientific curiosity to find that out and the other one never ever would have.
Comment by Al Henning — 10 Feb 2011 @ 5:22 PM
That’s not right Al.
Putting aside your silly attribution of percentage naughtiness (the naughtiness is all on the side of O’Donnell pretty obviously), I disagree with your “what’s true”, both in terms of common reasonableness and reality (in my experience):
1. You generally don’t do this. But there might be instances where you might ask that question. Not a big deal and hardly a hanging offence!
2. Yeah, O.K…
3. Like “1” that’s generally true, but there are instances where you might. I’ve done this on a few occasions.
4. I’ve published around ~80 papers are reviewed many more than that. I’ve never been involved in a double blind review process. I don’t see any problem whatsoever with knowing who’s paper you are reviewing. The point of the editorial process and multiple reviews is to ensure that the review process is fair and thorough.
5. of course…what’s your problem there? That’s not an issue in this little broughah
6. Like “5”…of course…what’s the problem?
7. Not always necessary. The editor can decide whether or not the authors response goes back to the reviewer. In fact for the journals I review for, it’s about 50-60% likely that I won’t see the authors response to my review.
8. yeah..of course
9. That’s up to the journal. Some open access journals are exploring publishing the reviews. Don’t see any real problem with that..
@chris:
Double-blind review has been proven (in the peer-reviewed literature) to remove review bias. Mostly applies to gender-based bias. Google it.
If I have a ‘guy’ in this matter, it’s Dr. Steig. From the science perspective, at least. However, in future, if I were he, I would recuse myself from any review of O’Donnell’s work.
That doesn’t change this fact: failing to follow rigorous peer-review policies and procedures — and instead engaging in endless, ad hominem, unethical rantings about these matters in blogs, including publishing details of reviews and reviewers in public — has resulted in a majority of the population believing a) climate scientists are all liars, b) global warming is a hoax, and c) AGW is worse than a hoax. Which means, all the ‘summaries for policy makers’ in the world, which IPCC may wish to construct, are worth nothing. And that outcome is a disaster.
Climate scientists have lost virtually all public credibility. And, that scientists spend their time trying to re-establish that credibility — instead of answering the hard science questions, and making fact-based statements on what, if anything, ought to be done about climate change and AGW — is an enormous tragedy.
As for my bona fides: you can find me on Google Scholar. Or SCI. FWIW.
[Response: Actually, research on public opinion shows that’s not true (about credibility). Perhaps you should read something other than blogs?]
My experienced doing peer-review has been here are three cases:
(1) Very bad. Easy to explain a reject recommendation;
(2) Very good. Also easy.
(3) In the middle. For these, reviewing was hard and time-consuming.
GSW asks: “But is there really no possibility of a behind the scenes reconciliation?”
I’m sorry, but what possible benefit could Eric reap from such a reconciliation? Both Steig et al. and O’Donnell et al. are attempting a very difficult reconstruction. The truth will probably not be definitively known for years. To think that these two teams will be the only players over than interval is naive.
Second, I would never author a paper with O’Donnell for the simple reason that I would never want to be associated with a person who bahaved that deplorably! O’Donnell and the denialists have revealed their true colors. I hope everybody got an eyefull. O’Donnell has exhibited the kind of behavior that could end an academic career.
Al Hennig,
I’m not sure where you come up with any “blame” for Eric, let alone 30%-40%.
From your list of 9 points, at best Eric was “guilty” of #3 (but not volunteering it, simply answering the question when asked — hardly a hanging offense — and in fact I suspect he would have been brutally attacked if he’d failed to respond).
And I believe Eric when he says that his reviews were critical with the intent of getting the paper published in a truly viable form. The most evil thing he could have done was not to say to the editor “don’t publish this” (since that wasn’t ever in his control anyway), but rather to let any errors he found stand, and let the published result be such a travesty that anyone could refute and demean it as quickly as possible.
Most of your points fall to the journal or editor to enforce, some to the person who submitted the paper… not to the reviewer.
I think it’s probably pretty easy to sit back and apportion blame when you’re not the one who’s been venomously attacked on the Internet by multiple bloggers over a fair period of time, and had your reputation smeared… for no good reason. Has anyone ever created an unflattering cartoon of you just for doing your job and coming to a conclusion that they don’t like?
Personally, I don’t know where Eric (and Gavin and Jim and Ray and Mike and the rest of them) get the patience.
What is the expertise of Mr O’Donnel? Is he [edit] Haven’t found any other [except the web editor for the Heritage Foundation!!]. Did he get sucked into being a surrogate citizen auditor by the McI branch of the contrarian party?
Not weak, just a Canadian thing, too bad a couple contrary Canadians demonstrate the exact opposite inclination, eh?
[Response: No he isn’t that one. I deleted the link. The last guy some poor professor of physics needs is to be accused of being that Ryan O’Donnell. –eric]
Based on what I have read I would agree with Eric that O’Donnell is an ethical player but one that has gotten poked with a sharp stick and so has violated what he knows to be what he should have done. The sharp stick was not sharpened or used by Eric, I suspect at most Eric’s contribution to the sharp stick was to point out in a article here that the O’Donnell paper showed that the Antartic is warming and to continue to take what was studied in that paper seriously enough to really think about what would make the science in it better. I can only imagine the amount of nonsense that got spewed and that then made for the rest of the unfortunate circumstance.
Now all this junk has gotten blown into the public eye and its going to take a whole lot more personal strength to be able to offer the apology that is called for. In particular because the same folks that jabbed the stick are going to keep doing it and to keep trying to claim that the errors in behavior weren’t from their side.
Well, as I said to MT earlier:
Remember, in the long run, all of us are contributing to the same record:
http://rsta.royalsocietypublishing.org/content/369/1938/1036.abstract
Re the comment by Steig:
To his credit, Ryan has offered to retract these allegations, now that he is a little better acquainted with the facts.
I cannot locate any evidence to support this assertion by Steig.
Could someone please post the link showing O’Donnell’s offer of retraction?
Thanking you in advance.
[Response: It was in a personal email to me, on which Andrew Revkin was copied. It will be interesting to see if he actually follows suit. –eric]
I think a lot of people are missing the point. In my profession, peer review is used to pick up flaws and improve a report. If a reviewer makes only a few random comments about typos and grammar on an early draft, I suspect they haven’t made much effort to digest or critique the report. (The only time I objected to comments, as opposed to accepting/rejecting what was suggested, including those from arch-competitors, was an objection to the way in which an individual expressed his point – using extremely foul language. And I’m not easily shocked by strong verbiage. That’s in more than thirty years of report writing.)
In scientific journals there are two purposes. Firstly to advise editors on the merits of a paper for publication (ie signal a go/no-go to publication). If the reviewer believes the paper is worthy, then they make suggestions for how to improve the paper. I don’t know whether reviewers get paid or not. My impression is that they aren’t, but do it for the good of the science to help add to knowledge. Therefore publishing scientists would see the review process as an important opportunity for a final check before their work is laid bare before the entire body of their peers. (Not that it should be relied upon by sloppy writers.)
I very much doubt that the paper in question would have got to the required standards for publication if not for the efforts of reviewers. This must seem like a real kick in the teeth to Prof Steig. If not for his encouragement I doubt this paper would have been prepared. If not for his comments and support as a reviewer and similar efforts of other reviewers, the authors would not have got it to the standard necessary for it to have been accepted.
I do not understand any suggestion to Prof Steig that there be a ‘reconciliation’. I do understand calls to O’Donnell and co for a heartfelt apology.
As others have pointed out, the peer review process does not end with publication. After publication, if the paper is of any interest or relevance, it gets subjected to further review by the broader readership of peers.
The paper is not a seminal work. It’s derived from work already conducted by Prof Steig. I don’t imagine there will be any more contributions from the authors on the topic. I don’t think they were very interested in the topic to begin with. From their subsequent behaviour one can only conclude that they were only interested in the never-ending audit and proving the scientists (not science, but scientists/personal) wrong.
This is another example of why it does not pay to engage with committed deniers/delayers, and to distinguish between such people and ordinary people with no axe to grind.
What? None of Steig’s reviews are 24 pages long; they’re 14, 6 and 4 pages respectively. None of O’Donnell et al.’s submissions are anywhere near as short as 8 pages; the shortest is 45.
@Bob:
If Dr. Steig had not responded to Dr. O’Donnell’s question, we wouldn’t be here. In my field, in my professional experience as an author and reviewer of technical manuscripts for peer-reviewed journals, the policies I laid out are the norm. Seeing what has transpired with ‘ODonnellgate’, and given the cavalier attitude toward the confidentiality of the peer-review process, as manifested by the comment by @chris, and by other posters, I understand pretty well the cause.
Regarding: [Response: Actually, research on public opinion shows that’s not true (about credibility). Perhaps you should read something other than blogs?]
I rarely read blogs. I do have conversations with doubters; one of them led me to ODonnellgate. If you don’t understand the assertion about public incredulity over climate science to be true, then your problem is even bigger than you know. The vehemence and disrespect shown toward climate scientists in general, and Al Gore and IPCC in particular, is unbelievable.
While a search of Google Scholar on ‘climate change public perception’ reveals numerous sources, public media sources regarding shifting public perception of climate science credibility can be found at these links:
NYTimes: http://www.nytimes.com/2010/05/25/science/earth/25climate.html?_r=1&th&emc=th
BBC: http://news.bbc.co.uk/2/hi/8500443.stm
Gallup on Australia: http://www.gallup.com/poll/141782/australians-views-shift-climate-change.aspx
Gallup on ‘scientists and progressives still lousy at messaging’: http://climateprogress.org/2009/03/12/gallup-poll-exaggeration-global-warming-deniers-media-messaging/
Yale Forum: http://www.yaleclimatemediaforum.org/2010/06/pnas-study-sparks-support-and-criticism/
What I find interesting is that people on all sides of the climate change debate are unwilling to admit their research findings are invalid. Given the findings of “Why Most Published Research Findings Are False” (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/) and the statistical nature of climate change research (eg paleclimatology) it would not surprise me that 50%+ of climate change studies are invalid. Perhaps climate change scientist feel the figure is closer to 0% but I’m sure scientists in every field feel this way about their own field.
[edit]
I do get the strong impression that you passed around his paper to your colleagues at Real Climate and used their criticisms in your own comments – is that impression fundamentally in error?
[edit]
[Response: I deleted the rest of your offensive ‘suggestions’, but I will answer the one coherent (but stupid) thing you asked me: You get that impression because that is the impression that O’Donnell & Co want you to have. But the facts remain that no one saw the paper except me and whatever other formal reviewers it was sent to. Stop making stuff up — it’s unethical and makes you look like an idiot –eric]
Al Henning said: “That doesn’t change this fact: failing to follow rigorous peer-review policies and procedures — and instead engaging in endless, ad hominem, unethical rantings about these matters in blogs, including publishing details of reviews and reviewers in public — has resulted in a majority of the population believing a) climate scientists are all liars, b) global warming is a hoax, and c) AGW is worse than a hoax.”
Isn’t that the goal of the deniers? Why, then, are you surprised to see O’Donnell and friends engaging in these things? And why are you trying to assign blame to Eric?
#101
Why do people keep bringing this up? It is standard practice in these cases for the author whose work is the main focus of a paper to be asked to be a reviewer.
In fact, if you want to see an abusive reviewer in action, check out McIntyre when he reviewed Wahl and Ammann (Climatic Change). He even brought misconduct charges against one of them (Ammann, I think) for withholding some material McIntyre wanted to get his hands on.
This sounds like a perfect description of those good people at Climate Audit. Not the climate science community.
I mean it’s hilarious. Who published the details of reviews and reviewers in public? RyanO.
Who engages in endless, ad hominem, unethical rantings about climate science in blogs, including endless assertions of scientific fraud, duplicitous behavior, incompetence, etc etc? Go read Climate Audit and WUWT.
Al Henning … blaming the victims.
Pfft.
Brief note about #47… Eric, your response suggests you were thinking of altimetry. GRACE, of course, is not an altimeter. It measures mass changes, not elevation changes. But you’re right about the time series length — a bit too short to address the issues discussed here.
[Response: You are right of course, but the point is the same — you are not going to be able to get at temperature by looking at mass changes! The satellite I was thinking of is IceSat.–eric]
That is remarkably naive.
> In my field … the policies I laid out are the norm
Published standards? Consensus? Your opinion?
Al Henning-
Whether or not you have a point, you’re missing the point. If you really want to find out why ‘tragedies’ such as deteriorating public trust in climate scientists and climate science happen, there’s a very simple formula to get you there: make a research finding that imperils the profits of a trillion some odd dollar global industry. That is all. You will find if you do that uptight adherence to the most obsessively compulsive right thinking northern european procedures afford you… 0.0 protection against a smear campaign the ferocity of which you are quite evidently unable to fathom (happy to put that out to a few more sig figs if you like).
Need more color? Picture this: some of the best hackers money can by in some locale somewhere on the planet illegally stealing years of your personal and professional correspondence and pilferiing through its entirety for the single cherry that fits the smear: the inartfully phrased reply to Halvard written between 7:06 and 7:07 on the morning of November 8th before you got that coffee down. Then picture that cherry being shorn of context, distorted beyond all recognition and then beamed around the planet at the speed of light to millions of eager ideologues across the globe via every media medium imaginable, in many cases before you had a chance to even register a response.
And how would you like to be bombarded incessantly with FOI requests compelling you to release every last bit of code you ever wrote or data you ever worked with to amateurs whose only goal was to crucify you with it? And for cries of fraud to come if you only were able to reach 99% disclosure of said for whatever practical reason. And how would you like to be virtually stalked by these same characters at you day in day out for quite literally years, your every sentence of your every paper and every calculation held up, charted, distorted and accused- with every, ‘mistake’ (and mostly amateurish or willful misunderstanding) attributed to fraudulent deception for the purpose of personal enrichment or some other nefarious scheme? How would you like to be threatened with criminal law suits for all and sundry? Ever had a death threat before?
Seriously, just how cavalier can a person be? In fact, given your apparent babe in the woods ignorance of the forces at play here, I’d be surprised if you wouldn’t be in serious trouble with a finding that threatened the high top basketball shoe market. And you want to, what, claim that it’s the peer review process and the occasional instance where climate scientist’s emotions get the better of them which has led to this whole high profile high tech lynching that’s been unleashed on them? Really? All I can tell you is that you should really rent ‘The Insider’ read this and then come back with your thoughts. If you’ve by then you’ve managed to descend that high horse, you might find yourself one helluva lot more empathetic to the abuse they’re forced to endure simply for doing their jobs.
Comments are off. Case closed. Thanks for your support (most of you). Comments on the West Antarctica is still warming post are still open. Science only please.–eric]
Re: O’Donnellgate,
O’Donnell has a
retractionapology“response” up on CA. More tedious recriminations about how Ericforced them at gunpoint to use“specifically proposed” iridge, etc. ‘Scuse the noise if someone already mentioned it.