Archives for June 2011

A paper in the scientific literature has to have some minimum level of content to be worth publishing (and regrettably, the search for the ‘Least Publishable Unit’ (LPU) of work is occasionally apparent for those wishing to pad their CVs). But what happens when someone has something worth saying that falls below that level? This is might be an update to an earlier paper, with modifications to figure, one extra sensitivity test, or some other minor addition, that could be of interest to readers of the original, but it doesn’t really get to the point where one would write a whole new paper. Some new journals (such as Geoscientific Model Development, GMD) have set up mechanisms to provide versioning of papers so that small updates can be made relatively easily, but this is rather uncommon.

A common reason to want to add additional material is in the light of subsequent commentary. In the usual case of an official submitted comment, the required response provides a good opportunity to give further details, add justification, or even agree with the comment (this last one doesn’t happen very often, but it does occur). Comments are often hard to publish for all sorts of non-scientific reasons, see Rick Trebino’s appalling story for instance, and note GRL’s 2010 decision to stop accepting comments altogether. Unfortunately not all comment/response pairs that do get published are worthwhile, but I still think they can be useful.

Where criticism occurs on a blog, there is no necessity to respond (as there would be for a submitted comment), but it is possible that there is something worth addressing (not everything is of course). Responses posted to that material – either in blog comments or in other blogs are however a little unsatisfying since the blog commentary and response are not tied to the actual paper (though mechanisms like that used by ‘Research Blogging‘ or JournalTalk could conceivably be used), and can quite frequently spiral out of control (with additional criticisms, responses, and often vast amounts of irrelevant commentary). As a useful archive of a discussion, this leaves much to be desired, nonetheless, the determined reader can usually find some nuggets.

But there is a third case where the comment/response effectively never sees the light of day. For a number of reasons critics will sometimes decide not to submit a comment, but rather a whole new paper. This might be because they want to include more information than a comment would allow or are making a comment on a previous work as part of a larger paper. Or it might be that they (correctly) note that comments are not as useful on the CV as a ‘proper’ paper, or indeed, a journal does not want to accept a comment for some reason. Less nobly, comments are sometimes avoided to try to prevent the original authors from having the last word. However, there is a risk that this paper never gets published at all (perhaps because it has less than one LPU, or it isn’t very good, or it clearly nothing more than a comment on a previous paper, or the authors lose enthusiasm). In that case, the criticism, and any response to it from the original authors (if they were asked to respond), simply disappears from sight. While possible, in my experience it is very rare that the critics then turn back to the official comment route.

It is very unusual for any scientific paper to the last word on anything, and there are almost always things that, in retrospect, one would have done differently. So it is not surprising that questions get raised through all this that the original authors might want to tackle without themselves submitting a whole other paper. Theoretically most papers would benefit from a well-refereed post-publication commentary. Yet, without a formal mechanism to shepherd this process, this material generally falls through the cracks.

As readers might have surmised, this is leading up to something.

Two papers that I was an author or co-author on in recent years generated a fair amount of blog commentary – Schmidt (2009) in IJoC and Benestad and Schmidt (2009) in JGR – mostly because they were explorations of issues raised by authors critical of the mainstream view of climate science. Despite the blog discussions, however, in neither case was a comment/response pair published. A comment was submitted on Schmidt (2009) by Jos de Laat, but this did not pass peer review (rightly I think) and no more has been heard of it. In both cases however, other authors whose work was criticised (specifically Ross McKitrick and Nicola Scafetta) submitted new papers to the same journal that were effectively just extended comments. A couple of valid points were made, but much in the draft texts was either wrong or irrelevant. I was asked to respond to both submissions by the editors involved, and did so in the form of a signed draft response as if the papers had indeed been submitted as comments. In both cases however, the papers were eventually rejected. A similar paper by McKitrick and Nierenberg appeared in another journal, while Scafetta’s paper has not been seen again.

So why bring this up now?

The fact is well-crafted comments and responses on both these papers would have been a useful contribution to the literature, and it is a shame that this didn’t happen (as I stated in at least one of the responses). That the authors were unwilling to submit ‘just’ a comment for whatever reason is part of the story (but it is not clear that any actual comment would have stuck to the points I thought worth making, or that the comment/response would have passed peer review either). But both papers have been mentioned recently in various contexts and it was apparent that the conversations might have been at least a little more interesting if the at least some of the unpublished correspondence on the papers had been available.

It strikes me that these are unlikely to be unique circumstances (or at least I would be very surprised if this kind of thing hasn’t happened to other people). So is there interest in RC providing space for these kinds of discussions? Authors, with something extra to say that they don’t ever think they’ll put in a new paper, could add some extra analysis and tie up some of the loose ends. I’d be happy to start the ball rolling by dealing with some issues relating to the papers mentioned above, but is this something else anyone would care to do? Should we instead be advocating for a rapid reaction online journal specifically for worthwhile comments and replies, especially since some journals have stopped accepting them? Thoughts welcome.