From blog to Science

There is a lot of talk around about why science isn’t being done on blogs. It can happen though, and sometimes blog posts can even end up as (part of) a real Science paper. However, the process is non-trivial and the relatively small number of examples of such a transition demonstrate clearly why blog science is not going to replace the peer-reviewed literature any time soon.

Way back in April 2005, I wrote a post on RC on the role of water vapour in the greenhouse effect and why it is considered a feedback and not a forcing in the IPCC sense. It was a basic enough exposition, and in lieu of finding a comprehensive paper on the components of the atmospheric greenhouse effect, I did a few very simple (even simplistic) GCM experiments to show what I was talking about. The bottom line was that CO₂ was indeed an important contributor to the present day greenhouse effect, and depending on how you calculated the percentage, could account for between 9 and 26% of the effect.

This proved useful, and soon the page was being quoted quite widely. But the calculations were not very sophisticated and I started to be concerned that they were being given more credibility than they deserved – not that they were necessarily wrong (they weren’t), but because a blog post doesn’t give enough context. For instance, some people incorrectly thought that the range 9-26% was the uncertainty in the calculation, rather than two different conceptual estimates. So I started to look for ‘proper’ references for these kinds of calculations.

I had already seen a few papers that calculated the importance of water vapour, CO₂ and clouds for a one-dimensional standard ‘profile’ (Ramanathan and Coakley, 1978 for instance), and I was pointed to a section in Kiehl and Trenberth (1997) that turns out to be the most useful reference. They too had used a single ‘typical’ profile. Both of these references (and a few others – like Ray Pierrehumbert’s 2007 paper which used the NCEP global distribution of water vapour and temperature) generally calculated the importance of CO₂ in one of two ways – either by looking at what happened when you removed CO₂, or by looking at what happened when only CO₂ was operating (though rarely both) (because of the spectral overlaps between the different absorbers, the second number is always larger than the first). Invariably, the treatment of clouds was highly simplified or neglected.

What I didn’t find was any justification in the literature for the most widely quoted ‘contrarian’ view of the issue that CO₂ was ‘only 2%’ of the effect. I traced this back to a book review that Lindzen wrote about the first IPCC report, but never found any actual reasoning in support of this.

So in putting together a real paper there were a number of necessary steps that went beyond what was appropriate for a blog post. First, the previous literature had to be collated and their results reported in a consistent way. Second, there were a number of differences between the more serious calculations done for the paper and the calculations done casually for the blog. We used a longer period of time (a full annual cycle rather than a single time step) to avoid a bias towards a particular part of the year. Then we rechecked that the radiation code was still giving good results at very low CO₂ levels…. and it turned out that it wasn’t – and so we needed to update the code via comparisons with a more complete line-by-line model so that all the tests we were doing were within the validated range of the radiative-transfer code. Finally, we did many more tests – more combinations, different baselines – to try and ensure that the results were robust.

When it came time to submit the paper, we first tried pitching it to BAMS as a popular science piece that would try and explain the concept and clear the air (so to speak). However, for various reasons this didn’t work out (two rounds of unsatisfying reviews). I’d say it was mainly due to the draft not really being pitched at the right level for BAMS. One amusing aspect of the process was that one of the referees initially suggested that our paper wasn’t necessary because it was common knowledge that the attribution to CO₂ was between 9 and 26% (sound familiar?). As it turns out, they were reading a page from UCAR which was quoting (without attribution!) from my original blog post.

There was one other interesting (and highly critical) review which objected to the criticism of Lindzen’s 1991 comment, though it is perhaps worth noting that they considered the 1991 comment to be ‘formally incorrect’.

After a period in which I was a little tired of the whole exercise (it happens), we then submitted the paper to JGR, where it had an easier passage. At the same time, the simulations we had done for the paper were also used as part of a broader paper that Andy Lacis wanted to put together for Science. Both these papers appeared in October 2010 – some five years after the initial post, 3 and a half years after the first journal submission, 5 rewrites and 11 reviews.

So why bother to turn ideas from blog posts into real papers? Well, first off, you get to do a much more thorough job. You have the time and space to check multiple variations of the method, and you can take the time to do a proper literature review. And because you have put more effort into it, it is rightly seen as more credible. It’s worth noting that in our case the paper benefited from comments from all reviewers (even the very critical one) – language was tightened up, a broader literature search was done, concepts were clarified and many of the additional issues raised were dealt with.

In turn, the more credible work on the topic forms a stable point around which to craft a critique (if desired), and hopefully provides more of substance to critique (versus a series of blog posts that can be laced with distracting commentary, conceptual errors and moving targets).

To be clear, it is not only the reviews by peers that makes a peer-reviewed paper better than a blog post. Since it is known ahead of time that there is an effort required to get past the peer-review hurdle, the resulting work is usually more reflective, more interesting, more concise and more of a serious contribution – even before it gets to the editor.

Thus when scientists who find themselves criticised in the blogosphere quite often ask their critics to submit their points for peer-review, the point is not to dismiss a critique, but rather to encourage the critics to make the critique as well formulated and a propos as possible. This doesn’t always work of course, but is it nonetheless worthwhile. The alternative, especially for high profile issues, is to try and deal with a multi-headed hydra of critiques that range from the ill-informed to the excellent. Unfortunately, technical commentary does not work well at the ‘speed of blog’ and conversations often take a personal turn (which only rarely happens in the literature).

The many existing critiques of peer review as a system (for instance by Richard Smith, ex-editor of the BMJ, or here, or in the British Academy report), sometimes appear to assume that all papers arrive at the journals fully formed and appropriately written. They don’t. The mere existence of the peer review system elevates the quality of submissions, regardless of who the peer reviewers are or what their biases might be. The evidence for this is in precisely what happens in venues like E&E that have effectively dispensed with substantive peer review for any papers that follow the editor’s political line – you end up with a backwater of poorly presented and incoherent contributions that make no impact on the mainstream scientific literature or conversation. It simply isn’t worth wading through the dross in the hope of finding something interesting.

In the end of course, the science will win out. No single paper is ever the last word on an issue, and there are always new approaches to try and new data to assimilate. But the papers will endure long after the plug has been pulled on a blog. I certainly think that blogs can be of tremendous value in bringing up more context and dispelling the various mis-apprehensions that exist, but as a venue for actually doing science, they cannot replace the peer-reviewed paper – however painful that publishing process might be.