Do we need a new venue for post-publication comments and replications?
Social media is full of commentary (of varying degrees of seriousness) on the supposed replication crisis in science. Whether this is really a crisis, or just what is to be expected at the cutting edge is unclear (and may well depend on the topic and field). But one thing that is clear from all the discussion is that it’s much too hard to publish replications, or even non-replications, in the literature. Often these efforts have to be part of a new paper that has to make its own independent claim to novelty before it can get in the door and that means that most attempted replications don’t get published at all.
This is however just a subset of the difficulty that exists in getting any kind of comment on published articles accepted. Having been involved in many attempts – in the original journal or as a new paper – some successful, many not, it has become obvious to me that the effort to do so is wholly disproportionate to the benefits for the authors, and is thus very effectively discouraged.
The overall mismatch between the large costs/minimal benefit for the commenters, compared to the real benefits for the field, suggests that something really needs to change.
I have thought for a long time that an independent journal venue for comments would be a good idea, but a tweet by Katharine Hayhoe last weekend made me realize that the replication issue might be well served by a similar approach. So, here’s a proposal for a new journal.
Commentary And Replication in Earth Science (C.A.R.E.S.)
It is well known that existing approaches to post-publication reviews of science are hampered by long delays while responses are gathered, the inbuilt disincentives of journals to want to publish commentary that is (even implicitly) critical of their editorial decisions, the high bar for what is deemed a worthwhile criticism, and the difficulty in tracking commentary that occurs informally (on social media, blogs, conference remarks). Indeed, key journals in Earth Sciences – Nature, Science, PNAS, GRL – have very limited or no comment facility at all.
Some recent attempts to remedy this have been useful – particularly PubPeer (as discussed here). But that effort is based on anonymous comments, and has had it’s greatest success in finding dubious imagery/claims in biological journals. For whatever reason, it has not taken off as a well-used commentary platform in Earth Sciences. Nonetheless, there are some very useful innovations that have arisen from this effort – notably the PubPeer browser plugin that highlights PubPeer comments that have been made on any doi quoted. For instance, if you have the plugin, you should be able to see a link to the Pubpeer comments below on a recent Nature editorial on post-publication review. Also, the automatic notification to authors of comments being made is sensible.
A new journal could provide several clear advantages over the status quo if it was able to reduce the barriers to publication while maintaining quality. For this to work it would have to have low overhead, so would be online only. Comments would be accepted related to any published paper in the field of Earth Sciences (broadly conceived). Editorial peer review would be present, but should be minimal (basically for tone and sanity). There would not be a requirement for a comment to be equivalent to a full paper. To aid recognition of efforts, comments/replications would be published and assigned a doi straight away and responses (and perhaps even conversations) would follow over time as part of the same (evolving) page. A browser plugin and notifications like for PubPeer would be extremely useful.
I envisage this journal being used in multiple ways:
- To make quick corrections to published work, that perhaps the original authors are unaware of, or deem too trivial or too unimportant to bother with (note that an author’s view on this is not necessarily universally shared).
- To replicate analyses (or not) using analogous data and perhaps different methods to demonstrate robustness (or not) of a result.
- To make ‘standard’ comments that take issue with some claim or conclusion of a paper.
- To be somewhere where original authors can append additional analysis that wouldn’t merit a new full paper, but that might be a relevant and useful appendix to the original paper.
Personally, I am aware of multiple examples of papers where data errors, incorrect equations, or unsupported conclusions have been made and that remain unchallenged and/or uncorrected in the literature. Often, the people who found these issues have not bothered with a formal comment because of the time and hassle of doing so, or they are under the (usually mistaken) impression that ‘everyone’ already knows. I have also had examples where supplemental material on a paper might be useful, but has never seen the light of day.
I don’t think that a journal or comments are the right venue for issues of misconduct or fraud. Journals and institutions have procedures for this (however imperfect), and trying to peer-review suggestions of misconduct without access to lab notebooks/machine logs/correspondence etc. is a job for investigative committees, not an online journal.
There are some potential pitfalls. It is possible that this journal could be used to target high profile papers for non-scientific reasons, and so occasionally a stricter peer review protocol might be needed to separate out genuine issues from worthless politicized critiques. Given that each comment would be reviewed and have a doi, some people might decide to bulk up their publication list by submitting a lot of comments. I don’t actually think that is too bad though – assessors of their research output should be able to see through this, if indeed the comments are of little worth.
As is well known, comments and responses can become contentious and editors would need to have the means to tone down inflammatory rhetoric, disallow questions of motivation and remove unsupported accusations of misconduct. Again, the experience of PubPeer with this kind of issue is instructive.
To help potential commenters, it might be useful to have a guide to effective commenting/replications, though it would be mostly common sense i.e. stick to the point, don’t add extraneous criticisms, don’t discuss motivations, assume good faith, don’t overreach etc.
Funding for this journal would need to be thought about. While #openaccess for the submissions would be ideal, page charges would be a disincentive to commenting, but might be modest enough for short contributions to not be too much of a hindrance. Author typesetting using Word or LaTeX templates might reduce costs further.
So, what do people think? Is this worth pursuing formally? Anyone want to sign up for the future editorial board? If you are a publisher, is this something you’d support?
Perhaps it should go without saying, but there is an obvious branding advantage for any publisher that takes this on (with the possible exception of the World Health Organization). I mean, wouldn’t you want to be the publisher or society that CARES?
References
- G. Foster, J.D. Annan, G.A. Schmidt, and M.E. Mann, "Comment on “Heat capacity, time constant, and sensitivity of Earth's climate system” by S. E. Schwartz", Journal of Geophysical Research: Atmospheres, vol. 113, 2008. http://dx.doi.org/10.1029/2007JD009373
- G.A. Schmidt, "Spurious correlations between recent warming and indices of local economic activity", International Journal of Climatology, vol. 29, pp. 2041-2048, 2009. http://dx.doi.org/10.1002/joc.1831
- "Post-publication criticism is crucial, but should be constructive", Nature, vol. 540, pp. 7-8, 2016. http://dx.doi.org/10.1038/540007b
On the one hand, sure. But on the other: I’d much rather have my discussions/debates embedded along with the original subject matter.
[Response: Agreed. But that doesn’t work most times. – gavin]
I like the idea. If the review takes a few weeks that will make it less attractive for people with non-scientific interests and give people time to cool down and write a more professional review. I do not expect it to go down to the level of blog comments.
Also for the authors themselves such a journal would be helpful. After publishing an article on benchmarking homogenisation algorithms we noticed some weaknesses of the validation dataset. Not big enough to write a comment or a new article, but big enough mention it at several conferences later and for the people who were not there to write a blog post about it. Had such a journal existed I would have made my blog post a bit more formal and submit it there:
Self-review of problems with the HOME validation study for homogenization methods.
Making short comments is also quite easy nowadays with the software of Hypothesis, which is also used by Climate Feedback for reviewing journalistic articles. It would only need an additional moderation option.
Next to publishing comments this way, every scientist (or small group of scientists) could easily start their own journal that way. No need to wait for submissions, you can start with reviewing the important articles published in other journals. Once that establishes that you do a good job in reviewing, people will be willing to also directly submit to your journal or in the end simply publish on Arxiv and ask for a review. For some first ideas see my post:
Grassroots scientific publishing
I also like this idea. It gets around a number of issues with the current peer review system. A big advantage would be the ability to get something on the record quickly and have a meaningful back and forth if its controversial.
The other thing it does is get around a problem I see too often, namely, in some cases current peer review practices can serve as a way for senior researchers to protect their own interests or flawed methods. I’ve seen this more than I think is ideal. It is I believe not a new problem, but is amplified by the strong association of career success with publication numbers and success in obtaining soft money. As you mention Gavin, people often say, “So what if that series of papers is wrong or misleading, people will figure it out eventually.” That’s not good for the reputation of the literature or science generally.
In general, we have been unable to replicate a lot of results from the literature but don’t have the time or motivation to publish them. In fact, there are career penalties for doing so if the result is public disagreement or controversy.
But would such a journal be viable? The reason it’s hard to get comments and replications/not published is that the existing journals have decided it isn’t what their readers want. They may be right. Think of the worst case, where all replications are successful. Good for science, but who would pay to read it?
Nick, I’m not so sure about what readers want. Certainly, the big institutions that really pay for the journals have a real interest in an accurate literature. Government is the largest such institution, but private companies might also be interested. I would argue that part of the problem here is the use of the literature as an advertising and career advancement vehicle, which has become a fine art. I know people who don’t read others papers because they assume that they are as much about advertising the other persons work as their own papers are.
I’d think about the acronym.
The other variation would be CARiES …
I’d participate in this journal and would be happy to be on the editorial board. I’ve left several PubPeer comments and do wish that site were more popular among earth scientists. I’m about to leave another comment there on one of my own papers – errata due to a typo.
This sounds like a great idea!
I wonder if CARES could be structured as a PubPeer overlay (on analogy to arXiv overlay journals in math, physics, etc. e.g.). So to submit a CARES article, it would first need to be posted as a response to the relevant paper on PubPeer, then go through the CARES review process to become endorsed as a CARES publication.
I can’t recall how many times I’ve been reminded that replication of someone else’s work, while certainly worthwhile either in lending further strength or accuracy to earlier work, provides almost no advancement for researchers.
Often, it is almost as much work as the original research required, and perhaps even more work, and if the results confirm the earlier work then at best it might go into a 6th rate publication somewhere.
Sure, if the work disputes earlier work then its own importance I suppose would be consistent with the importance of the earlier work their paper challenges. But it still doesn’t show that the researchers are capable of unique, imaginative contributions to the field. Instead, it just shows that they can poke holes and find fault. It’s not going to advance the individuals involved that much, though it might injure others. (Which gets to your point about needing some extra scrutiny about motivations for submitted papers.)
I think the idea is going to be a long road of learning where the sweet spot is in making it work well for all concerned, and a hard sell along the way, besides. It’s needed, no question.
I’d like to see some professional credit (though perhaps not as much as adding truly creative, novel contributions — whether the results are negative [that 6th rate publication again] or affirmative [chances at some fame, perhaps]) go for those either confirming or refuting difficult to challenge results. If the results of the effort improve the accuracy or breadth of application for earlier work or otherwise refute important parts of important earlier work, then that’s worth something substantial.
Not sure how that plays out in the end, though. I’d imagine that everyone has a limited life span and limited resources and replicating the work of others is pretty much always going to figure low. On the other hand, this might be a great option somewhere along the way towards becoming proficient and sufficiently comprehensive to be independently productive. Where along that path, I’m not sure. But perhaps.
Regarding the idea that “‘everyone’ already knows” (or doesn’t), it seems to me that anyone staying current, productive, and comprehensive within their profession should probably know. If they don’t, then that’s a problem for their own work, isn’t it? (Certainly, those who are in a position to guide research directions and/or periodically involved in making decisions about research funding.)
Factoids are seldom found growing in the wild.
Fact-checking popularizations of published science, is as important as examining the references the originals cite, as video can ampify and exaggerate not only the errors of the authors, but whatever misinformation screenwiters and producers electively add.
https://vvattsupwiththat.blogspot.com/2016/10/ad-john-podesta-ex-masters-of-disaster.html
Jon, It is I think very easy in most fields to find very interesting issues with current work. Smart scientists would not just pick things at random to replicate. They would choose things that seemed questionable or things that the person knew to be wrong or misleading. I would think the forum suggested by Gavin would be particularly useful for senior people who knew what the interesting issues were.
The idea that replication is just turning the crank is narrow minded. I would argue that many disputed areas are by definition interesting in their own right and provide people ample opportunity to demonstrate professional creativity and innovation.
In any case, this should not just be about helping individuals in the career rat race. Especially for senior people, there should be larger goals, like keeping the literature honest to maximize its usefulness for mankind as a whole, preserving the reputation of the field, and trying to raise standards. I personally know of several large interesting issues where even a small amount of work would clarify issues that are vaguely known by lots of people but where quantification and documentation is lacking. Just honest comparison of methods, algorithms, etc. is often lacking but of value.
How would publishing in CARES be different from writing a comment for the South-East Australian Journal of Theoretical and Applied Geo-sciences also willing to publish nearly everything. (Apart from the Pubpeer-like browser plugin).
In other words, is the problem getting published or the work to get your replication to a level that you can publish it? (And that this work in not valued, while the neo-liberals created a Soviet-style fake-competitive system to keep scientists stressed.)
If that is the problem, should we present it like a journal (with doi’s) or make it more informal, more like a blog post? Maybe even with the option of publishing anonymous? That way the authors would feel less pressure to produce a high-quality work, which takes time.
Just some ideas to provoke discussion. Where is the problem?
The problem I perceive is that it is too difficult to publish negative results and replication failures. In my case, the result is simply that these results were not published. That results in an inaccurate and misleading literature, where flawed methods and ideas continue to persist and fool outsiders. I think Victor, your question assumes that the literature is trustworthy and mostly correct. I believe that to be wildly inaccurate. The literature is often used as an advertising mechanism, a career advancement tool, and a way to ensure the funding stream. The tendency is to publish only “good results” and file in the desk drawer less appealing results. Often, older researchers are very aware of these problems, but don’t have the tools to try to improve things.
David (#11), I don’t disagree with the spirit and I wish the idea success.
I agree that replication isn’t just “turning the crank.” I’m an engineer who has been lucky enough to turn scientific ideas into novel instrumentation. Which means I work with scientists. Replication is almost all I do for my livelihood. And so I know from personal experience just how hard that also is. I do not need to be told. I also understand that to not only replicate (as an engineer might try), but to do so in a scientifically and comprehensively meaningful way? I respect the idea.
The point isn’t what I think about it, though. It’s what scientists themselves think about it. And then do about it. I would love to see the idea take flight and sustain itself well. Perhaps it will. I can hope.
But if you asked for my thoughts, based upon decades of developing commercial and scientific instrumentation (some used to instrument space flights) along-side folks who are active scientists?
Oh. Forgot. I already gave my thoughts.
In sum, I don’t have any substantial disagreement with the way you wrote. I’ll be curious as you, perhaps, to see how it goes. I’m an outsider, though. So all I can do is watch and replicate, from time to time, when allowed the opportunity to have such fun. (And it is a lot of fun. I enjoy my work very, very much and would do it for free, if others didn’t pay me.)
There might be a replication crisis is other fields, but I’m not convinced that this is an issue for the geo-sciences. I don’t want to entertain possible solution to something that might not be a problem. I notice a decent amount of replication in the literature, like whenever you see “… consistent with previous studies” in a paper, even though replication is never the primary motivation of the study.
I had a comment in review in a reputable, high-impact, journal for ~5 months before getting it rejected. One of the problems I feel is that editors tend to send comments out for review to the ORIGINAL REVIEWERS of the paper – who tend to take the comment as an affront on their own peer-review process, thereby being unduly critical. Considering that most of my PhD was on replication, I would very much be interested such a journal!
David says: “think Victor, your question assumes that the literature is trustworthy and mostly correct.”
Do enlighten me where I assume such?
Depending on whether I have a good or a bad day I would guess that 20 to 40% of papers has errors. Fortunately, most are likely not consequential for the main conclusion.
If it works, I think such a journal would be very helpful. To make sure it works we have to have the right format. Thus we need to have a candid conversation where the actual problem lies.
I do agree with Walter Hannah that I am not convinced that there is a replication crisis in the natural and geo-sciences. Everything is connected to each other by theory, studies building on others are implicit replications. But I am not a climate “skeptic” with only two categories “nothing happens” and “catastrophe”. Also without a crisis, a better organization of science can ease scientific progress.
The underlying problem is that that purpose of publishing is partly to establish one’s status in the particular field (the ideal purpose is to advance knowledge, but we are all human). Replications and, especially, responses are seen as cheap publications. It is, therefore, “unfair” to give any status points for publications that are too easy. It also is “unfair” to allocate journal space and reviewer energy.
Moreover replications and responses can produce “undo” uncertainty for the journal output and for the particular field. The powers-that-be in a field are not likely to encourage attacks on their work.
One way around these problems is to have journals that only publish replications. Economics is doing that, but as far as I know it is rare in other fields.
The bottom line, in my opinion, is that if replications and comments are hard to publish, then there is little reason to trust the body of research in the field. That is especially true of recent publications.
So essentially a scientific wikipedia?
People can buy the science they want today and ignore what they do not like.
The Global Restructuring of Science outtake
https://youtu.be/2J13SDqmaNw?t=33m15s
OK, Victor. Papers with outright “errors” are not the problem in my view. The problem is that the literature is strongly skewed toward “positive” results or results that agree with current consensus thinking. So in my field, people will run their code or method until they get a “good looking” result and publish that, forgetting about all the other runs that yielded less convincing results. There are ample examples in geosciences where there has been a serious problem, for example, with the use of uniform priors. These issues where difficult to straighten out. That is not an example of “errors” but just lack of statistical expertise and input. James Annan and Nic Lewis have done good work here, but much of the old literature remains uncorrected. That’s a problem I think.
John Kirwin wrote a few days ago “Regarding the idea that “‘everyone’ already knows” (or doesn’t), it seems to me that anyone staying current, productive, and comprehensive within their profession should probably know. If they don’t, then that’s a problem for their own work, isn’t it?”
I strongly disagree. If you are a scientist at a big name school and are well-connected to the field, then you are likely to learn if a certain paper or scientist cannot be trusted. But what if you are not so well-connected? Or what if you are moving into an area that is new to you? What if the paper is tangential to the main line of your research but has a result that is important for your research?
I don’t know about the specific solution proposed in this blog post, but I do agree that more needs to be done.
D 21,
I have never yet discarded a result because it wasn’t what I expected. I looked for errors when that happened and sometimes I found it. But the charge that scientists routinely publish the good stuff and suppress the bad stuff is pure slander. Rhine may have done that with his psi experiments at Duke, but I have known physicists and planetary astronomers and many others, none of whom would ever stoop to such a level. Produce your evidence or withdraw your charges, sir.
Barton, The way this works is obvious to anyone with experience in modeling fields and is well documented.
http://www.economist.com/news/science-and-technology/21707513-poor-scientific-methods-may-be-hereditary-incentive-malus
Building a model is a constant process of tuning and improvement. Scientists are often quite honest but biased in their evaluation of their models. One can always chalk up poor results to inadequate tuning, a “bad” grid, inexperience with the model, etc. The supply of witches to be burned is endless. These rationalizations enable a modeler to publish only the best result of a long series of “experiments” that produced less convincing results because all the previous runs were “flawed” for some reason. Yes, they were flawed in that they agreed with the data less well.
This is classic positive results bias. It’s very well documented in medicine for example where preregistration of trials since 2001 by one large funder of clinical studies saw a reduction in the rate of positive results from over 50% to a disappointing single digit number.
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0132382
Another classic example is turbulence modeling. Before roughly 2015, for 30 years you saw results in CFD published using various models with various versions of the model constants and functional forms. It is indeed remarkable how negative results appear when “improvements” are published that seem to have escaped publication before. This is not really science. More interesting is the sensitivity of the results to the model forms and constants. There are now a couple of recent papers on this in a literature that encompasses 100,000s of papers using the models over 35 years. If you are really interested I can generate some references. It’s common knowledge in the field.
I hate to be dismissive, but everyone who is senior in fields like CFD knows and will admit privately that the literature is biased and that they “select” their best results to publish. That’s just human nature in a highly competitive field.
Perhaps you are different and I will take your word for it. If you are in an experimental field, the situation is a lot better because employers know full well about the errors and biases in the data and expect them to be taken into account.
I will not withdraw anything because my assertions are true and in fact are common knowledge. Science needs reform if it wants to maintain its “special” status in the Western world.
For David, who is concerned about scientists, even if well-meaning, publishing only the best results:
So what are your thoughts about the successes of climate models in predicting the future? If you don’t know about these, you can start with the archives of this blog.
24 David says: “Science needs reform if it wants to maintain its “special” status in the Western world.”
REFORM Politics and the MEDIA and RELIGION first please, and then we’ll see how bad the situation REALLY is in science after that.