Guest post from PubPeer.com
The process of reviewing published science is constantly occurring and is now commonly being called post-publication peer review. It occurs in many places including on blogs such as this one, review articles, at conferences around the world, and has even been encouraged on the websites of some journals. However, the process of recording and searching these comments is, unfortunately, inefficient and underused by the larger scientific community for several reasons: To successfully impact the publication process, this database of knowledge has to accomplish two important tasks. First it requires participation by a large part of a given scientific community so that it reflects an average impression instead of an outlier’s impression. Second, it requires that the collective knowledge is centralized and easy to search in order find out what the community collectively thinks about an individual paper or a body of work. A recent initiative, the San Francisco Declaration on Research Assessment (DORA), echoes many of these same concerns.
In an attempt to assemble such a database, a team of scientists, have put together a website called PubPeer.com that is searchable and encourages participation by the larger scientific community. With a critical mass of usage an organized system of post-publication review could improve both the process of scientific publication as well as the research that underlies those publications.
Those of us involved in the creation of PubPeer.com believe that in an ideal world, a scientist’s main goal would be to discover something interesting about the world and simply report it to other scientists to use and build upon. This idealistic view of the scientific process is however not matched in reality because, for academic scientists, our publications count for much more than a simple contribution to the scientific record. For example, the majority of candidates are eliminated from consideration for tenure track positions at a major universities based on the names of the journals that have published their recent findings.
Review committees use this method because publications are the best measure of past and potential scientific output, but by potentially overvaluing “high impact” journal names, these committees and study sections effectively defer to journal editors to help them choose the best candidates for jobs and grants. However, these journals select their articles based on more than just good science – the papers also need to be of ‘wider interest’ and this can sometimes skew the publications towards ‘exciting’ results over those that are more measured, and perhaps more likely to be correct (for instance). The sometimes disproportionate attention given to a high profile paper also makes it a tempting target for more unscrupulous scientists.
It’s never going to be possible for us to thoroughly read all of the papers submitted to a job advertisement, nor all of the papers referenced in grant applications, but we can easily reduce the importance that journal names play in decisions and replace it with something that is more meaningful and directly in our hands instead of the hands of publishers. After reading any publication, we all have impressions about whether the reported observations are useful, interesting, elegant, irrelevant, flawed, etc. If a particular scientific field that is interested in a given publication were able to compile all of it’s impressions of that publication, that collective information would be infinitely more useful to search committees and study sections than the name of the journal in which it was published.
Outlined below are a few aspects of PubPeer.com that differentiate it from the current post-publication review systems and which will hopefully make it more widely used.
- A key issue that we have decided on is the importance of anonymity. One of the reasons that we have never commented on articles directly on journal websites is because the colleagues whose publications we are most qualified to comment on are likely reviewing our publications and grant proposals. Even the most well-intentioned criticism could potentially irk these potential reviewers. Since publications are so precious to everyone’s future career advancement, there is a huge psychological barrier for early stage scientists to attach names to any comments that could be considered critical.
Therefore, in order to encourage as much participation in this post-publication review process as possible, PubPeer allows comments to be left anonymously if someone is so inclined. Critics of this feature sometimes email us to point out that anonymity allows for baseless slander or to proclaim that a commenter’s name is essential for judging the validity of a comment. We strongly disagree with this second point because good comments are good regardless of whether they come from a senior scientist or a graduate student. We can all judge for ourselves the content of comments and on PubPeer it is possible to vote the good comments up and the bad comments down into the noise so that community as a whole can decide together what is worth paying attention to. Baseless defamation, rumors, and ad hominen attacks are not tolerated at all and are immediately removed from the site.
The people involved with PubPeer are all active scientists and we are trying to remain anonymous for the time being for several reasons: 1) we can imagine scenarios in which pressure could be put on us to remove/alter comments if our identities were known and 2) we would like to protect our families and private bank accounts from the more litigious among our readers.
- A main drawback of the current practice of post-publication peer review is that the reviews can be spread across many different blogs and journal websites. If one wants to know what the community thinks of a given body of work (whether it be a discipline, an author’s output, a university department, etc.) it takes a major time investment to track down the information from all of these different sources. PubPeer provides a centralized and easily searchable database that contains comments on all published articles.
- PubPeer also provides for a system of alerts. In order to be effective, authors and others interested should be able to be alerted to comments on their favorite publications or topics. Pubpeer automatically notifies corresponding authors of new comments on their articles and anyone can set up email alerts on articles they find interesting.
We’d like to thank Realclimate for this invitation to explain PubPeer and we welcome any suggestions, comments, criticism, or questions on the contact page pubpeer.com/contact or in the comments section below. Many conversations have already started on the site both with and without author responses that are sometimes quite detailed (a list of all of the most recent articles receiving comments can be found at pubpeer.com/recent). Some comments have already led to important corrections in high profile studies (see also Nature News, Science Insider, Labrigger). We hope that this can be even further expanded in future.
Excellent initiative, I hope is is very successful.
Can I suggest an extension? Writing comments papers ought to be an important part of the post-publication review process, so that where a high-impact paper is found to be flawed, there is a peer-reviewed comment that explains the errors. It would be very helpful if PubPeer.com could also facilitate crowd-sourcing comments papers (which are time consuming to write). Having a forum where the comment could be written collaboratively in the open, where the author of the original paper could engage with the process, would avoid duplication of effort and for papers on contentious topics (such as climate change) demonstrate that the comment paper was scientifically motivated and that the original author had been given an opportunity to put forward their objections before the comment were submitted.
“First it requires participation by a large part of a given scientific community so that it reflects an average impression instead of an outlier’s impression. Second, it requires that the collective knowledge is centralized and easy to search in order find out what the community collectively thinks about an individual paper or a body of work.
Average impression and collective thinking – the paradigms for climate science?
Jack Maloney – it is pretty difficult to have post publication peer review without the peers getting involved, the more the better! ;o)
I first ran into the PubPeer site a year ago. It looks like it is mainly for medical research. To register, you apparently have to have a paper with your name in the database, i.e. the qualifications to be a peer. Unfortunately, when I put my last name in the registration box, the only response is a paper by my MD sister-in-law. So, in effect, not many people can get in there to boot-strap the database away from a medical research focus.
The PeerPub people want comments so that is my 2 cents worth. I would have been using it a year ago, if I could figure out how to register as a peer.
Jack Maloney,
Methinks someone doesn’t understand scientific consensus…or anything else about science for that matter.
Here’s a hint. Any one of us can be wrong. Einstein was wrong about the cosmological constant…twice. He was wrong about quantum theory. And yet somehow, science has marched on around him. Now how do you suppose that happened.
Science. It works. Try it.
I have nothing but applause for post publication peer review. But pre publication peer-crowd review, as in Arxiv in the case of Physics, is also a great thing. Circulation of unpublished papers and a system for submitting comments about them allow authors to discover errors and shortcomings, to collect suggestions for improvement, and to have early warning about dissenting views. All do circulate their working papers with some colleagues and friends, but wider consultation with the scientific community would be much better, and would also facilitate the process of publication. Peer review is undoubtedly evolving in the Age of Information, and it is a wonderful thing to watch.
I tried a quick search. One might consider categorization for discipline or subject matter as well as other filtering aspects that are relevant?
Has anyone else played with it yet?
The biggest negative of this is that it’s taken 40 after the initial development of the Internet and 20 years after the introduction of the World Wide Web for this sort of forum to come into existence. For all of the hype in tech circles about ‘innovation’. (And I am in such circles) it’s these sorts of extremely obvious initiatives that need to be undertaken to advance humanity, not yet another way to combine micro blogging, personal photos and mobile devices. (not that there’s anything wrong with personal photos. But the disparity in resources put into such commercial activities versus those put into the basic ‘infrastructure of human understanding’ is orders of magnitude. And the triviality of many of the developments are disguised by the immense amounts of money involved and a non-critical group-think of buzwords, such as the above mentioned ‘innovation’.
I have a handful of published papers but none are in the database when I search for the DOI’s. This is because my papers were not published through PubMed but through Springer. My understanding is that PubPeer is predominately for medical papers with a vey limited number of exceptions. My field is Plant Pathology. It’s a shame PubPeer doesn’t have a wider usefulness because it’s a good idea.
Jack Maloney, your ignorance is on display.
Another good extended peer review website is Journal Lab, founded by a good friend of mine: http://www.journallab.org/
It has the benefit of focusing discussion around figures in papers, though right now its mostly focused on the medical field.
Having been involved in a group reply, allow Eli to say that while it was fun, there were issues, mainly it was very hard to get everyone to speak with one voice, so group replies are going to be difficult.
Other than that, getting a better search mechanism as some have said (e.g. by field, etc.) would help a lot.
Related but on a side note, NPR recently did a big story on the new literary term of CLI FI, for climate fiction, which I coined in 2007 during my work with polar cities, which still goes on. I recommend you google “NPR + Judith Curry + CLI FI” to see how NPR introduced this new subgenre of sci fi to the world. COOL! or hot?
Thank you all for checking out PubPeer and for your suggestions. We really hope that it is used and that it helps the community.
We hope that this will be one of the main uses of PubPeer. Realtime review involving authors is exactly what we wanted.
We would also love to see it used in graduate seminar courses where literature is reviewed by graduate students. It could be an excellent way to engage students by showing them that their literature review is taken seriously by the community as well as the authors.
We agree. We have implemented arXiv references into PubPeer as well. Anything in the arXiv can be commented on PubPeer as well. We will roll that out in the coming days.
We are trying to accomplish this via search. e.g. typing “climate” into the search bar will display all commented papers containing that category. This avoids the problem of requiring us to also subcategorize each comment (e.g. “climate” AND “trees”). The user can decide for themselves which categories to filter based on search terms.
Before this post went up on realclimate we worked hard on making sure we have every article published with a DOI. However, we apparently didn’t make this obvious enough and will fix that soon. By pasting the DOI of any article ever published into the search bar you should be able to find your articles no matter what the discipline. After this first DOI discovery the article will then be searchable/sortable as described just above. Please let us know if this is not working for you.
This is a very good point. Thank you for pointing it out. We will immediately make it so that DOIs can also be used for account creation. This will make the search for your articles more specific and extend the account creation to ANY published article (outside the biomed disciplines). Check back soon.
I’m not sure how revolutionary this is. If a paper is really useful, it’s results and methods will disseminate pretty quickly into the community. This is just doing on line what previously got done by coffee urns and at conferences.
@ Ray, #14
“I’m not sure how revolutionary this is. If a paper is really useful, it’s results and methods will disseminate pretty quickly into the community. This is just doing on line what previously got done by coffee urns and at conferences.”
This is kind of the point of PubPeer- it’s the internet version of a journal club. Now, I don’t think that it will fully replace the need for having conferences and face-to-face interactions; esp. as science gets more and more complex. What PubPeer will do is put the community reaction into the record, and make it easy for policy makers to see what peers think of a particular article. As one might imagine, there is some resistance to the idea of “losing control of the conversation”.
I do not see PubPeer as a loss of control, I see it as science correcting itself and returning power into the hands of the community from that of the “leadership”. Recall that the “leaders” in science are usually just the folks who are better at talking to non-experts than other folks. The leaders were never indented to control a field, they were intended to help explain it to everyone else.
Also, just to comment on people who have noted that PubPeer is mostly focused on biomedical papers at the moment: this is due to the fairly high amount of non-replicable results that the field has experienced over the past couple decades.
There is an extremely large amount of money involved in biomed research as well, making it very difficult for scientists- young and old- to run counter to the status quo. There are a very large number of biomed researchers worldwide; however biomed journals- the primary mechanism for “getting the word out” currently- are mostly run by the USA. In the USA, NIH funding is usually dispersed to those who publish in “high impact” journals. However, “high impact” does not necessarily correlate with “good science” any longer; it’s turned into a bit of a popularity contest over here. (I’m a chemist by training, and our funding system in the USA is a bit different than that of biomed. Esp. in regards to conflicts of interest.)
As PubPeer mentions, I am sure they will do their upmost to include other disciplines in the near future. Biomed is at a critical juncture right now. A platform like PubPeer is needed given the large amounts of money involved in research and the current global economic situation. (I am working on making the USA universities have transparent budgets for biomed grants. As an American, I have a strong desire to see exactly where my taxes are going.)
Just after finding out how difficult it is to criticise a high-impact article I tried to register. Two of my articles did show up. However, my institutional e-mail addres (knmi.nl) is not recognised, so hte service is not yet useful for me.
Regarding Ray Ladbury’s comment at 14:
It is easy to confirm Ray’s view. Go to scholar.google.com and put in an author’s name. Try D.K. Hall; she’s a prolific and highly respected.
Pick any one of her papers and then follow the citations of that paper by others by date. It is amazing how quickly a paper is cited by others. Equally amazing is how long a really good paper persists as a benchmark.
Try this one: “Development of methods for mapping global snow cover using moderate resolution imaging spectroradiometer data”. Cited by 451 other authors.
Thanks for you comments and suggestions both here and on our contact page. We have fixed the issues below:
Sorry about that. You should be able to register now as we expanded our list of recognized institutional addresses.
Any article with a DOI can now be used to create an account. Simply paste the DOI into the search bar during account creation.
Pub Peer reviews may be as subject to Sturgeon’s law as published papers .
> disproportinate … unscrupilous … it’s …
May I suggest:
Everything — everything — written that you intend to go out under the name PubPeer — everything that is meant for public reading (searching and indexing) — should go through a couple of rounds with your competent copy-editors. Do this before making things public.
Having done that in draft, thereafter, once you go public, never delete anything: use strikeout and underscore to show changes made in online material and explain the changes.
_________
Those using Word with spellcheck should, um, think about the risks and benefits.
Make the process as competent as the work you’re reviewing.
___________________
Yes, doing this is tedious, but less tedious than catching up with belated corrections.
@20 (WebHubTelescope), Could you give a pointer to your tabulation of contrarian “models”? This would be a convenient reference.
can anyone here tell me the “direction” of all the major climate forcings right now? i’m talking about things like ENSO, the PDO, the AMO, solar radiation, and any other forcings you can think of?
i’m trying to figure out if this recent slowing-down of warming could be because all the forcings are negative (except, obviously co2). is it possible or likely that this slowing-down of warming would be an actual cooling period if not for co2?
walter crain @22 — During and after an El Nino event temperatures rise. During and after La Nina temperatures should fall (except for ever increasing CO2). Nothing else matters much.
thanks david.