Science is self-correcting: Lessons from the arsenic controversy

Recent attention to NASA’s announcement of ‘arsenic-based life’ has provided a very public window into how science and scientists operate. Debate surrounds the announcement of any controversial scientific finding. In the case of arseno-DNA, the discussion that is playing out on the blogs is very similar to the process that usually plays out in conferences and seminars. This discussion is a core process by which science works.

The arseno-DNA episode has displayed this process in full public view. If anything, this incident has demonstrated the credibility of scientists, and should promote public confidence in the scientific establishment.

The story begins with a long-standing scientific consensus backed by an enormous amount of data: DNA is made with a phosphate backbone. Alternative backbones, such as arsenate, have long been considered unlikely for theoretical reasons.

Nonetheless, despite this consensus, reputable scientists have promoted the study of alternatives challenging the prevailing view. And NASA has willingly funded these studies.

Lesson one: Major funding agencies willingly back studies challenging scientific consensus.

The research team, Felisa Wolfe-Simon and colleagues, behind this study collected data and concluded that they had sufficient evidence to demonstrate incorporation of arsenate into bacterial DNA. Although the data were preliminary in nature, Science accepted the manuscript (pdf). With a high profile, potentially groundbreaking paper about to be published, NASA announced a press conference to publicize the findings.

Lesson two: Most everyone would be thrilled to overturn the consensus. Doing so successfully can be a career-making result. Journals such as Science and Nature are more than willing to publish results that overturn scientific consensus, even if data are preliminary – and funding agencies are willing to promote these results.

Within days of the arsenic paper’s publication, strong criticism of the study began to appear on scientific blogs. These blogs attracted the attention of the mainstream scientific press. Soon thereafter, media reported the wide skepticism within the scientific community – with some scientists going so far as to say that the paper should not have been published.

These scientific criticisms opened the door to those wishing to discredit science and the peer-review process, with the contrarian blogs suggesting that this study demonstrates that peer-review is “broken”. A comment on Watts’ blog summarizes their thinking:

It’s amazing how fast the scientific community came out to attack NASA for what they claim is plainly flawed science. Then again, NASA isn’t funding any of the attackers.

In the Climategate mess however, we still have heard very little from an awful lot of so-called scientists who should have been saying a lot more about flawed science but are too afraid to lose their grant money.

This raises an interesting question: just who is critiquing the NASA study? It turns out that many of the critics are also NASA-funded. In fact, many prominent critics of this study are funded by the NASA Astrobiology Institute – the very same program that funded the arsenic study.

Carl Zimmer gives us several examples:

• Norm Pace offers the critique: “Low levels of phosphate in growth media, naive investigators and bad reviewers are the stories here”.
• Shelley Copley suggested, “this paper should not have been published”
• Roger Summons remarked that a critical experiment was left undone, and backed the critical blog analysis of his NASA-funded former student.
• Michael Russell agreed with blogosphere critics, and offered his own critique of the study based on cosmic ratios of phosphorus to arsenic. Russell is a member of the Astrobiology Institute, as well as an employee of NASA’s Jet Propulsion Laboratory.
• Forest Rohwer observed, “the experimental evidence in the paper is pretty weak.”
• George Cody says he “cannot accept this claim until such an experiment [mass spectrometry] (easily done) is performed.”
• Steven Benner was an early skeptic. To NASA’s credit, they invited him to present his criticisms at the press conference. He has said “we are not expecting this result to survive”.

Each of these scientists is affiliated with NASA Astrobiology.

Lesson three: Scientists offer opinions based on their scientific knowledge and a critical interpretation of data. Scientists willingly critique what they think might be flawed or unsubstantiated science, because their credibility – not their funding – is on the line.

Regardless of whether or not ‘arseno-DNA’ survives the test of time and further study, scientists have shown that they will rigorously criticize science perceived as flawed, with no fear of reprisal from funding agencies.

This is the key lesson to take from this incident, and it applies to all scientific disciplines: peer-review continues after publication. Challenges to consensus are seriously entertained – and are accepted when supported by rigorous data. Poorly substantiated studies may inspire further study, but will be scientifically criticized without concern for funding opportunities. Scientists are not “afraid to lose their grant money”.

Finally, there is the issue of how scientists who publish papers that generate credible blog reactions should in turn react. In times past, it was simple to wait for properly crafted letters and comments to be sent in to the journal. This gave fixed targets to deal with and allowed for considered reflection and response; discussions would perhaps be published 6 months to year later. But today, serious criticisms can arrive immediately (as seen above). Nature (perhaps with a little schadenfreude) had an op-ed suggesting that the authors on this (Science) paper should be more strongly engaged in the reaction, while Science had a plea from the lead author for a little patience, since they were clearly a little overwhelmed.

In our view, this needs to be thought about clearly on a case by case basis. Some criticisms (that for instance accuse the authors of deliberate fraud or misconduct based on a dislike of the conclusions) are not worth rapidly responding to, but it is worth trying to head off any misinterpretations that might be emerging. Short form papers (even with copious supplementary information) do not provide full context for the results in themselves, and so putting together a response to frequently asked questions is certainly useful (as Dr. Wolfe-Simon and colleagues have). This doesn’t replace the need for technical commentary to pass via the peer-review process though. In the end, that is what people will refer back to.