Despite what is often claimed, climate scientists aren’t “just in it for the money”. But what scientists actually do to get money and how the funding is distributed is rarely discussed. Since I’ve spent time as a reviewer and on a number of panels for various agencies that provide some of the input into those decisions, I thought it might be interesting to discuss some of the real issues that arise and the real tensions that exist in this process. Obviously, I’m not going to discuss specific proposals, calls, or even the agencies involved, but there are plenty of general insights worth noting.
Scientists submit proposals to the various agencies mostly to cover a small part of their salary (summer months for most US university academics), or to employ postdocs, train graduate students, buy equipment or support logistics for work in the field. Some scientists are 100% ‘soft money’ meaning that they cover all their salary from grants, but it’s important to note that salaries are fixed by the home institutions – you can’t write a grant to pay yourself double what you got last year for instance. Many scientists get by without submitting grants at all (those with so-called ‘hard money’ positions – like university lecturers or government researchers), and if they do, it is often to support other people. For each person being supported by a grant, you have to budget for their salary, fringe benefits and the mysterious ‘overhead’ (some 30 to 60% of the total) that gets taken by the institution. Once you add in some travel and facilities money, most standard individual PI grants end up in the neighbourhood of $100-200K per year, and so for a 3 year grant, something around $300-600K. For fieldwork in Greenland for instance or for outfitting a new lab, the numbers can be substantially higher. This might sound like a lot of money, but the PIs never see this as a lump sum (it is a grant to their institution, not them personally), and as noted above, most of it disappears into the system (to fund necessary things of course) before it gets anywhere near the researcher.
Funding is highly competitive and, depending on the call, only between 10 and 20% of proposals will be funded. Choosing which proposals get funded relies on the good judgement of the program managers in the most part, but they are helped enormously by external reviewers and panels. Different calls can be very discipline-focused or rather interdisciplinary in nature, and both the proposers and the panel can have very diverse backgrounds and expertise. If used, panels consist of about 10 to 20 people (depending on a number of factors) who will meet for a concentrated few days of reviewing. For each proposal, someone on the panel is assigned to lead the discussion, and a few other people need to be able to discuss it in depth. There are also mail-in reviews from the wider community (anything up to 5 additional reviews). In typical cases, a panellist might take the lead on reading and analysing a few proposals, and have to provide additional in-depth input on a dozen more. That implies a couple of weeks of work to do properly. Over multiple days of deliberation, the panellists will review, perhaps less deeply, many of the other proposals as well.
In no particular order, here are a number of observations:
- At no time (in my experience) does anyone even hint that someone’s political position is in the least bit relevant to funding the grant. It just never comes up.
- Over-egging of the importance of their proposal by proposers is commonplace, and is generally poorly received by the panels. People who claim that their particular bit of the field was important for some goal when, at best, that is debatable and, at worst, completely irrelevant, do not enhance their credibility. Proposers do need to demonstrate some reason for why they should be funded more than anyone else, but there is a fine line between necessary self-promotion and overselling. Note that any overstatements are usually related to the importance of an idea, not on the drama of any implications.
- No-one gets funded to demonstrate a specific result. People get funded to investigate questions.
- Having someone on the panel whose expertise covered the specific topic of a proposal can be polarizing. That is, such a proposal was either more likely to shine or be dismissed than a proposal for whom no-one is as intimately knowledgeable. Those tended to fall in the middle unless a really good case was being made. Mail-in reviews are particularly helpful here.
- Sometimes there are some real outlier reviews – either praising something clearly mediocre, or slamming something quite interesting. But panels do discuss this and it certainly isn’t the case that one outlier (for whatever reason) is the sole determining factor in a decision. Note as well that the panels are not the final arbiter – the program managers are.
- The discussions about science during the panels can be really good and are great at helping contextualise specific contributions.
- Reputation of the proposers as people capable of good science goes a long way to judging the feasibility of a proposal. Judgements are made all the time on whether the proposers can credibly complete the plan of work they had laid out. Someone can propose doing all sorts of wonderful things, but a demonstration that at least a big part of it is actually do-able by the people proposing is important. For newcomers to a field, that does put on an extra burden – but one which can be overcome with original ideas and sufficient proof of concept. It can help if collaborators are more experienced, but this isn’t essential.
- Conflicts of interest exist – proposals can come in from a previous student of a panel member, or a current colleague or close collaborator. However, in all such cases, the conflicted person is asked to leave the room and not participate in the discussion on that proposal. This works well in avoiding “less objective” criteria in funding.
- In interdisciplinary calls, there are a lot of single discipline proposals submitted (and they may rank highly in the reviews). But these proposals get reviewed very differently from truly interdisciplinary proposals and it is very hard to legitimately weigh the contrasting approaches. In my opinion, mixing up technical ideas with synthesis proposals in a single call is a mistake – synthesis projects need to be funded separately and on a level playing field.
Overall, I feel this process does what it is designed to do. Given that there are far more good ideas proposed than can ever be funded, there is inevitably some subjectivity and different panels would have different discussions and a different emphasis. I’m confident however that almost any panel, given the same input, will have a reasonable overlap among the highly rated (and therefore most fundable) proposals. Clearly, these methods work best when the proposals are similar or in a similar field, and will not work quite as well when there is a lot of diversity (because the judgements in those cases can be more subjective, and thus more easily swayed by random contingencies). There could always be improvements (shorter proposals might be easier to get reviewed by outside specialists, calls can be clearer about what they want etc.) but none of the problems are anything like the contrarian imaginings of hysterical climatologists trying to outbid each other in who can come up with the worst case scenario.
Gavin — Well done, once again.
In my experience as a referee, both write-in and on panels, the quality of the writing, the exposition of the proposal, made a substantial difference in whether or not the proposal received favorable reviews.
[Response: Indeed. We may put to together a post on tips to get your proposals funded – feel free to make some suggestions. – gavin]
Gavin,
Thanks for this. In my experience, the granting process is one of the murkiest aspects of science in the public mind. I think one additional point I would emphasize is that those reviewing a proposal (or a journal article for that matter) are often passionately interested in advancing understanding of the field. The ultimate motivation for all of this unpaid work is curiosity, so it makes sense to allocate funds in the way most likely to advance the field. In an ideal world this not only satisfies curiosity, but is most likely to open new avenues for the growth of the field.
People need to realize that even when it comes to money, science is still works best as a curiosity-driven enterprise.
Sorry, I’m not in the academic world; what is a PI?
[Response: “Principal Investigator”]
Your description matches my experience in a completely different field exactly. The one thing you didn’t mention is the enormous amount of work that has to be accomplished before the panel gets together. Maybe it is all online now, but it used to consist of a large box of applications in the mail that had to be read. This substantial extra time is considered to be a part of ones job. There is no pay for reading multiple grant applications night after night and dodging teaching responsibilities and important experiment timelines, which are much more important to the home institution. Steve
Gavin,
I largely agree with your summary, but I saw cases when a single panel member wrecked a proposal. Since, only two or three rapporteur read the full proposal the rest of the panel is often left to rely on what others said. I witnessed, when a single panel member relatively easily set the mood against a proposal. At 10-20% hit rate, it is enough, if a couple of panel members downgrade a proposal so it falls to the lower part of the top third and no vehement support from other panel members can offset that.
I find that putting proposals to three bins (good, mediocre, and poor) is easy. Ranking the top one third when only the top 5-10% gets funded is difficult and mostly subjective or the outcome is almost random at best.
Actually, I see the low 100-200K/yr budget a major problem. I don’t think substantial work can be delivered at this funding level. NSF expects that money partly fund grad students (who are a toss up at best in terms of producing results). I don’t think self promoting in proposals happens too much, but over promising is quite common. Sometime, I wish if funding agencies put progress and final reports through similar evaluation as they do when granting awards.
The 10% or less hit rate is also highly limiting. I am probably not the only one, who saw on panels some old fart ranked first just because of reputation. I remember sitting on panel where the rapporteur literally said, here is mediocre proposal from a well know team that has a good track record so it should be ranked high. Out of the 4-5 proposal that gets funded from a pool of 50, if only the first two slots are taken up by these that leaves very little chance for truly innovative proposals.
I think, panels have a tendency to go by the flow, therefore proposals that are unusual for whatever reason have better chance to run into some devastating objection at least from one panel member, than enthusiastic panel support. I don’t think that the final proposal selections are purposely biased, but this tendency of going by the flow definitely makes the outcomes fairly uniform.
Thanks, Gavin. You say “No-one gets funded to demonstrate a specific result. People get funded to investigate questions.” This statement should be redundant but going by comments on realclimate.org and other places there is a belief by some people that scientific researchers somehow already know the answers and do research to ‘prove’ these answers. (This erroneous idea ties in with misplaced notions of ‘common sense’ always giving the ‘correct’ answer.)
I don’t know about what happens in the USA, but here in Australia the capability of the institutions themselves is a big factor in addition to the proven track record of the lead researchers. A less experienced lead researcher backed by a noted institution will rate more highly than a less experienced lead researcher from a lower profile research institute.
Collaborations are often sought to overcome this (eg a low profile institution seeks to team up with researchers in a large, established university or state government agency). Collaborative Research Centres have been set up over the past few decades here in Australia, many being Centres of Excellence – to help make sure the ‘critical mass’ (of facilities, equipment and people) is reached for good multi-institution, inter-disciplinary and cross-disciplinary research. Global collaborations are also looked upon favourably for the same reason.
Having sat on NASA review panels I have always found that sitting on a review panel is a great deal of exhausting work. After leading the discussion on 3 or 4 proposals NASA requires that the “lead” reviewer write a discussion of the strengths and weaknesses of the proposal as well as the summary of the panel discussion. On more than one occasion I have found very compelling proposals competeting for the same very very limited funds making it very difficult to find recommend one versus the other. I have always felt it was something I should do for the good of the community, but dreaded all the very concentrated work effort that was required.
When I get the call from the program manager asking if there is away to reduce the proposal costs, I almost always end up cutting out my own salary entirely to afford paying for the graduate students.
This fall I taught a graduate level instrumentation course where I asked the students to write proposals to investigate urban heat island effects given a specific Request For Proposals (RFP). Essentialy I wrote the RFP as if we were going to run METROMEX all over again. Since it was an instrumentation course I recommended concentrating on the field portion and to concentrate on what instrumentation would be appropriate. The department faculty then acted as the review panel and I acted as the program manager. Not only was the response from the graduate students overwhelmingly positive, but the reponse from the junior faculty who had never sat on a review panel before was enlightening altering how the approached writing a proposal.
Thanks for printing this article. We needed something to refer people to when questions on this come up.
As a climate scientist, I’d say that
” if it is just for the money, then most climate scientists should have left from the climate research field”.
As Kerry Emanuel expressed in http://www.nature.com/news/2006/060717/full/news060717-1.html , ” Climate science is a very hard physical, chemical, mathematical, and now, to some extent, biological problem, and it requires people who are very talented in these areas.” If they were more interested in money, these talented people could be much richer as business-men, lawyers, or software engineers rather than as climate scientists.
My personal experience: I graduated from a top university in China: Most of my classmates staying in China becomes successful businessmen and they often feel curious why I am still doing research which can not make me rich; Almost all of my classmates except myself who have moved in USA becomes software engineers or are working in financial business, and they also feel surprised why I am still doing climate research which will never make me rich.
: ) As a climate scientist I can enjoy many beautiful things, but definitely not for money. Be proud of being a member of climate research community.
[Response: Nicely stated Jianhua, and I would add that it applies to scientists generally.–Jim]
wouldn’t it be nice if we had a gov that funded all projects.
The one way that the PI does financially benefit from getting a grant is that it counts for quite a lot (almost everything ;o) with university promotions committees. It isn’t true that the PI doesn’t benefit personally from grant getting, but it is very nearly true. Gavin’s observations are in good accord with my experience as a researcher and as a reviewer; it is the science that matters most (although good communication skills are required to make sure the reviewers appreciate what it is you want to do and why).
Good article. My world (that of a Program Officer at the NIH) mirrors what you wrote. The only thing that may be different is that the funding where I am is almost exclusively dependent on the scores from the peer reviewers. There may be some wiggle room at the boundary, but going by the peer review is the most defensible decision.
With the submitted-to-funded ratio the way it is, many worthwhile proposals do not get funded, unfortunately, which is why a track record in which the PI has demonstrated success (good results – being correct counts for something) counts.
And Jianhua Lu really captured why we go into science, to which I might add insatiable curiosity, the thrill of discovering something new and the joy of being surrounded by people with the same love.
Who sets the questions in’policy-relevant research? The policy-maker, the individual research or the research institution?
Climate research in UK and EU at least has long been regarded as policy-relevant and has been funded as such. When I was a researcher fully dependent on grants, the search for buzz words abnd policy-relevant proposals was continuous and time consuming.
In MHO climate research will only be trustworthy again if firmly separated from the ‘decarbonisation of energy supply’ agenda of governemnts keen to replace carbon fuels for a large variety of reasons.
In my experience, too much environmental research has turned into ‘environmentalist'(ideologically motivated) efforts because of this dangerous funding link to government or corporate policy (and hence money-making or competence enhancing).
[Response: Do you have any idea of what actually gets funded in climate science? Please look up the NSF grants on the subject and then come back. You may have thought that your proposals in social science needed to have ‘buzz words’ (though I doubt this is as widespread as you claim), but that kind of shallow grantsmanship plays almost no (or very little) role in science funding. – gavin]
Good to see a post about what your site is really all about – getting grants. You’re lucky in the US – the politics is only implicit – in that the peer reviewer’s will tend to support the status quo because that’s mainly the business they’re in also, and will not deviate too far from what they perceive the public will find palatable (because the public pays the bills).
Here in New Zealand the politics is much more in your face and certain people and certain ideas will not get grants, although occasionally something politically incorrect slips through.
PS is there any way out stuff like wasting $$BB of the taxpayers money on mini-reflecting satellites in the latest grant round?
[Response: Since this is a science site, perhaps you’d care to provide some actual evidence for you claims? I have seen absolutely nothing in my experience that would support your claim that peer reviewers of grants pay any mind to what the public thinks is ‘palatable’ – they are there to assess the plausibility, tractability and usefulness of the science. – gavin]
#10 Pattaya Girls: Funding all proposals would be a bad idea. Back in the dark ages when there were significantly fewer researchers and much higher success rate, about half of the proposals did not deserve funding–mostly they were good ideas but had not been adequately developed.
The ideal success rate in my experience is between 30 and 50%–this sharpens the proposed research but does not turn getting funded into a lottery activity. As many people above have noted, we are throwing out important research proposals that should be awarded at the 20% funding mark. This unfortunately has discouraged some very good scientists.
Bravo Sonja I missed your post before I posted, but that would be exactly what its like in my country also, even in the “hard” [non-“social” :)] sciences.
I was speaking mainly about what happens in my country. However I do know from my field (biomedical science) that the NIH wastes billions of dollars following hoax and quack leads in seeking medical cures – they wasted over a $1B on trials of shark cartilage to cure cancer even though it was well known to be a scam.
[Response: Sorry, I’m very sceptical of this. $1 Billion dollars is a substantial fraction of the whole NIH budget. Reference? – gavin]
It didn’t take long to find evidence from your own link: $147,946,598.00 on “Climate change education”. I would sympathize with your colleagues in the non-“social sciences” if they were more than a bit pissed about that if they missed out on really good grants. And I bet not a cent of that goes on “teaching the controversy”.
[Response: Again, you are making assertions based on your own prejudice, not actual evidence. You need to do better than that. – gavin]
A recent gem from my country is the $800K that got awarded to a researcher to see if breakfast is a useful meal. (no prizes for guessing what the answer will be).
These are bones that the funders feel they must throw to keep the public/politicians/lobbyists on board so they can go on handing out money for real science as well.
Come on you guys, you’ve been around for a while – you must know this goes on – funding in one small defined area may seem transparent and corruption proof but funding in general is obviously fairly political in your country/ies also.
[Response: It’s pretty clear at this point that you don’t have any idea what you’re talking about.–Jim]
Gavin,
I can’t resist on commenting on response posting#13 from Sonja. I don’t make statistics about what gets funded, but I have hard time to believe that a thorough evaluation of the grants that were funded in the last ten years would support your argument. I saw many proposals primarily focusing on some specific science question with respect to land surface hydrological processes peppered at the end with some bogus exercise using \downscaled\ GCM output. These \add-ons\ are often there to satisfy policy relevance and broader impact.
[Response: I’m certainly not claiming that every proposal is perfect or that people don’t graft less interesting ideas on top of quite good ones – that happens all the time. I strongly disagree that using downscaled GCM output is necessarily bogus (where does that come from?) – it will depend completely on the context. ‘Policy relevance’ is not a required factor for science proposals, so that’s irrelevant, and ‘broader impact’ can encompass anything from real-time data websites, to high school projects, to graduate student education, to lesson plans, to input to large-scale assessments. None of these things are ‘bogus’. – gavin]
Just out of curiosity, how many funding do you have, which does not have climate change mentioned in the project plan?
[Response: Currently, I’m a co-PI on the GISS model development proposal (which is most of my money), I get a small amount of funding for museum climate science education projects, and I fund a programmer and other colleagues to study isotopes in the present day water cycle. Historically, I’ve had grants to look at Holocene isotope variability, understanding 10Be in ice cores, carbon cycle modelling, the sensitivity of the NAM to various forcings. All of them are related to climate, but why wouldn’t they be? That’s what I study. – gavin]
[edit]
Sonja, if you’d ever done science or even bothered to actually look at how it is done, you’d realize that it is curiosity driven. No, one cannot count on the wisdom of governments to choose the right areas to fund, but science itself dictates to a large extent the problems that can and must be solved for progress to occur. I have actually seen requests for proposals be under-subscribed when researchers deemed the prospects for meaningful progress to be dim, or when they were not sufficiently “interesting”.
The thing you need to understand (but which I have little hope that you will) is that the trajectory of a scientist’s career depends to a great extent on his or her record of working on problems that are important to the discipline. This is one reason why “directed” research tends to be inefficient.
The most successful publicly funded research efforts are collaborations between funding agencies that ensure the research benefits the public and scientific advisers who understand what is important and feasible in the field.
And your contention that policy is driven by an desire for “decarbonization” is absurd on its face. The basic science that tells us we are warming the planet by burning fossil fuels is over a century old. Do you contend that Arrhenius was driven by a “decarbonization agenda”. Here’s a hint, Sonja: How about going out and looking at actual evidence rather than looking around and cherrypicking little bits that you can fit into your pre-existing, post-modern theory.
Excellent piece. I’m an experienced NIH reviewer, including chairing a review panel (Study Section), and I think that Dr. Schmidt captures the essential features of the review process very nicely. The NIH Office of Extramural Research has some nice educational features about peer review at their website (http://grants.nih.gov/grants/peer_review_process.htm) and I particularly recommend the Youtube video at (http://www.youtube.com/watch?v=kfgzdLe92c0) which shows a dramatization of how a review panel works.
I want to underscore a couple of points made in prior posts. First, reviewing is a lot of work. Reviewers take their responsibilities very seriously and devote considerable effort to review work, often to the detriment of their other responsibilities, including their primary research work. Second, reviewers are very aware that they are influencing the expenditure of the public’s money and are very concerned with making sure that the taxpayer gets good value for their money.
Finally, I want to second (or third or fourth) Dr. Lu’s comments.
For the unaware, Sonja Boehmer-Christiansen is editor of Energy and Environment.
Sonja Boehmer-Christiansen admits in an article published online that “the journal I edit has tried to keep this debate [climate scepticism] alive” She also states elsewhere I’m following my political agenda — a bit, anyway,” … “But isn’t that the right of the editor?”
The Yooper
Excellent writeup.
I’m going to email it to a few of the whackadoodles in Congress who believe that the scientific community is in collusion regarding global warming.
Bob
re #1, I recall Eli putting up some advice http://rabett.blogspot.com/2010/07/advice-to-lab-bunnies.html
Re #17–
Nearly $148M for “climate change education?” How was this number arrived at? Over what time scale is it relevant?
I searched the term at the NSF link Gavin provided, and found a list of grants going back to 1989. For 2011, I found a total of about $17M, if my mental addition can be trusted.
Moreover, the titles of proposals make pretty clear that they are not ONLY doing “climate change education.” For example, the biggest grant I noticed for 2011 is a large study of water quality in the Great Lakes–nearly $5M. From the description, the intent is to study multiple factors impacting water quality, but especially phosphorus loading, with a primary focus on the western Lake Erie basin. One of those factors is climate change.
They are listed under the “education” rubric because they plan to involve students in the research at various levels, from high school to postdocs. But as far as I can judge, the main focus is going to be studying phosphorus loads in western Lake Erie.
See what you can make of it:
http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=1039043
As a biomedical scientist with close to two decades of experience on NIH review panels (called Study Sections or Scientific Review Groups by the NIH) I’d also like to say “well done” to Gavin. The general public (well at least my friends and relatives) have little understanding of how research grants are awarded and the work that goes into (writing and) reviewing them; the NIH does “pay” reviewers but it’s such a token amount that my wife refers to me as a volunteer for the government when I’m on a review panel. But it’s a service that is important not only in helping select work most deserving of funding but as a grantee I feel a responsibility to add any expertise I have to support a system that, while not perfect, has made considerable contributions to human health and to be quite frank, my career.
I think all grant reviewers can point to situations where the process has fallen down either because of forceful reviewers supporting a less than stellar application or an excellent grant failing to receive unanimous support because of a dispute over the science between reviewers. But these are usually quite unusual situations, although I suspect that as funding levels tighten even further these types of problems will become more common.
In terms of Sonja Boehmer-Christiansen’s #13 question of who sets “policy-relevant research? The policy-maker, the individual research or the research institution?” The answer is all of the above. The NIH funds investigator initiated projects (I believe the largest area of funding) as well as Request for Applications (RFA) which are specific (usually one time) requests for applications in a very specific area/question. Then there are Research Programs which are usually multi-investigator projects that address larger questions and may continue to be funded as long as a renewal application is competitive against all other applications submitted for funding. Plus there are organizations that lobby for increased funding either in general (like FASEB) or for specific areas (patient groups/associations), and yes there are legislators who have and are strong supporters of science. And institutions themselves can influence funding. The institution I work for is an extremely large non-profit focused on biomedical research in specific areas – if it didn’t exist the NIH funding we have would be distributed almost certainly among overlapping but different areas of research.
Also Kevin’s #17 comments that “the NIH wastes billions of dollars following hoax and quack leads in seeking medical cures” is really uninformed. I might cite the specific example that use of the NOD (non-obese diabetic) mouse in diabetes research has been very unsuccessful in finding therapies that can be translated to the bedside as an example of “waste” of what probably has been billions of dollars over many years. But in reality all that research has been of tremendous help in understanding not only animal and human diabetes but autoimmune diseases in general as well as in training pre and post-graduate scientists who have gone on to make their own discoveries in diabetes and other fields. (FYI: My research is in autoimmunity but not diabetes.)
Mike
It does matter what (and who) gets funded. I recall as a grad student, if I’d had my way, I would have done a model of an ice sheet. But, our institute had a researcher who was very successful at getting funding in seismology, and thats where the research opportunities were. So thats what I did. There really are a chain of stakeholders, who are usually not involved in writing and evaluating proposals.
I am simply amazed that people can discuss the money – climate connection without even mentioning the coal and oil companies. Actually, I’m stupefied.
In the last three years alone Exxon had well over $1 trillion in revenue and over $100 billion in earnings. Their SGA expenses, which includes all advertising, PR, and lobbying, runs over $50 billion annually. Then there is the rest of the coal, oil, and gas industries. And their trade associations. And their think tanks.
Does anyone doubt that industry’s PR and lobbying operations are vastly superior to those of the climate science community?
Why do I never see these funding sources compared, even by scientists themselves? Maybe it is just hard to compare an ant to an aircraft carrier battle group.
Gavin, (responding to your comment on my post#18)
I did not mean that all GCM downscaling are bogus, but when you see a 15 page proposal devoting 8 pages to something that the authors really want to do and a half a page lip service discussing some experiments with future forcings, it is hard to take that seriously.
I am sure that you are aware of growing number of literature showing that the differences between different GCMs are way bigger that the differences between IPCC or Millenium Ecosystem Assessment scenarios. Growing number of people are saying that GCMs are not ready for primetime. I actually have deep respect for GCMs, because I regard mathematical description of the world around us as the highest degree of understanding.
I am absolutely on your side that science and climate science in particular is not an endeavor to make money. There are way better means of getting rich instead of hunting for a few $k here or there. I also agree that there is no wide conspiracy forcing scientist to adhere to the “consensus”, but scientist are not immune to sheep mentality to go by the flow.
In reaction to Daniel Bailey’s posting #21, I suppose, somebody should dig out that Sonja even believes in intelligent design and affiliated to the Tea party movement, which are obvious disqualification for taking her seriously. I personally don’t think pressing the political divide between alarmists and deniers is particularly productive.
Good article & discussion.
I think it is worth mentioning that the grant review process is, largely, an intrinsic component in the full peer-review process.
Many recent discussions of PR only concentrate on processes for publication and neglect that papers are mostly the output of funded research… let alone that it will, normally, by peer-viewed at conferences, seminars, PhD/MSc viva, and by the grant commitee of the reseachers next application.
#17 Kevin: The total appropriations for the National Center for Complementary and Alternative Medecine from 1999-2010 is $1,064.5 million. If we deduct 1 billion for shark cartilege research….
As for the idea that scientific research is funded only when it reinforces government policy. I have difficulty with this concept. Listening to the 122 congress I haven’t heard anyone who I would think might have a clue as to what holocene isotopes, 10Be, or carbon cycle modelling is about. Mind you we all know what modelling about. OK if you have a thing for skinny women. Am I to believe that in a dark corner of a Washington warehouse Republican political operatives are reviewing the finalists for 2011 funding? That scenario would have scientists living out of their briefcases as every 2 years a new directive, 180 out of phase with the previous, is issued to review boards everywhere. That didn’t even work in the Soviet Union but it could explain why 1930’s forecasts of daily living in 2010 hasn’t come true. I really could use a nuclear furnace in my basement.
I love it when deniers talk themselves into a corner.
We have “Kevin” at #17 arguing that too much money is spent on research that ultimately failed. Yet the common denier theme we hear is that “sceptics” don’t get enough money, that ideas that go against the standard view are somehow de-funded. Well, Kevin has given us a great counterexample.
Apologies for leading the discussion away from climate science but since its about grants, and whether they are politically determined …..
The point is money spent in one place is then unavailable for good research in another place.
The diabetes and shark cartilage examples are poles apart. Yes science should go up dead ends (diabetic mice) but not when the road is blocked off and signs say do not enter (shark cartilage).
Yes you can argue that all research is useful research but not in the same breath as saying you are a reviewer and that your job is important – because then you get back to the ludicrous suggestion above that all research should be funded and hence as a reviewer you will have no job to do.
[Response: Your points are much better when you don’t extrapolate to ridiculous rhetorical positions. No-one thinks that ‘all research is worth doing’, and reviewers and panellists are scathing about research they think is silly. And the less reviewing we have to do, the happier we’d all be. Part of the low funding rate is because people are writing too many low quality grants – which they do because the funding rate is low. – gavin]
Over the 15 years or so I am sure the US taxpayer has spent at least a billion dollars. Again you could argue that it is money well spent – the heroic NIH/NCI has saved us from a non-efficacious and potentially dangerous remedy. But I could have told them that if they’d made me a reviewer and paid me the paltry stipend mentioned… in fact I would have done it for free and perhaps suggested the money was better spent on climate research :).
The best reading on shark cartilage is
Ostrander GK et al (2004) Shark cartilage, cancer and the growing threat of pseudoscience. Cancer Res 64: 8485-8491
Finkelstein JB (2005) Sharks do get cancer: few surprises in cartilage research. J Natl Cancer Inst 97: 1562-1563
Two phase III publicly funded clinical trials (and they are expensive) are…
Lu C et al (2010) Chemoradiotherapy with or without AE-941 in stage III non-small cell lung cancer: a randomized phase III trial. J Natl Cancer Inst 102: 859-865
supported in part by a U10 cooperative agreement as part of the CCOP program (Public Health Service Grant CA48509…)
Loprinzi CL et al (2005) Evaluation of shark cartilage in patients with advanced cancer: a North Central Cancer Treatment Group trial. Cancer 104: 176-182
conducted as a collaborative trial of the North Central Cancer Treatment Group and Mayo Clinic and was supported in part by Public Health Service grants CA-25224, CA-37404, CA-35113, CA-63849, CA-63848, CA-35195, CA-35272, CA-35269, CA-35103, CA-35101, CA-60276, CA-52352, CA-37417, CA-35448.
Re climate change education – yes you can find good examples and I can find bad examples. For example surely most rational people would consider that climate change education has no place in the K-5 curriculum when we struggle to produce literate and numerate 15 year-olds, at least down here we do.
[Response: Huh? I don’t why you would think that. You don’t think that kids shouldn’t learn about what makes the weather? the seasons? that it was hot in the time of the dinosaurs? or cold in the time of the mammoths? – gavin]
Frankly, the idea of climate scientists (fresh from a beating on the grant trail because most of the money was allocated to spurious feel-good research) running around the great lakes with high school students telling them that global warming is the most clear and present danger to humanity scares me somewhat I have to admit.
[Response: More fantasy. – gavin]
The National Center for Complementary and Alternative Medicine is certainly an example of funding being driven, at least in part, by politics. This is really how political influence on the funding process is executed–by creating specific pots of money for specific areas. However, if you look at the list of National Centers and Institutes, they define broad-based areas of public interest.
NCCAM has been controversial, because they often fund clinical trials of “treatments” that are used by the public, but for which no plausible scientific rationale exists. It is thus hardly surprising that studies funded by NCCAM often yield negative results (e.g. the recent trial of the cold “remedy” echinacea). Whether this is a waste of money depends upon the value you place upon providing accurate scientific information to consumers who are subjected to a wide variety of claims about “complementary/alternative” (which is frequently a synonym for “quack”) cures. NCCAM has been criticized both by scientists in “non-complementary” fields (who would rather see that part of the NIH budget applied to more science-based questions, ideally the ones addressed by their own grant applications), as well as by some of the Congressional advocates of establishing this center, many of whom were alternative medicine zealots who expected the NCCAM to produce data supporting their beliefs (the fact that NCCAM has failed to do so tells you something about the limits of political influence on science: politics can influence to some extent the areas of research that get funded, but it is hard for Congress to influence the outcomes, because peer-review panels of scientists will not approve funding for projects that are do not adhere to valid scientific methodology).
Some replies
#27 Mark Shapiro err? There is a massive industry around carbon and climate. And of course because governments are involved the numbers are so huge they’ll dwarf your puny little Exxon. We’re already getting fleeced here. Good opportunities for protectionism as well – which is your protectionist party?
[Response: Oh please. This is just nonsense. Fossil fuels dominate almost every economy in the world, the idea a few carbon offset firms or a consultant for the CDM is on the same scale is fantasy land. – gavin]
Didactylos #31 Denier of what?
# 30 w kensit
You are able to study your holocene isotopes, 10Be because the politicians and bureaucrats believe it is important (and you get a good review of course:)). But if your interest is in collaborating with Egyptian scientists on the holocene fauna of Egypt you’d better have the words “climate change” and “effect” in your grant application.
[Response: No. No politicians had any interest in isotope variability in Holocene speleothems. None. Not one. *Scientists* thought those things were interesting – there is a big difference. – gavin]
“political operatives are reviewing the finalists for 2011 funding” Do you study a different planet? Of course there are analysts, lobbyists and bureaucrats counting where each bean went to make sure they got their share – that’s what politics is all about – who gets what.
I think you think there is a bigger difference between Reps and Dems than you think. You may have bigger swings that us [the bureaucracy doesn’t change with the government here :(] but I’m sure its basically business as usual there too.
Finally I don’t know about the 1930’s predictions for 2010 but a fascinating read is this article where prominent US citizens predict the world in 1952. One prediction that stood out and has certainly come true was that the government would become an ever increasing influence on our daily lives :)
CANTRIL H (1943) The world in 1952: some predictions. Journal of Abnormal and Social Psychology 38: 6-47
[Response: Let’s not get off track here. You want a country with no government influence on your life? – try Somalia. – gavin]
trrll, What do you call alternative medicine for which there is valid scientific evidence?
Answer: Medicine.
NCCAM ain’t science. It was an earmark by Congress. Some scientists draw funding from it. Some even do valid research. But it is not science, because effort is directed by decree rather than by curiosity.
Here is an article by someone in the next echelon down than most of the commentators here (IE younger and/or less part of the “establishment”) which tells another side of the funding story. Its worth a read, although much has been said before ad infinitum.
Lawrence PA (2009) Real lives and white lies in the funding of scientific research: the granting system turns young scientists into bureaucrats and then betrays them. PLoS Biol 7: e1000197, http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1000197
Biomedical science biased again (sorry). I dont agree with all of it – still a bit rose tinted about the lack of corruption and political interferennce.
Apologies for so many posts in a row – I’m 12 hrs ahead of you.
Q. Why so many bio-scientists in smaller countries?
Ray, I read trrll’s post as making the point that even if congress *tries* to influence scientific outcomes by funding by decree, they’re unlikely to succeed, and NCCAM serves as an object lesson that supports his claim.
Just with regard to Exxon’s SGA expenses in #27 – the $50bn will include Exxon’s entire payroll expense. It’s not a terribly relevant number.
Oh Gavin were those grants for “climate education” or “climate change education”? There is a world of difference between the two. Once students have had a good dose of chemistry, physics, biology, geology, and climate education, they are equipped to deal with climate change. If under the banner of “climate change eduction” your teachers teach “climate education” I take my hat of to them.
Re “You want a country with no government influence on your life? – try S****.”
– I see we can both use reductio ad absurdum arguments but I get told off for it.
Balasz: Growing number of people are saying that GCMs are not ready for primetime.
BPL: But not *competent* people.
@32 Kevin,
The first clinical trial you list started in 2000. The second ran 2001-2003.
The earliest of the two papers you cite refuting the efficacy of shark cartilage wasn’t published until 2004- postdating the beginning of both trials, and obviously their initial funding, by several years. The second trial had already been concluded by that point.
While it’s possible that the NIH should have known shark cartilage was useless prior to funding the trials, the papers you cite as evidence can’t possibly support that premise. In fact, if you RTFP you see that there were multiple in vitro studies and at least two small-scale trials that showed promising results prior to the 2001-2003 trial that showed negative results.
Furthermore, you’ve still not provided a source for the claim that the NIH wasted $1 billion on the topic.
Kevin –
NIH funds a great deal of clinical research. NIH’s mission is to improve the health of Americans. A criterion applied commonly in evaluation of clinical research proposals is “will it change clinical practice for the better?” Negative trials of some interventions, either medications, surgeries, or alternative therapies, are important in improving patient care by reducing patient risks and waste of money on deadends. In some cases, trials are undertaken with the full recognition by investigators and reviewers that the actual scientific justification for the therapy is slight. These are usually treatments that have become established in practice or embraced by patients without good evaluations. Shark cartilage is an example but not all of the interventions in this category are alternative therapies. Knee arthroscopy for osteoarthritis pain and the Extracranial-Intracranial (ECIC) bypass surgery are examples of 2 common surgeries whose validities were undermined by well conducted negative trials. ECIC bypass has almost disappeared (there are still a couple of valid but unusual indications) and knee arthroscopy for osteoarthritis pain is on the way out.
There was a vogue a number of years ago for gingko preparation treatments for dementia. The scientific rationale was zero. Proper trials were done, and to the surprise of no one, including the investigators and the reviewers who were responsible for funding the trials, the outcomes were negative. This was useful for practicing clinicians (like me) in advising patients not waste their money on these preparations. I see significantly fewer patients taking gingko than I did a decade ago. These negative trial results have saved a lot of people a lot of money and the investment in trials has probably repaid itself several times over. This is also a patient safety issue. Supplements like gingko and shark cartilage are not regulated by the FDA and there is minimal oversight of manufacturing. There have been some fortunately rare episodes of supplement contaminants causing serious problems.
I, and virtually all the physician-scientists I know, would prefer that interventions not be used without good evidence of efficacy and safety. This is not, however, the reality of clinical practice. Trials for things like shark cartilage may not justified on scientific grounds but they are often well justified in terms of potential impact on clinical practice.
Kevin,
I don’t think anyone contends that government funding of science is perfect or that there are no aberations. However, in most cases, there is an advisory body of scientists that assists funding agencies in determining the interesting questions as well as the relevance of those questions to policy and public interest.
It is not surprising that the reality of climate change and its effects on ecosystems, agriculture, infrastructure… are among those interests. They are not the only such criterion in any RFP that I know of.
I do not know whether your argument is based on a bad personal experience in some other field or whether it is based on your own personal conviction. I am sure, however, that it isn’t based an a broad assessment of the facts wrt climate research.
The point on reputation (second last in Gavin’s list) is both understandable but at the same time has a problematic side to it, which Gavin mentions: For newcomers, the bar is in effect set (a lot) higher than for well known researchers in the field. The same goes for one’s chances of getting a publication in Nature or Science: Without a well known name on the authorlist, chances are smaller to make it (though of course it’s still possible). It’s understandable, because a name is a convenient shorthand for the quality of a piece, but it’s problematic because it keeps a hierarchy intact which isn’t necessarily conducive to good science.
39, Kevin,
The argument of how much government is the right amount of government is a good one, but belongs in another forum.
The argument that the politics of climate change are evidence of an intrusive government is absurd. On the other hand, if the world ever had a situation which required some degree of coordination and unity — not one world government, or invasive government, but merely enough cooperation and direction to accelerate the evolution of the world’s archaic and ill-advised energy infrastructure — then this is it.
Interestingly, the way we got into this military industrial complex world that seems to both so frighten and enamor people of a particular political persuasion was through the advent of World War II, the last great challenge which faced the peoples of the world, and required collective unity, coordination and personal sacrifice.
Sixty years ago, I wonder if you would have been one of those Americans arguing fiercely that the U.S.A. needed to stay out of what was a European problem?
Kevin, I was taught about the greenhouse effect in school. I must have been about 12, and that was long ago now.
Education is key.
Also, Kevin, you seem a little hung up on having a “rationale” for research. That we have to know how or why something works (or doesn’t work) before we study it. But that makes no sense! We do the research so that we can answer these questions. And, in the case of drugs, sometimes we end up with a product that is effective, but we still don’t know how it works.
It’s all very well picking out one case that particularly annoyed you, but to deny the general principle is to stop research dead.
#34 Kevin re. Gavins response:
[Response: Let’s not get off track here. You want a country with no government influence on your life? – try So-malia. – gavin]
Nailed it.
The issue of “newcomers” having a tough time getting grants from NSF and NIH (one’s that I am familiar with) is not as bad as you think (Bart 44). I think that some reviewers (including me) and some program directors try to give promising newcomers a chance. Let’s say that someone published 2 or 3 promising papers as a part of their PhD or postdoc research and this leads to an interesting line of research. Such a person, recently hired as an assistant professor has a good shot at fundiing, most likely on the second round. Such a person will usually be asking for less money that someone with a large lab that includes postdocs.
When scientists insist on their love of what they do, they may think they will thereby defuse public anger about funding. Thing is, though, what a lot of people resent about scientists is precisely that they get to spend their lives doing meaningful work. The widespread dislike of teachers has a similar rationale: how dare you ask for pay commensurate with your qualifications if, as you say yourselves, you’re doing what you want to do. We don’t get to do that!
This discussion of the mechanisms of funding is valuable in itself and touches on many serious issues, but public resentment about science has emotional roots that have nothing to do with the fairness or unfairness of grant proposals.
Gavin it is too bad you and Your boss suffer so poorly in the FY2011 budget for Climate change study NASA 438.1 million, NOAA 437 million.NSF 480 million. I belive this proves alarmist work not science.
[Response: Actually it proves that satellites are really expensive. But if you’d rather not know what is happening on the planet (weather, deforestation, pollution, etc.), I can imagine that you’d think that wasn’t worth paying. Congress (and many others) disagree. – gavin]