Gavin Schmidt and Eric Steig
The last couple of weeks saw a number of interesting articles about archiving code – particularly for climate science applications. The Zeeya Merali news piece in Nature set the stage, and the commentary from Nick Barnes (of ClearClimateCode fame), proposed an ‘everything and the kitchen sink’ approach. Responses from Anthony Vejes and Stoat also made useful points concerning the need for better documentation and proper archiving. However, while everyone is in favor of openness, transparency, motherhood and apple pie, there are some serious issues that need consideration before the open code revolution is going to really get going.
It would help to start by being clear about what is meant by ‘code’. Punditry about the need for release of ‘all supporting data, codes and programmes’ is not very helpful because it wraps very simple things, like a few lines of Matlab script used to do simple linear regressions, along with very complex things, like climate model code, which is far more sophisticated. The issues involved in each are quite different, for reasons both scientific and professional, as well as organizational.
First, the practical scientific issues. Consider, for example, the production of key observational climate data sets. While replicability is a vital component of the enterprise, this is not the same thing as simply repetition. It is independent replication that counts far more towards acceptance of a result than merely demonstrating that given the same assumptions, the same input, and the same code, somebody can get the same result. It is far better to have two independent ice core isotope records from Summit in Greenland than it is to see the code used in the mass spectrometer in one of them. Similarly, it is better to have two (or three or four) independent analyses of the surface temperature station data showing essentially the same global trends than it is to see the code for one of them. Better that an ocean sediment core corroborates a cave record than looking at the code that produced the age model. Our point is not that the code is not useful, but that this level of replication is not particularly relevant to the observational sciences. In general, it is the observations themselves – not the particular manner in which they are processed – that is the source of the greatest uncertainty. Given that fundamental outlook, arguments for completely open code are not going to be seen as priorities in this area.
By contrast, when it comes to developers of climate models, the code is the number one issue, and debugging, testing and applying it to interesting problems is what they spend all their time on. Yet even there, it is very rare that the code itself (many of which have been freely available for some time) is an issue for replication — it is much more important whether multiple independent models show the same result (and even then, you still don’t know for sure that it necessarily applies to the real world).
The second set of issues are professional. Different scientists, and different sciences, have very different paths to career success. Mathematicians progress through providing step by step, line by line documentation of every proof. But data-gathering paleo-climatologists thrive based on their skill in finding interesting locations for records and applying careful, highly technical analyses to the samples. In neither case is ‘code’ a particularly important piece of their science.
However, there are many scientists who work on analysis or synthesis that make heavy use of increasingly complex code, applied to increasingly complex data, and this is (rightly) where most of the ‘action’ has been in the open code debate so far. But this is where the conflicts between scientific productivity at the individual level and at the community level are most stark. Much of the raw input data for climate analysis is freely available (reanalysis output, GCM output, paleo-records, weather stations, ocean records, satellite retrievals etc), and so the skill of the analyst is related to how they choose to analyse that data and the conclusions they are able to draw. Very often, novel methodologies applied to one set of data to gain insight can be applied to others as well. And so an individual scientist with such a methodology might understandably feel that providing all the details to make duplication of their type of analysis ‘too simple’ (that is, providing the code rather carefully describing the mathematical algorithm) will undercut their own ability to get future funding to do similar work. There are certainly no shortage of people happy to use someone else’s ideas to analyse data or model output (and in truth, there is no shortage of analyses that need to be done). But to assume there is no perception of conflict between open code and what may be thought necessary for career success – and the advancement of science that benefits from a bit a competition for ideas — would be naïve.
The process of making code available is clearly made easier if it is established at the start of a project that any code developed will be open source, but taking an existing non-trivial code base and turning into open source is not simple, even if all participants are willing. In a recent climate model source code discussion for instance, lawyers for the various institutions involved were very concerned that code that had been historically incorporated into the project might have come from outside parties who would assert copyright infringement related to their bits of code if it were now to be freely redistributed (which is what the developers wanted). Given that a climate model project might have been in existence 30 years or more, and involved hundreds of scientists and programmers, from government, universities and the private sector, even sorting out who would need to be asked was unclear. And that didn’t even get into what happens if some code that was innocently used for a standard mathematical function (say a matrix inversion) came from a commercial copyrighted source (see here for why that’s a problem).
Yet the need for more code archiving is clear. Analyses of the AR4 climate models done by hundreds of scientists not affiliated with the climate model groups are almost impossible to replicate on a routine and scalable basis by the groups developing the next generation of models, and so improvements in those metrics will not be priorities. When it comes to AR5 (for which model simulations are currently underway), archiving of code will certainly make replication of the analyses across all the models, and all the model configurations much less hit or miss. Yet recently, it was only recommended, not mandated, that the code be archived, and no mechanisms (AFAIK) have been set up yet to make even that easy. In these cases, it makes far more sense to argue for better code archiving on the basis of operational need, than it does on the basis of science replication.
This brings us to the third, and most important issue, which is organizational. The currently emerging system of archiving by ‘paper’ does not serve the operational needs of ongoing research very well at all (and see here for related problems in other fields). Most papers for which code is archived demonstrate the application of a particular method (or methods) to a particular data set. This can be broken down into generic code that applies the method (the function), and paper-specific code that applies that method to the data set at hand (the application). Many papers use a similar method but in varied applications, and with the current system of archiving by ‘paper’, the code that gets archived conflates the two aspects, making it harder than necessary to disentangle the functionality when it is needed in a new application. This leads to the archiving of multiple versions of essentially the same functional code causing unnecessary confusion and poor version control.
It would be much better if there existed a stable master archive of code, organised ‘by function’ (not ‘by paper’), that was referenced by specific applications in individual papers. Any new method would first be uploaded to the master archive, and then only the meta script for the application referencing the specific code version used would need to be archived with an individual paper. It would then be much easier to build on a previous set of studies, it would be clear where further development (either by the original authors or others) could be archived, and it would be easy to test whether the results of older papers were robust to methodological improvements. Forward citation (keeping track of links to papers that used any particular function) could be used to gauge impact and apportion necessary credit.
One could envision this system being used profitably for climate model/reanalysis output analysis, paleo-reconstructions, model-data comparisons, surface station analyses, and even for age-model construction for paleo-climate records, but of course this is not specific to climate science. Perhaps Nick Barnes’ Open Climate Code project has this in mind, in which case, good luck to them. Either way, the time is clearly ripe for a meta-project for code archiving by function.
Ray Ladbury wrote @142 wrote
so let’s say you are living on Mt. Merapi, and the government seismologists tell you it’s going to erupt. Do you leave? After all, you can’t do repeatable experiments.
I would leave even if I thought Mt. Merapi wold not erupt for the same reason I bought life insurance when my son was born. It’s the precautionary principle.
I try always to make a distinction between the science and my decisions in the real world. Within science I have a high bar for being certain of a theory or fact. It has to be reproducible and controlled… there can be no other factor which might possibly affect the outcome. Newton’s law of gravitation meets (taking in account relativity) meets my bar for high certainty; the models from climate science do not. By the way, it is logically consistent not to have a high certainty for climate models but still advocate action regarding climate change. We all have to make decisions with some uncertainty. Decisions based on Newton’s law (e.g. sending man to moon, assuming all engines fire correctly, etc.) have, for me, no uncertainties. Decisions based solely on climate models have uncertainty. The amount of uncertainty depends on your faith in the models.
John Reisman @146 wrote
The main issue that is tossed about in the denial of human caused global warming is that the models are wrong and we are not warming, or that it is not human caused.
Now, let’s set a side the models and start from scratch for a second.
Yes I agree there is more evidence for climate change than just what the models predict. But I was just concerned with computer models.
Let me review this how this subthread evolved.I suggested that every climate scientists publish the their programs because science has to be reproducible. If the results of a paper depends on computer code, the only way to see if the code is correct is to examine the code. Describing the methods is not enough because if another scientist got different results, who’s code do you trust? In mathematics one must publish the reasoning behind a new theorem.
The thread morphed into a discussion regarding comments in code. I don’t think comments in the code were very important.
So far, so good. I know something about computers and I think I was on firm ground. Then I made a statement about the relative certainty of the results of CFD and climate science models. That statement is more about the epistemology or philosophy of science than about computer science. Now I feel I’m being invited into discussing climate science. I’m not a climate scientists and I’m don’t want to go there. I think I can add something, or at least present a view for you consideration, on computer science that may be informative to some on this board.
I don’t know much about climate science and there’s nothing much I can add on that subject. My view, what whatever it’s worth, is that climate is warming, that humans are adding to the warming and that we should reduce CO2 emissions due to the precautionary principle.
klee12
Robin said in reply to Steve: “Accusing me of being unprofessional”
Robin, he didn’t accuse you of anything. He said leaving outdated comments is unprofessional, which it is.
Further, this discussion wasn’t just about bad comments – it was about good comments. Which from your description, you have never encountered. Which is sad, because it implies that you and your team write worthless comments (by your own evaluation).
We agree that readable code is important. But if you can’t enhance the readability, and, in particular, decrease the time needed to understand code with judicious use of comments, then you lack an important skill.
Why do you insist all comments are “useless and dangerous”? It’s just unreasonable. Yes, we know there are bad comments out there, just as there is bad code. That doesn’t change the value of good comments any more than it changes the value of good code. It’s not a choice – we can demand both.
klee12 said “I was talking about controlled tests that were reproducible”
The problem isn’t that climate science doesn’t have reproducible, definite tests. The problem is that climate isn’t reproducible and definite. In particular, the weather isn’t consistent at all. It is chaotic.
This means that in climate science, evaluating the results of climate models needs a different benchmark than evaluating a wind tunnel test. Other branches of science have other standards, developed to match whatever peculiarities the particular field has. Particle physicists see gazillions of collisions, so they require a ludicrously high confidence to compensate. Biologists often have tiny samples, so they do everything they can to extract statistically significant results.
And climate scientists deal with chaos, so they expect a wide range in their model output. In fact, if climate models produced predictable weather patterns all the time, that would indicate the model is broken.
Note also that climate isn’t weather. Weather models will diverge if you run them beyond reasonable time-periods. Climate models only show useful results when run over long time periods.
People with experience of other kinds of model often make assumptions about climate models. I recommend you check your assumptions against the FAQ on climate models (see part II also).
Klee12 says: “Within science I have a high bar for being certain of a theory or fact. It has to be reproducible and controlled… there can be no other factor which might possibly affect the outcome. Newton’s law of gravitation meets (taking in account relativity) meets my bar for high certainty; the models from climate science do not.”
OK, so let me get this straight: You are saying that biology, geology, astronomy, astrophysics, geophysics, meteorology, and for that matter about just about any science you encounter outside of freshman lab is not science? Dude, do you really want to take a position that is that silly?
FWIW, I’ve actually walked through the crater of Mt. Merapi–albeit in one of its quieter phases. It’s a pretty amazing place. You’re missing out.
klee12 #151: Newton’s Laws do not guarantee reproducible results without uncertainties. Why do you think a space probe has to carry extra fuel to correct its course? More on my blog.
You may find it instructive to study the Muir Russell report on the CRU email theft if you think that publishing methods and not code renders results irreproducible.
I know its a bit off topic, but I thought I’d enter a defense of Fortran. The language name is short for Formula Translation, which is a big part of what scientists do when programming. Beyond that the accidental way the lanquage evolved leads to the possibility of very efficient usage of modern computers. Our particular application is mostly written in Fortran. Because the new vector like constructs available in modern computers are pretty restricted, performance is greatly enhancd if the compiler can make certain assumptions about low level address alignment, i.e.
“does every array in this loop has the same alignment with respect to cahche line boundaries”. With Fortran common blocks asuring that is a snap. It is hard for me to imagine the compiler would be abe to examine C code and figure that out. So you can leave a lot of performance, which I think for large climate simulations limits how much you can accomplish, you are going to throw quite a bit of that away by using other more modern lanquages. I also find C much more prone to subtle errors. Twice this week I’ve wasted a couple of hours tracking down C code performing weirdly and wrongly, and in all cases, its been a misplaced equals sign, like missing == in an if test. The human brain (at least mine) just glosses over these sorts of typos, but then eventually you gotta figure out why your code is acting insane. All languages have their traps. Fortran got a bad name because of all the code written in the early years, before anyone had heard of software engineering. But, the modern evolutions of it are actually quite good.
I agree emphatically with Thomas. Modern Fortran (95 and later ANSI standards) is a nicely structured, extremely useful, very readable programming language. And nothing out there but assembly language beats it for crazy speed if you’re running a simulation. They’ve had 53 years to optimize its performance. I can’t think of any other language that’s been going that strong, that long.
For those who want to try it, the “Personal Edition” of Salford Fortran-95 is free to download. That’s what I used until I turned into a Visual Basic 2008 freak. I retain it in case I have to translate a program into Fortran for professionals.
#151 klee12
I agree certainly with the precautionary principle you ascribe. However, it seems that when you combined the computer models, the error bars, combined with the observations and the maths and physics combined provide a rather stronger metric than merely the precautionary principle, for reasoned action.
Besides, the precautionary principle is not enough for those that approach this from the perspective of belief and political bias.
Since we are dealing with tremendous inertia’s of energy and related feedbacks, we need to apply the brakes now and as firmly as possible without degradative disruption of the socio-economic system.
From this one can reasonably conclude that meaningful policy action is needed now, rather than alter and that the confidence levels of the correlation of multiple disciplines of science concur with the general results, scientific consensus is strong and that our economy depends on said meaningful action.
The combined lines of evidence, models and error bars included have resulted in what is now claimed as an unequivocal result in that humans are in fact influencing the climate. So the precautionary principle is not really even needed although not unimportant.
That being said, I would like your opinion regarding Fee and Dividend policy as an effective means by which to exercise the precautionary principle, and adding your perspective to the mix of even possibly signing it?
http://www.climatelobby.com/
http://www.climatelobby.com/fee-and-dividend/q-a/
BTW, for those that may have actually tried to click on the top two links below, I did fix them :)
Economics: Balancing Economies
October Leading Edge: The Cuccinelli ‘Witch Hunt”
—
Fee & Dividend: Our best chance – Learn the Issue – Sign the Petition
A Climate Minute: Natural Cycle – Greenhouse Effect – Climate Science History – Arctic Ice Melt
Philip Machanick (#155)
I don’t think his comment was related to the small amount of error in those calculculations (newton’s law). It was related to how much you can believe the results represent reality. Newton’s Law has be validated much more that climate models. This is a fact and there is no way you can argue that. Besides, Newton’s Law is included in the climate models so any uncertainty there shows up in the climate models as well.
IMHO, the main difference between the climate models and Newton’s Law is that there are inherent uncertainties in the climate system (ie. even with perfect information they won’t go away) while most uncertainties in Newton’s Law results from imperfect information. I do think most of these uncertainties are well represented in the models, but there may be some that aren’t understood fully.
In summary, I have greater confidence in Newton’s Law than Climate Science (shouldn’t this be self evident?) but that doesn’t mean the GCM models are useless. The results of the models should be qualified though, when they are communicated to the political actors (in a form they will understand). Generally, I think scientists have been ernest in trying to communicate this information but haven’t been able to communicate this information across the political spectrum (clearly!).
Re #152
ROFL… Like in the last 30 years, I’ve never looked at code written by someone else…
I don’t bother writing comments since my code is perfectly readable. The professional software engineers I know, agree with me on comments so I know I am not alone. And just to be clear my teams have produced high-volume, reliable SOA applications that generate millions of dollars in revenue so we’re \serious\. My experience with comments comes from having to review and fix reams of code written by others. Even \good/valid\ comments can become obsolete (ie useless and dangerous) because of changes in library or application code being called by the commented code. The called code is usually completely unaware of the caller’s code – so how can it possibly be kept up-to-date over months and years of development? Fundamentally, comments are never actually \tested\ or debugged in any real sense. So, their content is inherently suspect and even \quality\ comments degrade over time. Believing a comment instead of following the code back properly through the layers is often a fool’s errand. The bottom line is that knowledge of the underlying purpose of the code exists external to the code (and the comments). Without that knowledge – the code and comments will not be comprehensible.
Re #156
Fortran is perfectly fine and considered very efficient for doing calculations. Where Fortran falls down is \housekeeping\ – memory management, threads, sockets, system resources, shared resources, object allocation/deallocation important to building large complex applications.
BTW C++ is perfectly capable of doing cache alignments [and other fancy performance enhancements – including allowing inline assembler] and most C++ compilers produce highly efficient code – when the code is written with that goal in mind.
Sambo@159 “IMHO, the main difference between the climate models and Newton’s Law is that there are inherent uncertainties in the climate system (ie. even with perfect information they won’t go away) while most uncertainties in Newton’s Law results from imperfect information.”
Ever hear of the three-body problem?
Thomas:
I almost fell over laughing* when you started talking about “modern” languages like C. The wheel has turned. Today’s modern languages usually make issues such as address alignment moot, since they will handle structure padding and things of that nature quietly, by default. C is indeed prone to some very subtle errors, largely thanks to pointers. And pointers are evil – useful but evil. The “if” test example you quote wouldn’t be an issue in a language such as C#, which doesn’t allow that sort of implicit casting. Language designers have learned a lot (mostly by stealing from other languages).
You did raise an interesting question in my mind: is there room for a new scientific computing language? One with all the pros of Fortran, but that also benefits from the decades of language evolution since then? I don’t know enough about Matlab to know if that comes close to fitting the bill, and for all R’s benefits, it may be newish but it is neither modern nor suitable for general use.
I keep hearing how Fortran is great for scientific computing, but it really seems to lack any specia-lised features. I think it gets used purely due to legacy code.
* I didn’t.
PS: Dear moderators: the value of a spam filter that can’t distinguish between a drug name embedded in another word and real spam? That can be fooled merely by adding a hyphen in the offending word? Zero. You have turned on reCAPTCHA. This is a 100% moderated blog. Perhaps you should consider turning the useless spam filter off?
Robin:
“in the last 30 years, I’ve never looked at code written by someone else”
“My experience with comments comes from having to review and fix reams of code written by others”
I’m sorry, I can’t reconcile these comments. Are you saying you retired a long time ago? Software engineering has moved on.
“changes in library or application code being called by the commented code.”
This is really important. Comments can capture assumptions made about library code that aren’t explicitly verified by the calling code. But if the library code changes in ways that break calling code, then you have a more serious error of process than a few outdated comments!
“Fundamentally, comments are never actually \tested\ or debugged in any real sense.”
Why should they be? They should be checked at the same time as the code, but their function is to help. They aren’t normative. Nobody should assume the comments are perfect, any more than they should assume the code is perfect.
“Believing a comment”
See above.
If as I suspect, you left programming 30 years ago, then perhaps you just aren’t aware of the sort of standards expected from today’s programming graduates. We look at code from way back when, and wonder how you ever got anything done.
Ray Ladbury (#161)
In the three-body problem, most of the uncertainty comes from small perturbations to the ideal solution. Often, this comes from assumptions that are inexact (point mass, rounding error is small). If this information were perfect, I believe it would be exact solutions (and I’m quite confident of that!). Moreover, the discussion was about decision’s under these uncertainties (practical space probes). The lack of analytical solution to the three body problem says very little about the practical aspects I was commenting on. Yes, it needs to be considered, however in most cases it has relatively minor affects (if not, you are not planning your mission very well).
Re #163
ROFL… Are you kidding me? Did you simply misread what I said or deliberately misquoting?
“Like in the last 30 years, I’ve never looked at code written by someone else…”
The usual interpretation of that is “Yes. I have looked at other people’s code.”
As for…
“If as I suspect, you left programming 30 years ago, then perhaps you just aren’t aware of the sort of standards expected from today’s programming graduates. We look at code from way back when, and wonder how you ever got anything done.”
Again you failed to read or comprehend my previous posts. How about I suspect you don’t even write production code? Not that I think that matters particularly.
But I DO design and write production code, for Unix and Windows, yesterday and today. Over a million lines of robust C++ and counting (40,000 in just the last two months alone – lots of new stuff recently…). You know, fully object oriented, using exceptions, templates, multi-threading, blah, blah, blah…
And seriously if you wanna comment your code – be my guest. I don’t care. But I don’t think Climate Scientists should be “expected” to comment their code since the industry sure as the world doesn’t think its useful. The general consensus of practicing engineers is “useless and even dangerous to rely on comments for anything serious”.
Didactylos wrote @ 153
The problem isn’t that climate science doesn’t have reproducible, definite tests. The problem is that climate isn’t reproducible and definite.
Yes I agree, but I draw the conclusion that models can not give a high level of certainty. This is reflected, in part, by the acknowledged long term predictions has a high level uncertainty. If I send a space probe around the solar system I have confidence I can predict its path for a hundred years, baring exogenous effects like solar wind, unforseen meteoroids, and so on.
Ray Ladbury @ 154 wrote
You are saying that biology, geology, astronomy, astrophysics, geophysics, meteorology, and for that matter about just about any science you encounter outside of freshman lab is not science
Not really. Long, long ago, biology was just the classification of various organism into genra, species, or whatever. No predictions, nothing to reproduce. Then Mendel made some observations and experiments about heredity of certain traits and the experiments were reproducible. Today we can isolate genes, and create new strains of rice or corn, which is reproducible.
I suppose I could go on. Geology has observations (I think) but I don’t think you do large scale experiments so there is nothing to reproduce. You work with the theory that seems to fit the work the best and I’ll call that science. And I’ll act assuming the latest theory is the correct one. But it won’t have the level of certainty (for me) as Newton’s laws.
Philip Machanick @155 wrote
Why do you think a space probe has to carry extra fuel to correct its course?
There may be many reasons (1) On lift off there may be a slight error in the trajectory or speed (2) solar wind (which really exerts a very small force) may push it slightly off course. Maybe very slight force over years might make you miss a target millions of miles away by thoudsands of miles. (3) friction or the gravitational effects when flying through or near asteroid belt between Mars and Jupiter.
You may find it instructive to study the Muir Russell report on the CRU email theft if you think that publishing methods and not code renders results irreproducible.I stated earlier if someone tried to reproduce the results and got a different result, it is very difficult, if not impossible, to see where the discrepancy arose. The code may differ from the method described (error in specification) or in the code (bug. Re the Muir report, I briefly took a look at the table of Findings and couldn’t find where they discussed models. A page number would help.
John Reisman wrote
Besides, the precautionary principle is not enough for those that approach this from the perspective of belief and political bias.
I separate the the science aspect from the public policy aspect. The scientific part is concerned with nature and only nature and should be as objective as possible. The public policy part involves making decisions in the face of uncertainty and different people of good will may have different opinions. Scientists should take part in the public policy debate but they should be careful not to oversell their position. I cringe when I hear scientists say “the science is settled”. They do not specify what is settled and when unconvinced scientists examine the evidence some may, in good faith, not be entirely convinced that the “science is settled” and resent what they perceive is an attempt to mislead. I think that Al Gore is not held in high regard, and I think that in the long run his efforts have hurt rather than help his cause. Scientists should not oversell the science because it may lose credibility and hinder their cause.
My position there is that the science is not settled, and from a public policy point of view I invoke the precautionary principle to advocate action.
On the Fee and Dividend policy, that seems much better than Cap and Trade, but it is still too unwieldy. I don’t see it happening. This open is not based on science, it is not the efficacy, it is based on my reading of the political situation which may be wrong.
Did 162: I keep hearing how Fortran is great for scientific computing, but it really seems to lack any specia-lised features. I think it gets used purely due to legacy code.
BPL: What part of “faster than anything else but assembly language for numerical simulation” did you miss?
sambo 164: In the three-body problem, most of the uncertainty comes from small perturbations to the ideal solution. Often, this comes from assumptions that are inexact (point mass, rounding error is small). If this information were perfect, I believe it would be exact solutions (and I’m quite confident of that!).
BPL: Then I take it you’ve never actually taken a course in celestial mechanics, or read Laplace.
BPL said: “What part of “faster than anything else but assembly language for numerical simulation” did you miss?”
That’s your opinion, and I pretty much just ignored it. Mostly because you didn’t bother providing anything to back the claim, but ultimately because you are wrong. Any benchmark is limited in utility, but there is absolutely no way your bizarre claim is justified based on all the benchmarks I’ve seen.
I did not base my opinion on the fact that you think Visual Basic is a good programming language. Instead, I checked those benchmarks again.
Robin: you wrote 40,000 lines of code in 2 months? Yay for you. I hope I never see it, and I *really* hope you don’t work anywhere important.
I’m optimistic that you just don’t know how to measure SLOC, and that you don’t really produce high volumes of dreck and imagine that all the effort equates to productivity!
Either way, your comments about comments reflect poorly on the software industry.
I do apologize for misunderstanding your attempt at humour.
“The general consensus of practicing engineers” – no. No. That’s your wildly misinformed opinion, not shared by everyone.
BPL (#168)
You’re being presumptious. I have taken a class in celestial mechanics. I stand by what I said (it is an opinion statement in any case). Argue against that if you wish, but I’d rather you not ressort to false assumptions about me.
BTW, to what are you refering in terms of Laplace. As far as I know he didn’t write much that would argue against what I stated.
klee12: you have some very odd ideas about science. It is waaaaay beyond my ability to correct your ideas. Perhaps try reading a few popular science books, or something?
The Muir Russell report discusses reproducing results using completely different methods and data. That is far, far more valuable from the point of view of the result. It may not help advance any single method, but it shows that minor differences in method are irrelevant.
And you really need to do some research on the n-body problem. Don’t get distracted by details such as launch imperfections. The key point is that useful positional data is the result of a complex numerical simulation, and not a neat equation.
Well, at least he can’t be accused of counting comment lines as lines of code! /snark
Sorry Robin, one more thing that really needs to be said regarding the commenting issue: the people that write system libraries. Bet you’re glad *they* wrote comments, huh? Or do you always procure the source for every library method and read *every line of source code in every library routine* that you use to make sure it has no bugs in it before you use it? Yeah, didn’t think so. That would be really productive. Not.
You would also need to do that just to figure out how to even *call* every freakin’ library routine if no one bothered to document the parameters. And that applies to *everything* your esteemed co-workers write that you need to call.
Now do you realise how ridiculous the position you are taking looks? Enough said.
#166 klee12
I would agree that the science is not settled and in fact never will be settled.
However, there is enough certainty to in climate science to take action without the precautionary principle. IN other words the science is understood enough in key areas to see that certain conclusions can be drawn with sufficient confidence on the robust results of the science to take action without inordinate amounts of guessing or exercising the precautionary principle.
In other words, the science is settled enough in understanding the major forcing components, error bars included, multiple disciplines included, same general conclusions included that we really don’t need to guess about the fact that virtual certainty exists and humans are impacting the climate.
My point is that all this science, while uncertainties do exist in certain areas and models are not perfect allows reasonable basis to invoke meaningful action on sufficient magnitude to counter the future impacts of the increased emissions at various levels of impact as estimated by those imperfect models.
I think we can agree that the political situation is far from reasonable comprehension at this time. but that does not diminish the potential of Fee and Dividend, it merely illustrates a different problem.
In addition to #172 Didactylos
Don’t let the perfect be the enemy of the good. A perfect example was Steve McIntyres statistical analysis of the Hockey Stick. While it may have added in method, it weakened the model. And, the result was that the Hockey Stick is still a Hockey Stick and the McIntyre fix proved statistically insignificant.
Economics: Balancing Economies
October Leading Edge: The Cuccinelli ‘Witch Hunt”
—
Fee & Dividend: Our best chance – Learn the Issue – Sign the Petition
A Climate Minute: Natural Cycle – Greenhouse Effect – Climate Science History – Arctic Ice Melt
Re: 170 LOL Whatever… And that is 40,000+ high quality lines of code in two months. With design documentation, QA and in production used by customers.
We do numerous releases a year with rigorous SDLC processes – considered a model in the company (we wrote the processes). My team and our code generate millions of dollars in revenue every year. No unplanned downtime, 24/7/365. And no stupid comments. We’ve been doing the same for years. What have you done?
klee12 #166: I didn’t find it hard to find the discussion in the Muir Russell report on reproducing results independently but it starts on p 150.
sambo et al.: a space probe has to navigate under much more complex conditions than the 3-body problem. We have over a million asteroids for a start, of which a small fraction have been mapped. You can theoretically model the entire solar system with differential equations and solve them exactly but in practice this is impossible. If your space probe goes nowhere near the asteroid belt this specific correction may be minor, but there are many unmapped bodies in space. The climate system is subject to many more variables but the same principle applies. In theory you could apply classical physics to the entire system rendering it totally deterministic but in practice, it’s impossible. I direct this argument specifically at people who claim climate science isn’t “proper” science because it can’t yield exact answers. The same is true of any science of the real world that works with large-scale phenomena like oil geology.
BPL: What part of “faster than anything else but assembly language for numerical simulation” did you miss?
Did: That’s your opinion, and I pretty much just ignored it. Mostly because you didn’t bother providing anything to back the claim, but ultimately because you are wrong. Any benchmark is limited in utility, but there is absolutely no way your bizarre claim is justified based on all the benchmarks I’ve seen.
BPL: When I run a certain radiative-convective model written in Fortran-95, it runs in 33 seconds. In VB 2008, it runs in 72 seconds. In Borland C/C++ it runs in 58 seconds. Fortran beats everything.
The “benchmark” programs assume every program is doing everything you can do with a computer. Real programs don’t. That’s why I specified “numerical simulation.” Numerical simulation doesn’t have to get on the web, move pictures around in a GUI, send signals, coordinate coroutines, or dereference pointers. I’m talking about crunching numbers, about fast, highly efficient routines for arithmetic and transcendental functions. For that, Fortran beats anything but assembler.
sambo 171,
The 3-body problem is only perfectly soluble for a perfect case. Maybe you did take celestial mechanics, but if so you must have skipped numerical analysis. Any real-world solution to the n-body problem is an approximation, always and everywhere. See Laplace 1826.
klee12@166
OK, so I’m not seeing a difference between the situation in geophysics, for example, and climate science. And you do know that Newton’s laws don’t work so well under nonclassical conditions, don’t you? I wouldn’t use them in the vicinity of a black hole or in a degenerate gas, for instance.
Sambo, the 3-body problem is not soluble in the general case in the real world. That reality, she’s a bitch, ain’t she?
BPL (#179)
I suggest you stop being presumptious. Read what I stated in 164. I said there isn’t an analytical solution, and the perturbations are due to the numerical methods and imperfect information. I think Laplace 1826 would support that statement.
In any case for the practical engineering decisions, you rarely need the three body solutions. You typically restrict the problem to a two body solution since you don’t need the three body one.
If you reply again please don’t hypothesis what I have and haven’t understood from my degree. It’s rather anoying and you’re making a lot of false assumptions.
Oh, BPL…. I know from experience that you are immune to reason, so I’m not surprised to see you base your beliefs on a single test that you ran yourself.
But just to give you a clue where you are going wrong: you are confusing “benchmark” with those general purpose PC-benchmarks that the computer magazines invented for consumer testing.
Why do you have to be so absolute about everything? The algorithm is the important factor, and for some implementations Fortran will beat C, and in other cases C will beat Fortran. Fortran code is only “nearly as good as assembly”* if an expert spends some considerable time making it so. But the same is equally true of C, or any language that allows that level of control. For a while, Fortran had a significant advantage in that the available Fortran numerical computing libraries were already highly optimised. But that advantage has gone long ago.
* What you meant, of course, is assembly hand-crafted and tuned by an expert. It’s perfectly easy to write inefficient code in assembly, you know.
Did 182: I know from experience that you are immune to reason
BPL: Your kind of reason, anyway. I think most people are.
Didactylos wrote @172
The Muir Russell report discusses reproducing results using completely different methods and data. Ah, but I was talking about verifying that the method used to obtain the result is correct, i.e. meets specs and no bugs. I think we will have to agree to disagree on this point.
And you really need to do some research on the n-body problem. I’ve retired gave away my books but IIRC, Newton’s law (ignore relativity) says the trajectory of a probe satisfies some ordinary (not partial) differential equations. The equations cannot be solved analytically but it can be solved numerically and maybe iteratively. It is an initial value problem but there are errors in measurements if the initial values. Let t=0 at the time the engine on the probe separates. The solution to ordinary differential equations can be calculated. But the initial position and velocity vector at t=0 has errors. The force solar winds and effect of the asteroid belt cannot be calculated with great precision. Hence the need for some error correction.There is no discrepancy with the theory. However, I think the corrections (margin of error) is very small.
John Reisman @175 wrote
IN other words the science is understood enough in key areas to see that certain conclusions can be drawn with sufficient confidence on the robust results of the science to take action without inordinate amounts of guessing or exercising the precautionary principle.
Ah but who understands the key areas. The average voter without a degree in climate science does not understand. He/she takes much on faith. The problem is which faith.
A climate scientist may evaluate the literature understands and makes judgements based. His certainty may be fairly high, but I don’t think it should be as high as Newton’s law. And based on his/her knowledge the scientist advocates takes a position.
I have degrees in physics and math to morphed into a non-degreed computer scientist. In my training in physics I was exposed to some of the history of physics and mathematics. In particular the history of the quantum theory and non-euclidean geometry made an impression on me and helped form my epistemological view on knowledge. In regard to climate science, I decided to investigate about a year ago. I have looked at some of the papers and didn’t understand much of what they said, though I understood the abstract of most of them. So I would rate myself somewhere between the average voter and a climate scientist. But still I form my opinion based on what I know. Climate scientists who have more knowledge will form their opinions on what they know and it may be different from my opinion.
I guess what I am saying is that different people of good will may reach different opinions of climate change. The reasons for the different opinions is reflected in part by their knowledge of the subject.
Philip Machanick wrote @177
it starts on p 150
I don’t have time now to read and comment, but tomorrow maybe
Ray Ladbury @180 wrote And you do know that Newton’s laws don’t work so well under nonclassical conditions, don’t you? Yes, I knew some special relativity, never got into general relativity. You all my previous statements you can replace my relevant references to Newton’s law by special or general relativity. There may be singularities in general relativity but those, I think, are predicted by the theory.
klee12
klee12 said “However, I think the corrections (margin of error) is very small.”
Imperceptible differences in initial conditions can have huge effects on the later trajectory. This is really the very definition of chaos. Did you look at the animation I linked to? The secret of rocketry is that in nearly all cases, small gravitational fields can be ignored, and the problem reduced to a simple one. I don’t think this is true when considering longer trips. Even a trip to the moon involves three significant gravitational forces (a 4-body problem).
BPL: You are right. Most people don’t bother with reason, instead preferring to rely on their instincts, their hopes, and their fears. This is really what denialism is all about. I’m surprised to see you grouping yourself with the deniers, though (unless you were oh so subtly trying to insult me).
Thanks Dr. Schmidt and Dr. Steig.
In the past three years I think we have come a lot closer to the position that people like Nick Barnes and I take.
In general, I think the central argument I would make is this. I do not push for open code because it is scientifically interesting. I do not think that repetition is better than replication. I push for open code because it is beneficial to scientific advance. Absent any cogent argument, absent any factual argument that open code is detrimental to science, I think the arguments for the benefits win.
Because of the release of GISS code Nick and others can now answer some annoying silly questions that Dr. Ruedy should not have to bother himself with. Because of that release people can now engage in “what if” scenarios that the word constraints in journals do not permit. For example, in the forthcoming H2010, hansen will extend the use of nightlights ( 1995 DMSP lights based on F12) to the entire station inventory. because the code is released and up and running in individuals hands we will be able to do the following.
A. re run his analysis using the improved station location data that WMO will publish on nov 8.
B. re run his analysis using the Nightlights data from F16 ( the F12 data has been deprecated according to the PI and he doesnt want anybody using it )
C. Run the code with the Beta V3 data.
and other things as well.
That is a large part of what people like me have wanted to do. Take the code behind a paper and do things that we find interesting without re writing “accepted science” from scratch. It took a long time to get here. But I think you can agree that what CCC have been able to do is a benefit.
So, the point is this. Replication is more scientifically interesting that repetition. No one ever argued otherwise. What many have wanted is the code required to produce the results, so that more investigations can be done with the same code. Is it possible to write code from the descriptions in papers. Sometimes yes, sometimes no. But that has never been the important question. The question has been “is it better to release the code or not?”
In the particular areas I’m interested in, I’ve never seen a piece of code that was better off unreleased.
Anyway, despite all the harsh words over the years, I do want to thank you and Dr. Steig ( and Dr. Benestad as well) for your efforts in freeing up more code. and with that. peace out
P. Machaninck wrote @177
I didn’t find it hard to find the discussion in the Muir Russell report on reproducing results independently but it starts on p 150.
OK, I got it.
It seems that the Muir commissioned replicated the CRUTEM analysis using the data and algorithm and methods listed on following page(s). This replication does not mean verification in my mind, especially if someone else tried and failed to replicate the results.
I took at quick glance at the “Code to Process the Data” which I assume constitutes an methods and alogrithms description. The second bullet item had phrase “ we required good data …. It doesn’t define what good is. I know you want assume that a weather station reporting -100 degrees whatever near the equator is probably in error, but, still, how do you define good? In the next bullet item choose the most valid monthly normals How precisely did you choose. And so on.
I realize that the methods and algorithms of the developers of CRUTEM may have been much more precise than the Muir Committee, but this does illustrate the type of errors, which I call specification errors, which perhaps the scientific community, does not take as seriously as they code.
Ordinary language is not precise a computer program is. On large software projects people work for a long time on drawing up specifications (aka requirements) and break up the project into modules, and assign teams of programmers to work on each module which have precise specifications. The team programs the module up, checks there are no coding bugs, and then the whole program with all the modules are tested together. Frequently, (dare I say always) there are still errors due to misunderstanding of the specifications … something has to be changed. Often the delay in a project is due to delays in the testing part of the project when you put everything together and it doesn’t all fit. I think it is very difficult to detect these types of errors without the code. Replication doesn’t detect speicfication errors if one groups of programmers assumes one interpretation of the specification and another group assumes another. The correct interpretation is revealed unambiguously in the code of the original author.
There may also be coding bugs that should corrected, no matter how insignificant, if they exist.
Didactylos wrote @185
Imperceptible differences in initial conditions can have huge effects on the later trajectory. Correct. I think what the scientists in charge of the probe did was calculate the ideal trajectory, follow the actual trajectory, and if the probe started to deviate from the ideal, give the probe a nudge to get back on the ideal trajectory. There was probably a nudge early on to correct for errors in initial conditions and smaller ones later to correct for unforseen events like solar wind, etc.
klee12
klee12: I feel you are going in circles. “It doesn’t define what good is”, “How precisely did you choose” – the whole point the report was making is that these issues don’t have a significant effect on the result. Hansen’s various treatments of the UHI effect or Menne et al’s analysis of the surfacestation.org data show that even going to ridiculous extremes in excluding suspect data, the result is not significantly altered.
Your comments about specifications are absolutely correct for commercial software, and they should probably apply to large and complex scientific undertakings, too. But there’s no way they apply to the typical scientist’s work, when they are either working alone or in a very small group. Yes, specifications are still important. But the “specification” is the important part, the published part. The code validates the specification, not the other way around. Hence, an error in the validating code should not be propagated to an independent validation. That’s what replication is all about.
You are right that ambiguities still need resolving, but science has mostly managed to resolve ambiguities without publishing every last byte of code. Besides, even if you do have the code, you still don’t know whether the code or the description is correct. Code may be precise and unambiguous, but it is easy to be precise, unambiguous and wrong.
Didactylos wrote
the whole point the report was making is that these issues don’t have a significant effect on the result. I shouild have been more clear. I was not looking at the methods as someone who is trying to replicate the code, but as someone who thinks there is something wrong with it. He/She writes the code with one assumption of “good” and the original author has a different interpretation. Their code gives different results. This was meant as an example of what could happen and could be easily avoided by publishing the code. Even if one assumes in this example the errors will be insignificant, there are other cases where the differences are not insignificant. By the way, how do you define “insignifcant”. When I port code, if the differences between my ported code differs from the results reported for by the code I am trying to port cannot explained by round off or similar machine dependent factors I don’t feel comfortable.
Your comments about specifications are absolutely correct for commercial software,… But there’s no way they apply to the typical scientist’s work, when they are either working alone or in a very small group.
I agree with that statement, but my point (which I did not emphasize to enough) was that what I call implicit specifications are important and can cause others not to be able to reproduce your results using just methods and algorithms. Evidence for the importance is the efforts to spell them out on large projects and even when immense efforts are expended to write the specifications errors often occur. The code is so much easier than spelling the specifications out. You don’t need to write out the specifications; the specifications are implicitly very obvious in the program … “good” may mean within 3 standard deviations of an average and it’s in the code. Again I’m looking at this from the point of view of a person who thinks the author made a mistake and want to check.
klee12
klee12 said “By the way, how do you define “insignifcant”.”
I try to leave that to the statisticians – but speaking informally, I mean “does not alter the conclusions”.
“Even if one assumes in this example the errors will be insignificant”
Nobody is assuming anything. In this particular example, the errors ARE insignificant. In other examples, they may not be – but in that case, the very first attempt at replication should identify the problem. Remember that scientists do share code and talk to one another, even if the whole codebase can’t be published.
“looking at the methods as someone … who thinks there is something wrong with it.”
Exactly. And it is this assumption of bad faith that really irritates scientists. Then they open up the code, and people STILL call them frauds. Where’s the upside? No. Scientists should be allowed to open up code for their own reasons, not for yours.
I’m going to ignore your tiresome repetition of “code = perfect specification”, since it doesn’t hold water and it has been explained to you dozens of times now.
Did 185: I’m surprised to see you grouping yourself with the deniers, though
BPL: Wake up, and it will all be over. You’ll be back in your bed, ready to go to school.
klee12, OK, let me get this straight. With a BS in physics and a single year of trying to read papers you didn’t understand, you feel you are competent to pronounce judgment on a century and a half of science done by an entire community.
Dude, I’ve got a PhD in physics and have been actively doing research in physics and it took me 3 years of fairly concerted effort before I felt I really understood the basics of climate science.
Here’s a place to start: Google “Dunning-Kruger”.
Ray Ladbury wrote “it took me 3 years of fairly concerted effort before I felt I really understood the basics of climate science.” Me too and I started with the advantage of 50 years, now, as a amateur (hobby) geologist.
David Benson,
I wouldn’t call myself a geologist. I’m a rockhound–nearly every surface in my house would wind up coated with rocks if my wife let me.
Ray Ladbury @194 — I prefer to leave the rocks where they are and go visiting. I’ve also read a middling amount of the literature, which is what did help in picking up some aspects of climatology.
Didactylos @ 190
Let’s just agree to disagree
Ray Ladbury @192 wrote
With a BS in physics and a single year of trying to read papers you didn’t understand
you feel you are competent to pronounce judgment on a century and a half of science done by an entire community. Nope, never said that. I said @184 So I would rate myself somewhere between the average voter and a climate scientist., and I stand by that statement. In @151 I wrote
Now I feel I’m being invited into discussing climate science. I’m not a climate scientists and I’m don’t want to go there….I don’t know much about climate science and there’s nothing much I can add on that subject. My view, what whatever it’s worth, is that climate is warming, that humans are adding to the warming and that we should reduce CO2 emissions due to the precautionary principle.. You might look at the context also to understand what I was getting at.
I don’t think paper degrees are relevant in science. What is important is does argument reasonable. If you think I made a mistake, let’s discuss it in a civil manner. If we reach agreement good; if not that’s OK too. I did pontificate a bit on computer science, on which I felt I was on firm ground but I am sure I didn’t express a definitive position on climate science.
klee12
Klee12,
At this stage, what I can do is recommend resources to further your education. I can also recommend that forays into philosophy of science (e.g.climate change is less certain than Newton’s laws) wait until you’ve actually done some science so that you see how the process works. Newton’s Laws work great in everyday life. However, the Newtonian model of space and time is falsified, and therefore so are Newton’s equations. The consensus model of Earth’s climate is alive and well and quite unlikely to be “falsified”
If you have not read Spencer Weart’s history, I recommend you do so. It will give you some idea of how we got where we are. Beyond that, ask questions. This is a complicated field. It takes work to understand it.
#192 Ray Ladbury
The only degree I have is a degree of insanity and I still don’t understand much (ok, I did have two years of engineering and electronics, but nothing can prepare you for the real world of climate science ;)
But what I don’t know, does not prevent me from at least a reasonable understanding of the basics.
There’s that GHG blocks IR thing
CO2 and spectra wavelengths
the H20 thing
albedo
land use
carbon sink
ocean acidification
lat shift
oceans warming
bunches of letter combination’s (ao, aao, nao, pdo, enso, etc.)
path divergence and attribution
CCN stuff (ugh)
atmospheric lifetimes of different thingys
seasonal shift
inter/intra/multi-decadal
columns, and models, and layers oh my!
oh, and the um, the, heck what was it, oh yeah, the feedback things, fast and slow…
Of course, I’m still learning :)
Economics: Balancing Economies
The Leading Edge: The Cuccinelli ‘Witch Hunt”
—
Fee & Dividend: Learn the Issue – Sign the Petition
A Climate Minute: Natural Cycle – Greenhouse Effect – Climate Science History – Arctic Ice Melt
Hi All
I find all the talk of making the code available interesting, however as a Systens Project Manager, I invariably find the actual code to be far less interesting that the journey that was undertaken in building the code.
Are there places where the following is available? Or are you considering making the following available? Its not clear from the discussion..
Functional Specifications
High level and detailed designs
Test scope by phase of testing
Test conditions for unit, assembly, etc
Expected results from test runs vs the actuals?
Defect trend reporting by test phase
Systems Test closure reports
Acceptance testing: scope, conditions, expected vs actual, defect reports, and closure
With all due respect to my colleagues who write the code, in my opinion, making the above available would have a far greater impact than merely the code which is simply one (albeit important!) output of a specification?
Gerry
It is interesting to see how similar problems are addressed by scientists in widely different disciplines.
The open/closed code argument spans many disciplines and subdisciplines. Although specific code is relevant to the interpretation of behaviors of the code under different, sometimes slightly so, initial conditions or context, it is not the most important issue with respect to the value of the science that employs it. Nonetheless, it is often the case that different perspectives as to what the relevant issues are may welll be decided by subtle issues such as ill-conditioning, aberrant behavior under certain rare but specific circumstances, etc., such that having the code in hand, if necessary is important.
Consequently, much of the debate here could be easily addressed not by archiving every line of code or endless debating what should or should not be archived, but rather by providing a registry, where authors could provide either sufficient metadata or code as the need may be to settle substantive differences in interpretation.
The community need only decide what constitute adequate or sufficient metadata to extract suitable information, and sometimes code, from the registry. Relevant metadata might include descriptors say, original publication, authors, source language, licence issues, method of distribution, website for more information, etc.
One might view a useful registry as simply a multigraph, with nodes representing the particular issues/formulas/phenomenona and the edges as the implementations (aka code or algorithm). Since there may be a large number of possible ways to code for the same problem, the nodes may have many parallel edges. For a variety of reasons, some edges would be more interesting/useful than others since they may do a etter job at addressing ill-conditioning, computational-efficiency, convergence, etc. Where two papers utilize the same algorithms and/or code they would in principle share the same nodes and edges.
It may be that the graph may need to evolve/change to either include more or different nodes, depending up need/scope/context for such a registry to remain current or useful. Probably, a very simple vocabulary to differentiate nodes would provide a flexible starting point for such a registry. This would require some standardization for the metadata, so that searches could be usefully constrained and specific algorithms/research could be systematically categorized.
Authors (registrants) would be only required to meet minimally sufficient requirements in metadata usage to classify their work. Those who register or extract from the registry would be responsible for proofing/usage/interpretation/distribution/etc. beyond the minimal metadata requirements. Consequently, management overhead could be kept to a bare minimum. After all, scientists should be willing to stand on the merits of what they register.
At that point, one has the framework that can be pointed to address concerns as to precisely what was done and how it was done. The public, as well as the research community, then has an effectively open record and the community has a resource that can be tailored to better share useful methods/implementations/code/algorithms that can be refined as the need arises and resources become available. Such a registry would also have the useful benefit of providing a snapshot as to what methods/code/issues are most used, sources of controversy, etc. Importantly, it would go far to address the public and political perceptions that somehow something is being hidden (there may of course, be perfectly good reasons, proprietary code, copyright issues, patent rights, contractual obligations, etc) that may make totally open source a goal rather than something always achieved.
Nonetheless, a registry would facilitate clarification and ongoing research, as well as debunk the too often bandied about notion that scientists have something to hide, while leaving the onus on those who do the work to have used appropriate methods/procedures as well as on those who would criticize them.