Rescuing data…

Guest commentary by Vicky Slonosky

One of the lesser-known branches of climatology is historical climatology, the study of past climates from historical records of instrumental observations and weather descriptions, with “historical” often loosely taken to mean “pre-government-organized-meteorological or weather service” observations. In the last ten years or so, with the development of digital imaging, increased storage and processing capabilities of computers, it’s become possible to think about analyzing these past records on a daily basis. Snowstorms, hurricanes, cold snaps or heat waves all happen on daily scales, and daily information over the past 200 years or more let us learn more about them. All over the world, efforts (many unfunded) are going on to try to find and preserve these historical data from personal weather diaries, scientific observatories and naval records. This is often described as “data archeology and rescue”, which has a nice ring to it, and even manages to make going through dusty boxes in dustier basements and typing in the original (often handwritten) observations sound glamorous and exciting. It’s not a simple task though:

“All my attention for recent years has been directed to that branch of meteorology connected with ascertaining the mean temperature of Lower Canada. I had not made up my mind as to the fact of the climate having materially altered but was impressed generally with the belief that the climate had improved, or become warmer, as it had increased in population and cultivation … I resolved to make diligent search for such records as might be found in the Province, in order to answer this interesting question. The difficulty of this experience was even greater than I had anticipated. Few individuals had turned their attention to this (even now) infant science, at a period sufficiently removed to bear comparison with the present, and of those which I was fortunate enough to discover, many were unsuitable from the irregular manner in which they were kept … None but the most zealous meteorologist knows how very difficult it is to obtain observations in this science which can be depended upon.”

John Samuel McCord, acc. 0882, McCord Museum of Canadian History Archives, circa 1838.

Nearly 175 years later, anyone who has spent time trying to work with observational climate data, especially in their original form, can understand McCord’s frustration.

There are entire conferences devoted to the issue of data quality and homogenization. Not only are there the classic issues of instrument changes, changes in observing practice, changes in observing times, changes in instrument exposure and environmental change (especially urbanization) to contend with, but there are also the newer issues of digital formats, instrument reaction times, and automatic weather stations. That the same complaints about data quality have been going on for nearly 200 years (including a debate about urban effects not mentioned in the above quote) is certainly thought-provoking. Knowing that the history of climatology, as well as the instrumental observations themselves, both go back farther than many of us may realize makes this much more interesting, and makes it even more important to recover what we can of their work.

I can certainly sympathize with McCord. It happens fairly regularly that, once a series of handwritten historical observations have been typed up into digital format, and I’ve squinted over two hundred-year-old handwriting in faded ink (or even worse, pencil), checked thousands or even tens of thousands of observations for mistypes; converted to metric; re-checked for mistakes against modern climate normals; and adjusted for the time of day (which is a project in itself), gravity, elevation, or other sundry instrumental effects, I find out that the previously unnoticed, nearly undecipherable comment in the bottom corner of a page turns out to say something like “back in town again after spending the summer at the country cottage”, or “new [unspecified] thermometer arrived from London”. At this point I usually start tearing my hair out, wonder why on Earth I became a climatologist, and decide that almost any other undertaking would be preferable to trying to sort out these messy data, even cleaning out the kitchen cupboards. (My family spends a fair amount of time wondering where the plates are).

But of course, for all my complaints, like McCord who kept records of the weather until a few days before his death, I love this work. I always come back to it, once I fail yet again to figure out where the best place for the teacups would be, and use Google Earth and historical maps to find out just where the country cottage was, or look for receipts for shipments from London (and fortunately my hair grows quickly). Finding an unexpected weather event recorded in the observations, like a waterspout traveling up the St-Lawrence River, or descriptions of a strange haze which might have been caused by an Icelandic volcano in 1785, is like finding long-hidden treasure. Some of these early meteorologists were marvels of dedication and diligence, such as Alexander Spark of Quebec City who recorded the 8am temperature on 97% of the days between 1798 and 1818, or John Bethune who kept records in Montreal nearly every day from 1838 to 1869. Sometimes non-climatic events get mentioned too: the abdication of Napoleon, a description of the noise and smoke of the guns from a battle during the War of 1812, or comments on the first publication of “On the Origin of Species” turn up in notes and margins.

“Knowing from experience how difficult it is to obtain even these scanty data on which to base a comparison, I have [tried to preserve them] in order that future students of this interesting but infant science…may be saved all the trouble and research which fell to my lot.”

John Samuel McCord, acc. 0882, McCord Museum of Canadian History Archives, circa 1838

For the area around Montreal and Quebec City, a large proportion of historical weather records are either the observations kept by McCord himself, or those he collected. A few years back, the Canadian Environmental History organization NiCHE undertook to scan in the paper copies of some of the diaries, and this time last year, with the help members from the Canadian Meteorological and Oceanographic Society, I started a citizen science, volunteer typing data project. The results have been wonderful so far, and enthusiastic and dedicated volunteers have typed in about half of the 19th century weather observations for Montreal and Quebec City. There are always more data to be typed up, though! For the US, New England is an especially rich source of historical weather data, and the Atlantic provinces of Canada also have weather diaries going back into the 1700s.

Example pages of different weather diaries are posted on a website, where volunteers can view sample images and choose a diary or section of a record to type in. I then set up a spreadsheet form for each different record, and put this, the image files of the diary, and a readme file with questions, explanations, tips, etc into a DropBox folder which is shared between my account and that of the volunteer. As the form and abbreviations used by each observer were generally idiosyncratic to that observer, each volunteer works with one diary at a time, as it can take a while to figure out the abbreviations and notation of the diarist. Some long diaries are split into sections and shared out between several volunteers.

If you think you might be interested in helping to rescue North American historical climate data, you can look at this website. There are similar projects around the world, listed at ACRE (Atmospheric Reconstructions Over the Earth) site, including projects digitising German radiosonde data and old station data from the Solomon Islands.

So, if you feel you’d like to make a contribution, note that more volunteers are always welcome!