AGW Climate Change Culture Dissent Global Warming Omniclimate Science Skepticism

Non-technical Comment For Curry's Technical Thread

This comment of mine in reply to Judith Curry’s “Pierrehumbert on infrared radiation and planetary temperatures” might or might not survive the night at Climate Etc: 😎

Well, if you are wondering whether any progress in understanding of an issue like this can be made on the blogosphere you can’t leave this thread as purely technical can you?

raypierre is the perfect example of what has gone wrong between mainstream climate scientists and the rest of the world, starting from the “blogosphere”. One can be the smartest kid this side of the Virgo Supercluster of galaxies, but if in one’s mindset questions are considered as instances of lèse majesté and examples for the general public are routinely simplified in the extreme, so as to make them pointless, well, one will only be able to contribute to the hot air: because the natural reaction of most listeners will be to consider whatever one says (even the good stuff) as just a lot of vaporous grandstanding.

From this point of view, SoD‘s work should be more than highly commended.

AGW Climate Change Global Warming Omniclimate Policy Science

Data Sharing And Proprietary Periods, Away From The Climate Madness

The policy for the future mission will require the immediate dissemination of data.
“You get a larger community and you get a bigger workforce for free,” he says.
“It’s clear that the more people you get involved, the more support you get.”

How often have climate scientists lamented their inability to share their data, either because evil people might use them “improperly”, or because it is unfair that somebody take advantage of all their hard work.

Is this a problem of climate science alone? Perhaps, it is. How strange.

From the UK Space Agency, about the Planck mission:

the proprietary period, which nominally ends with the release of the scientific products to the community 3.5 yr after launch, i.e. in January 2013.

From the Planck mission organization itself: “Planck Science Management Plan

The nature of the mission is such that the data products can only be generated from an analysis of the full set of data; i.e. distribution of partial sets of data before the complete set is processed is not useful and will be avoided. In particular, all of the data acquired over the duration of the mission will be simultaneously used to calibrate and remove systematic effects. Thus, the time at which the 1-year proprietary period begins can be appropriately defined as the time at which the 9 all-sky channel maps (the main product described above) have reached a level of maturity such that they could be delivered to the community. The exact time when this level is reached will be determined by the PST, but will not be later than 1 year after the spacecraft power has been switched off.

It is worth noting here that the 1-year proprietary period will also be used to prepare the physical means of distribution of the scientific data products to the astronomical community, and the associated documentation (the ?Explanatory Supplement”); given the large amounts of data involved (in particular in the time series) this operation will require a non-negligible amount of time.

The Planck Mission Science Case, from Helsinki:

The observing program lasts 14 months. The Planck data will be made public two years after completion of the mission, in 2012. Prior to this, during the proprietary period, the data is analyzed and used by Planck Scientists and their collaborators

And now about the large European astronomical observatories (note how much emphasis there is on making the data easy to share):

Larger telescopes, more sophisticated instrumentation and solid state detectors have increased the efficiency of the astronomical observations by several orders of magnitude. All the data collected by these new instruments are produced directly in digital form and stored on computer accessible media. Controlled calibration procedures are applied to the raw data that then fill Science Archives that are readily accessible via the WWW for scientific analysis and research. For most of these Archives, the data they contain are becoming publicly available after a short period of proprietary time.

Most observatories are currently distributing Principal Investigator data to their user communities after the proprietary period has expired. This enables the scientific utilisation of these data for completely different purposes than the ones of the original proposer, and thus optimises the scientific return of observation infrastructure. But the distribution method is quite variable, from distribution on demand to full on-line archives with sophisticated data retrieval tools.

Observatories and organizations such as ESO, ESA, PPARC, CNRS and Jodrell Bank have invested telescope time and financial resources in producing archives with the potential to be reutilized for a variety of scientific purposes. The AVO offers the opportunity to federate these resources and provide astronomers with access to an integrated multi-wavelength repository of data for data mining and enabling new science.

Astronomical archives are potential a valuable teaching tool for school and university students. ESO and ESA have received funding for education purposes through the European Week for Scientific and Technological Culture Program. Through these efforts ESO has realized the magnitude of the potential utilization of archives by teachers and students. By enabling uniform and interactive access to distributed data archives, the AVO will provide an environment in which the public can participate in exploring the digital cosmos and in which teachers can build new educational resources.

A basic issue to be addressed during the initial phases of the AVO design is the uniformity in the quality of the data. Different Archives and, within them, data from different telescopes and instruments, have adopted different quality control procedures and indicators. The issue is particularly acute for the data collected by individual PI (Principal Investigators) that become public after the expiration of the proprietary period. The first step in addressing this issue is a review of the scientific quality of the existing Archives and the drafting of the scientific requirements for the standard quality control procedures to be adopted by Archives that want to be part of the AVO. The study will proceed by defining a set of quality indicators that are suitable for use by automatic querying and browsing systems. In some cases, the production of higher level Archive products (metadata or specific survey products) will be suggested and implemented.

Blast from the past, when oceans could be worked upon without having to worry about politics: the World Ocean Circulation Experiment (WOCE) 1990 – 2002 (including an appropriate data quality control process)

The WOCE Hydrographic Programme Office in Woods Hole was the first WOCE Data Assembly Centre (DAC) to be established (in 1990). Three years on we can look at how well we are doing. WHPO ‘data’ is digital data of either station locations and times or CTD and bottle data. These data with their attendant documentation come from PIs and at the end of the WHPO involvement are transferred to the WHP Special Analysis Centre (SAC) in Hamburg. After the proprietary period is over, data are made available to the WOCE community on an ftp server. The start of the path is the submission to WHPO of data sets and documentation by individual Chief Scientists. The WHPO usually re-formats the data and, in some cases, generates information files that are not otherwise available. Data are then sent out to a Data Quality Evaluator (DQE) for independent assessment of the quality. In the case of some repeat hydrography cruises, we hold a data set until we receive another one from the same group since it is as easy for a DQE to evaluate two data sets in the same region as one. Once the DQE reviews are complete, they are sent to the Chief Scientist, who is asked to revise his data set, (usually only by changing data flags or submitting additional information). The revised data set is re-submitted and, after further checking at the WHPO, is sent on to SAC and made publicly available once the proprietary period of 2 years elapses.

In fact this is what the WOCE home page shows:

The WOCE data set is the most comprehensive data set ever collected from the global ocean.This site provides access to the final (3rd) Version of the data set collected during the World Ocean Circulation Experiment (1990-1997).

This page provides access to:

Browse the WOCE Data DVD Disk One

Browse the WOCE Data DVD Disk Two

Browse updates and amendments to the WOCE Data DVDs

WOCE 2002 Conference Presentations

WOCE Global Data Resource (PDF, 19.8MB)

How about the Hubble Space Telescope (where we are told what NASA does when it’s not climate in the picture)

In accordance with NASA policy, all science data from the Hubble Space Telescope is archived with a one-year proprietary period by default. This period may be extended or shortened at the request of the principle investigator (PI) and on approval by the STScI Director’s Office. Calibration data (i.e., data obtained under calibration proposals), by default, carries no proprietary period; neither does engineering data, calibration files (derived from calibration observations), and observatory monitoring data.

In addition to the regular proprietary period, observations by General Observers (GOs) which are found to be duplicates of concurrent observations by a Guaranteed Time Observer (GTO) may be placed under restriction. Data under restriction (or “embargoed”) cannot be distributed to the GO until the restriction expires (usually, when the GTO data goes public).

How easy is it, to get a Proprietary Rights Modification? Not much:

Requests for proprietary rights modifications (extensions, reductions, and/or restrictions) should be sent to the archive hotseat ( The hotseat staff will forward all requests to the appropriate authorizing officials. All requests must be approved by the Director’s Office before being enacted.

More at the Space Telescope Science Institute, details about the “Extention of Proprietary Data Period”:

While individual images of my target will have some value, the scientific goals of this program are possible only when the entire dataset has been collected. I am now requesting that the proprietary data period be set to run from the completion of the last visit in the set. The timing on this request is important since your answer will greatly impact the upcoming budget submission.

Requests for non-standard proprietary data periods are unusual and are granted only in exceptional circumstances. Such a request should be addressed to Duccio Macchetto as the Associate Director for Science Policy.

And finally…on 14 April 2010, Nature magazine hosts a protest against “Telescope team may be allowed to sit on exoplanet data“:

on Monday a NASA advisory panel recommended that Kepler be allowed to censor 400 “objects of interest” — presumably good planet candidates — until February 2011, giving the mission team more time to firm up discoveries, rule out false positives and publish. If enacted, the new policy would represent a selective editing of data on the basis of its science content, rather than its quality — unprecedented for such NASA missions.

Many astrophysics programmes allow researchers a proprietary period with the data. For instance, guest observers on the Hubble Space Telescope get exclusive use of their data for a year before public release. But the tradition for NASA Discovery missions — small, principal-investigator-led missions like Kepler — is to make calibrated data available immediately. That policy has already been changed once for Kepler, last year, when the team was given more than a year to pursue confirmations and work out the kinks in its data processing.

Exoplanet astronomers outside the mission, however, are critical of what they say is an overly cautious approach. Scott Gaudi, an astronomer at Ohio State University in Columbus, says external astronomers might help the Kepler team, as it will be unable on its own to follow up and confirm all its candidate planets. “I think Kepler is being far too conservative, and far too closed about what’s going on,” he says, “and I think it’s to the detriment of science”.

[Malcolm] Fridlund, who is ESA’s study scientist for a planned follow-up planet-hunting mission called PLATO (Planetary Transits and Oscillations of Stars), wants to do things differently next time. The policy for the future mission will require the immediate dissemination of data. “You get a larger community and you get a bigger workforce for free,” he says. “It’s clear that the more people you get involved, the more support you get.”