Operational Database

The Operational Database (OPDB) archives and provides efficient access to all data necessary for the conduct of Survey Operations.

Implementation

The OPDB consists of two components:

1. A commercial object-oriented database (OODB, Objectivity), providing disk-based storage to that data which must be efficiently accessed for normal Survey Operations. Access to the data is via C++ and TCL wrappers for the C++ code.
2. A hierarchical tape storage system, providing storage of pixel-based data (raw and corrected images, atlas images, masks, etc), for which efficient access is not required for normal Survey Operations.

Functions

The OPDB provides the following functions (all but the first are implemented entirely within the OODB):

Archives the Data Acquistion Sytems (DA) and Pipeline outputs.

Multiple copies of all data are stored. The DA and Pipeline outputs can be reproduced "exactly" from the OPDB. The exact manner in which the Pipelines ran are recorded, including the Pipeline version, settings for tunable parameters, and the specific inputs to the Pipelines.

Provides efficient recalibration of the raw Pipeline outputs.

The raw outputs (pixels, counts, etc) from the ps and frames Pipeline are stored in the OODB, along with the astrometric and photometric calibrations (from the astrom and fcalib Pipelines, respectively). Multiple calibrations may be stored. Calibrations may be applied on the fly. New calibrations may be obtained by re-running the astrom and fcalib Pipelines from inputs reproduced from the OODB alone (re-running ssc, ps, and photo all require fetching inputs from the hierarchical tape storage system).

Provides efficient QA.

Objects in overlapping areas are matched up in the OODB. Tools are provided to perform QA based on the matched pairs.

Tracks survey progress.

The following functions are performed within the OODB (on "lite" versions of the objects for efficiency) to track survey progress:

1. Objects in overlapping regions are matched.
2. Data quality is assigned on a field-by-field basis.
3. Pieces of sky are defined as done by assigning segments of imaging scans, along with specific astrometric and photometric calibrations, to those pieces of sky. Objects within those segments are assigned as "primary" or "secondary" survey objects.

Perform target selection.

Target selection is performed within the OODB, applying calibrations on the fly.

Track tile assignment and plate drilling.

All spectroscopic targets assigned to tiles are recorded in the OODB. Each time tiling is run, the OODB provides the list of untiled spectroscopic targets. Similar functionality is providing in support of plate drilling.

Export calibrated objects to the Science Database.

Calibrated objects are exported to the Science Database. Objects may be re-exported when new calibrations are available.

Status

All functions outlined above are currently supported. The performance speed in all cases is satisfactory. There have been no problems to date as the database grows (the size of the OODB component is currently 40 Gbytes). Security of archived data has yet to be proven. Problems, due to bugs in the commercial OODB, have been encountered both with schema evolution and incremental backups (work arounds exist for both problems).

Still To Do

The following functionality is not yet implemented:

1. Archive Spectroscopic Pipeline outputs. While the Spectroscopic DA outputs are archived, the Pipeline outputs currently are not.
2. Full back-up strategy must be implemented. Currently, back-ups are done by hand on a semi-regular basis. Next release of the commercial OODB should support the full implementation.
3. Data verification tools.
4. Interface to survey planning tools.
5. Interface to more of the day-to-day tasks of Survey Operations (tracking data reductions, etc).
6. Archiving Photometric Telescope data and reductions within the OODB (which is currently fully supported, however we're not doing it now, and may never want to).