1 / 24

Observing with Modern Observatories: The Data Flow

This lecture layout discusses the data flow in modern observatories, including proposal submission, observing modes, data acquisition, archival, data distribution, and support systems.

alambert
Download Presentation

Observing with Modern Observatories: The Data Flow

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NEON School 2004 Observing with Modern Observatories (the data flow)

  2. Lecture Layout • Proposal Submission • Phase I • Observing modes • The proposal submission • The OPC selection • Long term scheduling • “Web Letters” • Phase II • P2PP, OBs, etc. • Observation • Short-term scheduling • Data acquisition

  3. Lecture Layout • Archival • Connecting meta data to pixels • Data shipment (or replication) to Garching • Quality Control • Preparation of master calibration files • Preparation of reduced products for PIs of service mode programs • Archive use • Data distribution to PIs • Data distribution to archive users

  4. The data flow: Another View

  5. Phase II proposal submission observing blocks medium term schedule User Support Group Operations Technical Support Data Archive & Supporting Databases storage and re-distribution PI packages distribution Phase I proposal submission program selection time allocation Observation short term schedule archival of data Paranal/La Silla Observatories Quality Control Master Cal files prod. Instrument trending PI products preparation Data Flow Operations Group The Data Flow The ESO Data Flow Model and its Supporting Organizational Structure

  6. Phase I: Proposal Submission • Provide information: • Identification of PI/CoIs • Facilities to be used • Split program in “runs” per facility/group of targets • Observing preferences • Scientific Justification • OPC selection: • Peer review of proposal • Below or above threshold? • Long term scheduling • “Web Letters” • Did I get time or not? • Message sent to all PI with link to explanations program run

  7. Phase I: Observing Modes • Two observing modes: • “Visitor” mode • Keep contact with telescope and instrumentation (observing should not become a black box/magic activity) • Need to use a specific observing mode • Need to be able to assess quality and react immediately • “Service” mode • Program is to use normal, supported instrument modes • Program requires multiple visits of the targets • Program has lots of targets spread in r.a. space

  8. Phase II: Send Details • Provide detailed information: • About each RUN in the PROGRAM • About each target in the proposal • About instrument settings to be used • Define observing sequence • Define exposure times, filters, constraints (e.g., on the seeing) • ==> Observing block (OBs) program run OB

  9. Observation • Steps in the observation sequence: • OBs sent to La Silla/Paranal • Short term scheduling to match required target visibility, moon brightness, seeing, … with prevailing conditions • OB execution • ==> exposures • Two types of exposures: calibration and science program run OB exposure

  10. Archival • Connecting meta-data to pixels • Collecting telescope, environment, instrument and user provided details • Building a header with this information according to a pre-defined dictionary. • Assembling header and pixels into a FITS file. • Sending file to the archive system • Storing the file onto the archive medium (DVD, Mag disk) • Decomposing header and storing info into database • Replicating database to Garching • Sending archive media to Garching

  11. Data Quality Control • Building calibration database • Assembling master flats, darks, biases • Sending those to the archive for future • Preparing calibrated science products • For each run, build a structure with raw, calibration and calibrated data for the PIs of service mode runs. • Send a “pack & send” signal to the archive

  12. The Archive • Collects data: • Information about PIs, programs, runs, OBs, exposures • Pixel data (raw science and calibration files) • Distributes data: • Proprietary data to PIs • Public data to anyone

  13. Phase II proposal submission observing blocks medium term schedule User Support Group Operations Technical Support Data Archive & Supporting Databases storage and re-distribution PI packages distribution Phase I proposal submission program selection time allocation Observation short term schedule archival of data Paranal/La Silla Observatories Quality Control Master Cal files prod. Instrument trending PI products preparation Data Flow Operations Group The Data Flow The ESO Data Flow Model and its Supporting Organizational Structure

  14. The Archive: Content • Contains ESO and HST raw data • Started operations in 1991 • Growth never stopped • Need to constantly adapt to new storage technology to keep costs low • Need to frequently adapt to current software user interfaces to provide users with better search tools (more and more data) • Need to look for “advanced” products to empower users with doing science

  15. The Archive: Content

  16. The Archive: Output • Data retrieval model is that of the Web Shop: • Browse catalog • Mark articles of interests • Go to “cashier” • Select delivery options • Confirm submission

  17. Request Submission: 1) Browse

  18. Archive: Selection of Frames of interest

  19. The Archive: Selecting Request Options

  20. The Archive: Output

  21. Two types of Archives • Technical vs. Science Archive • Current system more tuned to serve the operational needs of the Observatory (data collection and distribution to internal/external users) • Science users still mostly left out (emphasis on raw data) • Discrepancy understood early by the VO “founding fathers” • Need to provide access to well-described, instrumental signature-free data • Need to provide hassle-free access to different archives without affecting their operation critical mission • Science users will get better products and access through the VO

  22. Archives for the people! • The three layers • Separate the mission critical support role from the delivery of products • Leave the science service provision to the VO • Provide interfaces to the top layers (e.g., image and spectra access protocols, high-level data processing grids) Virtual Observatory VO tools and facilities Instrumental signature-free data “Real” Observatory Data In/Out

  23. Two type of archives • Technical vs. Science Archive • Archive downloads continue to increase, but usage increasingly technical: • Multiple data deliveries to PIs/CoIs • Easy retrieval of own data • Streamlining of internal data channels • Science usage still limited: • ESO Archive still has too few instrumental signature-free products to offer • Astronomers will produce archive-based science if they are paid and/or supported! (e.g., Hubble’s archive proposals, ASTROVIRTEL)

  24. On-going efforts in the ESO archive • News from the “technical” archive: • Efforts on-going to put more reduced products available • Better web-based search screens under development • Data available world-wide probably in 2005 (ESO Council to decide) • News from the “VO” side @ ESO: • On-going efforts to provide a multi-archive query system (Querator) • Image and spectral VO access implemented as soon as standards approved • Extra ESO resources arriving soon to help with all of the above

More Related