1 / 18

The VISTA Data Flow System

The VISTA Data Flow System. Jim Lewis, Mike Irwin, Peter Bunclark, Simon Hodgkin Cambridge Astronomy Survey Unit. Introduction to CASU. Small group within the Institute of Astronomy, specialising in survey astronomy. Pipeline reduction of imaging data APM (Schmidt Plates 1 x 40k x 40k)

chaue
Download Presentation

The VISTA Data Flow System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The VISTA Data Flow System Jim Lewis, Mike Irwin, Peter Bunclark, Simon Hodgkin Cambridge Astronomy Survey Unit CAL 07 Garching

  2. Introduction to CASU • Small group within the Institute of Astronomy, specialising in survey astronomy. • Pipeline reduction of imaging data • APM (Schmidt Plates 1 x 40k x 40k) • INT Wide Field Camera (4 x 2k x 4k) • ESO WFI on 2.2m at La Silla (8 x 2k x 4k) • MOSAIC-1 on KPNO 4m (8 x 2k x 4k) • MOSAIC-2 on Blanco 4m at CTIO (8 x 2k x 4k) • AAO WFI on AAT (8 x 2k x 4k) • CIRSI on INT (4 x 1k x 1k) • INGRID on WHT (1k x 1k) • UFTI on UKIRT (1k x 1k) • WFCAM on UKIRT (4 x 2k x 2k) CAL 07 Garching

  3. Introduction to VDFS • PPARC funded facility to provide an end-to-end data-flow system for VISTA and WFCAM. • Quality control and calibration pipelines • Paranal and Garching • Science pipeline for full calibration of science data. • Cambridge • Science archive acts as the point of access of the reduced data. Plus some further processing. • WFAU, Edinburgh CAL 07 Garching

  4. Data Flow • Raw telescope data is assessed by the summit pipeline (QC1) • Shipped to Garching (discs) • Shipped to CASU for science reduction and calibration. (discs) • Calibrated data shipped to Edinburgh for archiving (ftp) CAL 07 Garching

  5. IR Data Reduction Worries • IR detectors are currently inherently more unstable than optical CCDs. • Some odd electronic effects • Sky emission > 100x brighter than most objects • And it’s variable both spatially and temporally! • Exposure times are short, so data rates are very high. • 200-500 Gb/night expected for VISTA public surveys • Rice tile compression can save factors of 3-4 in 32 bit integer data CAL 07 Garching

  6. VDFS Pipeline Recipes • Create master calibration frames (dark, twilight flats, confidence maps, etc) • Linearity analysis • Detector noise & dark current properties • Persistence and crosstalk analysis • Illumination correction analysis • Full reduction recipes for standard star and programme fields CAL 07 Garching

  7. Processing Steps • Reset correction (debias) • Linearity correction • Dark and reset anomaly correction CAL 07 Garching

  8. Reset Anomaly (WFCAM) CAL 07 Garching

  9. Processing Steps • Reset correction • Linearity correction • Dark and reset anomaly correction • Flat field correction • Background correction (defringing) • Destriping CAL 07 Garching

  10. Stripes Close Up (VISTA) CAL 07 Garching

  11. Stripes (VISTA) CAL 07 Garching

  12. Processing Steps • Reset correction • Linearity correction • Dark and reset anomaly correction • Flat field correction • Background estimation and subtraction (defringing) • Destriping • Image persistence and detector crosstalk removal CAL 07 Garching

  13. Crosstalk (WFCAM) CAL 07 Garching

  14. Processing Steps • Reset correction • Linearity correction • Dark and reset anomaly correction • Flat field correction • Background estimation and subtraction (defringing) • Destriping • Image persistence and detector crosstalk removal • Interleaving • Dithering/Jittering • Catalogue generation • Astrometric calibration • Photometric zeropoint calibration • (Tiling) CAL 07 Garching

  15. Summit & Garching Pipelines • QC1 parameters • e.g. photometric zeropoints, astrometric fit quality • Written using ESO qfits/CPL infrastructure • Both use the same software modules • Reduce a pawprint • Amount of processing can be scaled down • Calibration images (flats etc) from a master library. CAL 07 Garching

  16. Cambridge Pipeline • Full reduction • Tiling and sky correction • Catalogue generation is done for both pawprints and tiles • Results shipped to WFAU • Some of the same CPL based modules reused. Some additional modules required. CAL 07 Garching

  17. Current Status of VISTA Pipelines • Version 0.4 ESO pipelines currently being tested • Version 0.5 to be released end of February • Full compliment of recipes (almost!) • PAE • WFCAM pipeline is a prototype for the extra functionality required by Cambridge VISTA pipeline CAL 07 Garching

  18. A Typical Recipe Run… CAL 07 Garching

More Related