1 / 25

EIS Data Reduction System

EIS Data Reduction System. May 12, 2005. Top requirements. User-friendly interface (free critical resources) Data management transparent to user Integrated operation environment Un-supervised reductions & processes (minimize errors, increase throughput)

manju
Download Presentation

EIS Data Reduction System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EIS Data Reduction System May 12, 2005

  2. Top requirements • User-friendly interface (free critical resources) • Data management transparent to user • Integrated operation environment • Un-supervised reductions & processes (minimize errors, increase throughput) • Automation of the system to support 24/7 operation cycle • Instrument independence (optical/infrared) => Reduction mode • Support different obs strategies (wide/deep, crowded) => Reduction mode • Quality assessment of products (self-describable products with QCP) • Uniqueness, versioning and history of products • Integrated WEB services (progress, release) • High-throughput (VST, VISTA) • Platform independence and portability requirements, resources, quality => time

  3. Survey operation model • Every 6 months define/update surveys automatically triggering: WEB pages; OBs, compilation of auxiliary data • Periodically scan ESO Science Archive automatically triggering: WEB update; data request; reduction and calibration of data on nightly basis • End of observing period: create stack/mosaics; catalogs and targets, version 0.5 releases • Yearly: garner comments from users; version 1.0 releases for completed surveys • All pipeline actions registered into an Action Request System; process history easily accessible

  4. Software development (June’00-Sep’04) • Common optical/IR image processing engine (> 90,000 lines of C-code) • System Wrapper (Python > 400,000 lines of code) • Science Applications (development & adaptation) • Software & Hardware Infrastructure • Over 25 man-years Medium-size project (by industrial standards) done by non-specialists

  5. Infrastructure • Python interfaces (e.g pcfitsio) • Tcl/Tk Graphic User Interface (GUI) • Graphic and image displays • Data Management: data model & database (Sybase) • Hardware architecture • Code versioning (CVS); Action Request System (Remedy) • XML/SVG technology (configuration; logs; WEB) • Web services • VNC, Condor

  6. ESO Data Flow System ESO Science Archive AR System (Remedy) System wrapper (Python) XML/SVG & WEB services GUIS (Tcl/tk) Image Processing Engine (C++) Search engine & DAL Internal DB tables (Sybase) EIS Integrated System: components

  7. Advanced Features • Search Engine: Simple/Extended • Scanner: Database/Directories • Batch/Workflows • Configuration: Tcl/Tk & XML interface • Process/Product logs • System e-mail alerts • QCtool/View Repository • Versioning/History • Tracker/Tracer • Import/Export • Product/Release

  8. Pipelines and workflows Image Processing Engine (C++) Photom Calibration Data Request Web System wrapper Database SE/DAL XML config Stacks Mosaics Science Plug-ins Catalogs) Data Release

  9. EIS survey system

  10. check header create RBs & filter bad images reduce images find, match & measure standards Find photometric solutions Calibrate images data from DVDs, ftp, uncompress Trigger image processing search for survey program-id update exposure, observed OBs, nights and runs Survey definition (strategy, regions, fields, filter, integration time) OB creation; finding charts Astrometric reference catalogs & all-sky catalogs, model predictions Access to database, repositories, on-line documentation … EIS survey system Front-end NEW ADMINISTRATION

  11. create SBs & filter bad images Validate SBs Stack or mosaic images QCtool & export NEW ADMINISTRATION Extract low S/N catalogs Prepare science grade catalogs Create color/mosaic catalogs Identify outliers & other targets by color properties Compute photometric redshifts, color-selection mass reconstruction; matched-filter cluster ident Move data to ESO science Archive Update WEB pages: index of release, request form Create entries in image gallery Send alert; statistic of requests (SVG) EIS survey system back-end

  12. Image processing engine (EIS/MVM) • High-throughput code memory bound (no R/W) • Extensively tested and validated (internal/external users) • Advanced features • Weight-map • Robust and efficient astrometry using wavelet decomposition (pixel-based, cross-correlation) • Gain Homogenization • Illumination correction from the data • Unbias small-scale background estimator (de-fringing maps) • Automatic masking of satellite tracks (Hough transform) http://www.eso.org/science/eis/publications.html (Vandame 2004, PhD thesis)

  13. Gain Homogenization

  14. Illumination correction

  15. Effect of corrections

  16. Calculation of Illumination Correction

  17. Satellite track masking

  18. Data rates • Observation data rates • SOFI = 0.02 Mpx/sec • WFI = 0.21 Mpx/sec • VST • 32/8 x WFI = 0.84 Mpx/sec (16 Tb/year; 43 Gb/night) • effective = 0.28 Mpx/sec (1/3 of night rate) • Survey = 0.08 Mpx/sec (25% of telescope time) • VISTA (IR) • 16 x 4 x SOFI = 1.28 Mpx/sec (23 Tb/year; 63 Gb/night) • effective = 0.43 Mpx/sec • Survey = 0.1 Mpx/sec • VISTA (optical) • 50/8 x 2 x WFI = 2.6 Mpx/sec (48 Tb/year; 130 Gb/night) • effective = 0.9 Mpx/sec • Survey = 0.2 Mpx/sec EIS reductions = 1.6 Mpx/s (8 CPUS)

  19. Database • Major re-redesign of database (> 150 tables) • sets • workflows & configurations • objects (object-relational) • Technical improvements • performance • uniqueness (checksum) • versioning • suitable interfaces for user-defined tables • extension of administration toolbox • Documentation • Master table • XML technology to describe column and objects • Database Design Reference document

  20. Web • EIS WEB pages: • New design with simpler navigation • More content • Automatic • Non-verbose • Database-based • New infrastructure to generate data release pages • README • Logs • JPEG images • Plots of dataset characteristics • Plots of vital data request statistics

  21. Data release infrastructure • Make products: nightly products, stacks/mosaics; other advanced data products; software • Make README • Make Release: • versions • associate a README • associated product logs (links to process and configuration) • check previous version log • notification • plots summarizing data set characteristic • JPEG B&W and color images • Distribution statistics

  22. Data Analysis Toolbox • Galactic model – (L. Girardi, Trieste) • Color catalogs by association (C. Benoist, Nice) • Counts-in-cells (C. Benoist, Nice) • Angular correlation function (C. Benoist, Nice) • Matched Filter (L. Olsen, Copenhagen) • Photometric Redshift (S. Arnouts, Marseille) • KSB and mass reconstruction (Bonn group) • χ2-fitting classification (EIS) • Outliers (EIS) • XGOBI and R (open source)

  23. Building blocks of operation • Initialize system (define survey; enter program id) • Scanner • Data Request • Main front-end workflow (process >> match std >> phot >> calibrate) • View Repository and Products (zero-point, stacks) • Main back-end workflow (stacks >> catalogs) • Data Release

  24. Examples of Workflows • Front-end: image reduction • Front-end: QCtool (3 min) • Back-end: image products (stack, mosaic) >> catalogs (5 min) • Back-end: QCtool stacks & catalogs • Back-end: QCtools mosaic • Back-end: color catalog • Administration Application

More Related