1 / 16

The EIS Experience: Lessons Learned

The EIS Experience: Lessons Learned. May 12, 2005. Background. EIS 1996-2004 Original mission: Limited-term experimental project Preparation of data sets for VLT commissioning and early operation Coordination with ESO community (SWG, visitors) Train & disseminate tools as possible

dulcea
Download Presentation

The EIS Experience: Lessons Learned

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The EIS Experience: Lessons Learned May 12, 2005

  2. Background • EIS 1996-2004 • Original mission: • Limited-term experimental project • Preparation of data sets for VLT commissioning and early operation • Coordination with ESO community (SWG, visitors) • Train & disseminate tools as possible • In 1999 mission expanded • Support long-term public surveys (1997-present; 8 years) • Develop infrastructure for public surveys • Develop image processing engine • Develop survey software system • Ramp-up to VST and VISTA Mission: R&D & operation; science & software; but ...

  3. Community Participation • OPC (approval) • Survey Working Group (design, oversight) • 29 participants (2 committees) • 24 institutes • 7 member states • Visitor Program (operation/development) • 43 participants • 15 institutes • 7 member states Full & broad community involvement in every aspect of the project

  4. EIS Team responsibilities: • Observations (preparation/observing) • Software Development: • Infrastructure (GUIs, database, architecture) • Image processing engine • Scientific algorithms • System integration & tests • Hardware procurement • Preparation, verification & delivery of survey products • WEB development & maintenance • Publications & reports • Recruitment of visitors & administration of intense visitor program • Composition: • 5-6 FTE/year (astronomers & software developers)

  5. EIS Surveys

  6. Summary of surveys(07/97- ) • 9 surveys in < 6 years (FORS, FLAMES & VIMOS) • 13 strategies • 3 Telescopes (NTT, VLT, 2.2m) • 5 imagers (EMMI, SUSI2, SOFI, ISAAC, WFI) • 22 filters • 240 nights in visitor mode alone done by the team • 73,000 frames of raw data; 37,000 science exposures • WFI 7,000 science; 23,000 total • SOFI 18,200 science; 31,000 total • ISAAC 11,800 science; 21,000 total • 27 public releases (several data products, software, zeropoints) • Observations still ongoing (SOFI 09.04; WFI: 02.05)

  7. Phase-1 (03.97-10.99) • Best-effort • Simple image reduction pipeline using • Available packages (IRAF, Eclipse) • Tools (Drizzle, LDAC, SExtractor) • Wrapper: shell scripts • Limited software development (adaptations, bug-fixing, small-scale new developments) • Help from experts in-house and in the community

  8. Limitations of best-effort approach: • Problems with the data: • Calibration (loss of flux in 1998 release associated to Jitter) • Serious problems with the astrometric calibration of WFI • Rapid increase ( > 6 x) in data volume in 1999 • Reductions mostly manual; no history • Over-reliance on a few people making operation vulnerable & fatigue • Departure of key team members => 6 months interruption in reductions & development • Different environments for optical and infrared • Unsuitable hardware & software

  9. April ‘99 April ‘02 April ‘04 SOFTWARE Phase-2 Phase-1 EIS Data volume: time evolution Project split into 2 phases

  10. Lessons learned • Best-effort approach one-off (not 24 times in 6 years) • unsustainable in the long-term • error-prone, hard to recover • disruptive, leaves no legacy • impedes progress of development (developers become operators) • Need of framework and stable core group to preserve know-how and inherit code • Resource-limited operation requires large degree of automation • Handling large amount of data/information big challenge in survey context (differs from data-in/data-out problem) Requirement: 1) develop new image processing code 2) develop integrated reduction system

  11. EIS data reduction system (06.00-09.04) • Procedures standardized and accessible via GUIs • Access to data/information transparent to user (SE/DAL) • Integrated environment (CVS, ARS, Database, Web) • Common optical/IR image processing engine (90,000 lines of C-code) • System Wrapper (Python > 400,000 lines of code) • XML technology (configuration; logs; Web; database contents) • Self-describing products with quality parameters • Uniqueness, versioning and history of products Medium-size project (by industrial standards) done by non-specialists

  12. Releases • Maximum interval between end of survey and release < 3 years • Products • Night (XMM, DPS, PF) • Stacks • Mosaics • Source catalogs (SExtractor, DAOPHOT) • Catalogs of clusters of galaxies, quasars, low-mass stars • WFI zero-points for 150 nights over 5 years • WFI data covering 29 square degrees • Highlights 2004 • Release of EIS/MVM code • 9 releases in 4 months; 11 releases in 2004 (1 release/month) • Data releases with product logs and READMES • 7000 ISAAC frames; 3000 WFI frames

  13. Raw data Requests: 1367 Products: 84589 Volume: 6.1Tb Survey products Requests: 672 Products: 9292 Volume: 0.93 Tb Software: ~ 62 users Total of last 46 months Requests: 2039 (44/mo) Products: 93851 (2040/mo) Volume: 7 Tb Over project lifetime 27 releases 40 requests/mo Products: > 100,000 12,500 prod/year; 34 prod/day Data Request Statistics

  14. Project Legacy • High-performance, instrument-independent image processing pipeline (tested for all ESO imagers; publicly available) • Integrated, end-to-end data reduction system to monitor & reduce multiple surveys from a single desktop • Survey infra-structure (WG, database, Web interface, release) • Blueprint for a modern data reduction & analysis system • System easily adaptable for different applications

  15. Summary • Over 25 man-years of development • System supports • optical/infrared, single/multi-chip instruments • configurable workflows for un-supervised operations • System consists of several pipelines • Image processing • Photometric calibration • Stack & Mosaics • Catalog (DAOPHOT, SExtractor) • Science applications (plug-ins) • Mature, extensively-tested system before UKIDSS, VST and VISTA commissioning • Work in progress EIS Data Reduction System

More Related