ASTRO-WISE

1. ASTRO-WISE An Astronomical Wide-field Imaging System for Europe Konrad Kuijken, Edwin Valentijn Kapteyn Institute, Groningen Towards an International VO - Garching bei Munchen

2. VST + OmegaCAM 6x WFI 1o x 1o 16k x 16k pixels Towards an International VO - Garching bei Munchen

3. The Problem • Many wide-field imaging instruments, vast amounts of data • E.g.: VST = Southern sky (30 min exp, 300 nights/y) in 3 years. Vast amount of data! 100 Tbyte • Science on these data should be archive-based • Handling of the data is non-trivial • Reduction • Calibration and re-calibration • Image comparisons and combinations • Working with source lists • Visualization • ASTRO-WISE aim: • provide a system for maximal data handling flexibility Towards an International VO - Garching bei Munchen

4. Concept solution • Environment which • Has access to all raw and calibration data • Can run and modify reduction/calibration pipelines • Can run source extraction algorithms • Archives reduced data and source lists, or regenerates these dynamically • Can be federated to link different data centers • This dynamical archive continuously grows as more raw data enter the system, and can be used for • `small’ or large science projects • generating and checking calibration data • exchanging methods and scripts • A key functionality is the link back from source data to the original raw pixel data and calibration files Towards an International VO - Garching bei Munchen

5. Example uses • Deep multi-colour fields • No need to take all data in one campaign • Combine data of particular quality, assess results • Select sources, visualize interesting ones, … • 1-in-1,000,000 events spurious or not? • Large homogeneous surveys • Eg weak lensing maps, cluster searches, star counts • Variability (source list- or pixel based) • Proper motions (asteroids, nearby stars) • Flux variations Towards an International VO - Garching bei Munchen

6. Example uses • Monitor instrument (calibration files) • Planning observations • View quality of existing data • Build on what already exists, add more filters, more exposure time, better seeing, … • All this can be done on the continuously growing archive Towards an International VO - Garching bei Munchen

7. Philosophy • System is not geared to a single data product, but to being a flexible tool. • Not focused on making a single, homogeneous survey with static data release(s) - but could be used for this • Optimise interaction between users and their data  Dynamic archive • Geared to optical (IR) wide-field image data Towards an International VO - Garching bei Munchen

8. Partners • Distribute the load/environment over National centers, each serving their community • Partners: • NL - Groningen (+ Leiden) • ESO – EIS • I – Napoli + • F - Terapix • D – USM + • UK - VISTA Edinburgh/Cambridge • 1.5 MEuro grant from EU RTD programme Towards an International VO - Garching bei Munchen

9. How? • Procedurizing data taking at telescope for both science and calibration data • Full integration with data reduction OmegaCAM FDR • Data model (classes) defined for data reduction and calibration • Design • View pipeline as an administrative problem Towards an International VO - Garching bei Munchen

10. Towards an International VO - Garching bei Munchen

11. Towards an International VO - Garching bei Munchen

12. Tbyte source lists- brains make the associations Link -lists Towards an International VO - Garching bei Munchen as fast as possible

13. Share the load • Hardware: Beowulf processors – 32 (most cases) • Hardware: multi Terabyte disks (10 – 100) • Data reduction: derive calibration • Data reduction: run image pipeline (1 Mpx/s) • Archive: storage of images (100’s Tbyte) • Archive: storage of source parameters (1-10 Tbyte) Network speed free parameter: - from 5 Mb/s (24 hours/day provides full replication of all OmegaCAM data) - to 200 Mb/s no replication Towards an International VO - Garching bei Munchen

14. Federate • Calibration results • Methods (pipelines) for processing calibration • Raw data • Reduced images • Source lists - catalogues Towards an International VO - Garching bei Munchen

15. Concepts • Federation maintained by asingle db • Full history tracking of all input that went into result providing on-the fly reprocessing – tell me everything tools • Dynamical archive - Context areas in object attributes • Project: eg Cal, Science, Survey, Personal • Owner: eg pipeline, developer, user • Strategy: Standard, Deep, Freq (monitoring) • Mode: Stare, Jitter, Dither • Time: time stamping • Standards • classes/data model/ procedures • 00 – inheritance/ persistency • Python scripts/ libraries Partitions Link to AVO VO table Towards an International VO - Garching bei Munchen

16. DB – engine Oracle contract: – reference – licenses - consultancy Oracle 9i: • federation through “Advanced replication” to evolve into cross site links (=references= pointers) • Python I/F SQL – OCI- Oracle db • Full oo support + Python to db persistency • Terabyte scalability through “partitioned tables” • Administration tools • Interoperability: STREAMS connects to others: SyBase, MySQL Towards an International VO - Garching bei Munchen

17. Intra-operability peer to peer • CVS • “Advanced Replication” evolving to pointers Towards an International VO - Garching bei Munchen

18. WP s • WP 1 – pipelines / same as ESO-DMD • Implement and operate • WP 2 – visualization tools • Panorapix – fast Gbyte image display + true colour • Graphic cards – OpenGL - • WP3 – federated db’s • CVS • Oracle • WP4 – processing power 4 Beowulf centers • WP5 – direct access 100 Tbyte storage • WP6 – coordinate – AVO – CERN-HEP- Datagrid ASTRO-GRID Towards an International VO - Garching bei Munchen

19. Schedule • ADD – review Q2 2002- OK • Basic operations – Q4 2003 • Evaluate and prepare for mass production 2004 • Qualify and populate 2005 • Deliver survey system – satellites • Links to LOFAR, JIVE,WHIRCAM, GAIA Personal Observatory Towards an International VO - Garching bei Munchen