Australian Virtual Observatory – 2003 Workshop

1. Gemini and the Virtual Observatory Warrick Couch, UNSW(aided and abetted by Colin Aspin, Gemini Obs) Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003

2. The Gemini Observatory Gemini North – Mauna Kea Gemini South – Cerro Pachon • twin 8.2m telescopes, 4-5 instruments per telescope, many different modes per instrument. Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003

3. The Gemini Observatory Gemini North – Mauna Kea Gemini South – Cerro Pachon • 6 partner countries, 1 host country (Chile), 1 host institution (UHawaii) broad and diverse set of scientific users & programs. • programs (currently) involve ‘PI’ rather than ‘survey’ science. • majority (80-90%) of observations done in ‘queue’ mode science data must come with good calibration & site weather data… but greatly facilitates a good archive! • standard proprietary period = 18 months • all new instruments undergo ‘science verification’ data available in 2-3 months! Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003

4. The road to a Gemini Science Archive (GSA): • Apr 1998 : GSC/Board endorse GSA resolution. • Sep 1998 : Gemini sponsored GSA workshop and budget line approved by Board/Director. • Nov 2000 : Functionaland Performance Requirements Document and Operational Concept Definition Document issued by Gemini. • Sep 2001 : Colin Aspin became GSA Project Scientist. • Nov 2001 : Phase II Operationsand Development Plan review at CADC. • May 2002 : Concept Design Document issued by CADC. • Jan 2003 : Content Overview Document and Meta-Data DataBase Definition Document issued by Gemini. • Feb 2003 : Phase II ODP contract signed! 10 November 2003: the Gemini Science Archive prototype (GASp) goes public!! Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003

5. The Basic Archive/Prototype(http://cadc.hia.arc.ca/gemini) • Basic retrievalof raw science & calibration data to (registered) users after proprietary period. • Web-based search engineat CADC with numerous options for retrieval. Same functionality as CFHT, JCMT, etc archives. • All ingested datashown in ‘results page’ but only non-proprietary data available for download. • FITS headers:only meta-data in prototype. • Data retrievedvia e-transfer on extraction and packaging. • Operational:data ingest of GN (facility instrument) data up to date, of GS data (visitor instruments) – 90% complete. GMOS-S ingestion under way. Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003

6. Transferal of Data to Archive • Current Status : data from GN and GS are packaged on DVD-R & shipped each 2-4 weeks. • Data Ingestionis therefore delayed by packaging and mailing of hard media. • Planis to setup e-transfer of data from GN and GS to CADC. VPN connection to be used for security. Staging areas at GN and GS will be monitored by CADC and when new data is found, e-transfer, validation and ingestion takes place (by end 2003). • Current I2 networkgives ~150KB/sec/connection. Tested up to 6 simultaneous connections -> 900KB/sec. (typical GMOS image is ~50MB). Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003

7. Data Distribution to PIs • Current Status : data from GN and GS are packaged on CD-R or DAT and shipped directly to PI. • Planis to have PI obtain data from Queue or Classical run via CADC GSA interface via e-transfer. • Ifhard-mediais required by PI then CADC would make media from archive data and ship. • Requires special/secure accessfor PIs to proprietary data (their own). • Requires accessto image data, calibration data and meta-data. • Timescale : mid-2004. Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003

8. The fully capable GSA: • Addition of complete meta-dataaccess using MDDB. This includes ‘results page’ display of meta-data. • Ingestion of results of OLDPsystem as on-line preview. • Associationsof datasets (e.g. J,H,K) • Associationsof calibrations datasets. • Improved user-interface(s). • Batch user-interface (e-mail). • Networkdata transfer to CADC : PI distribution from CADC. Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003

9. The Meta-Data Data Base (MDDB) • MDDBresides at CADC. • Ingest tools(provided by CADC) run at GN/GS for ingesting meta-data into MDDB. • Meta-data : • augmented FITS headers, • environmental & weather data including images (via Gemini weather Server), • observing proposals/programs/sequences, • observing logs, • publication lists (continually updated?) Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003

10. Data Integrity/Security • Geminimakes and keeps ‘local’ copies of all data on-site. • Data sent/e-transferredto CADC and ingested. • CADCimplementing off-site backup at UVic for science, calibration and meta- data. • All datae-transfers done through VPN, encrypted. • Checksum integritycheck planned at all sites (summits, base facilities, CADC, UVic). • Likely setup effective ‘mirror’of GSA at GS for performance improvement/security. Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003

11. Advanced Capabilities(to be further discussed/finalized) • On-line, on-the-fly, data processingusing Gemini iraf package for (re-)generation of optimally reduced data. • Observations Tables: archive-independent tables of scientifically interesting data supersets. Allows cross archive searches. (NGC1333 x-ray to sub-mm) • Source/Object Tables: archive-independent tables of sources extracted from Gemini data and other archives. (SVS13 fluxes 0.5um to 1mm). • GUI improvementsto support above and other developments. Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003

12. GSA within the VO World • Overall Goal: to produce Gemini ‘data products’ which are VO-compliant. • Data products:conform with the VO concept that they are independent of the telescope/instrument data taken with, and conform with some (yet to be?) agreed VO standard for packaging data + metadata + anything else, so that it can be read by any VO system. Natural extension of GSA and advanced capabilities! Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003

13. Documentation • Geminiweb-site: http:/www.gemini.edu/sciops/data/dataArchive.html • Prototypecookbookin preparation. • Support for GSAthrough Gemini HelpDesk • Tier 1 : National Gemini Offices • Tier 2 : CADC • Tier 3 : Gemini (Colin Aspin) Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003

14. Possible Aus-VO involvement • Biggest Bottle-neckto generating scientific results from Gemini observations is in the data reduction. (IRAF-based reduction scripts are available but have proven to be user-unfriendly and unreliable.) • Problem recognised by Gemini and more resources now being devoted to developing “fully processed” data pipelines; however, more FTEs needed and welcomed!! • Channel some Aus-VO resources into developing pipelines for instrument modes popular with Aus Gemini users?? Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003

15. Two points to take away….. • GSA  increased Aus access to Gemini through making ALL science observations taken across the partnership available (remembering that HST archive has increased its usefuless by ~ x3!!) • Gemini must be a success story for Aus only 8m telescope facility we have a direct share in; it is not cheap (US$1/sec); need to MAXIMISE its use (science time + GSA) and scientific outcomes (papers, publicity etc)! Australian Virtual Observatory – 2003 Workshop ATNF, Marsfield, 17-18 Nov 2003