1 / 8

DMD Data Flow Operations David Silva Data Management and Operations Division

DMD Data Flow Operations David Silva Data Management and Operations Division. Data ingestion & delivery Chile sites, WFCAM, surveys Data quality control and processing (Paranal) Archive/database content management Data Interface Control Board (DICB) chairman, members

muriel
Download Presentation

DMD Data Flow Operations David Silva Data Management and Operations Division

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DMD Data Flow OperationsDavid SilvaData Management and Operations Division

  2. Data ingestion & delivery Chile sites, WFCAM, surveys Data quality control and processing (Paranal) Archive/database content management Data Interface Control Board (DICB) chairman, members Archive systems administration Database administration HST and VO support Remedy and Web administration Mission: ESO Archive Operations http://archive.eso.org http://www.eso.org/qc

  3. DFO Personnel 14 ISM, 11 CFTE Missing: Mignani (QC),Percheron (QC), Sjøberg

  4. Department re-organization Good-bye and thanks to Benoît Pirenne! Merged OTS and DFO/QC Absorbed Web/Remedy administration Transition to Service Contract Before: many 1-on-1 contracts Now: Data Conversion Services Duration: 3/2004 - 3/2007 (option: 3/ 2009) One year summary: going well New services ESO library database interface Proposal abstracts on-line Archive query forms for [FILL IN] Data Interface Control Document 3.0 QC services for MIDI Selected numbers More than 100 hardware servers Over 8 hours of Venus Transit Web event, 1.5 TB and 45 million web hits served Ops_log database warehouse reached 1 billion rows in March 2004 Archived ~1 million files, approximately 1/3 of all files in Archive Over 150 000 QC processing jobs (VIMOS = 54 000), 5 Tbytes processed, 1364 DVDs delivered 2004 Highlights

  5. Archive opening to world Per endorsement by Council Monday 4 April 2005 Improved Archive interface(s) (see VOS presentation) WFCAM data ingestion UK in-kind, raw data, 6 TB/year New systems Database server migration New hardware, Sybase ASE 12.5.3 Web servers to Linux New compute cluster New DFO/QC tools Net-based data transfer (again?) 2005 Outlook • New QC Services • Apr: SINFONI, VISIR • Oct: AMBER • Improved science products • Expand to Visitor Mode (?) • New Archive services • Support for VO • Collaborate with VOS department • Better ToO/RRM support • Better SCI/CAL association • Ingest & serve science products • Prepare for VST/OmegaCam • Date volume/processing accelerandobegins…

  6. Data Volume Challenge HST: 0.2 TB/year VSTCam: 30 TB/year VISTA: 150 TB/year ALMA: 180 TB/year

  7. Drivers: volume, complexity, demand More pixels = more FLOPS Example: Cam, VISTA More processing per pixel = more FLOPS Example: VIMOS, SINFONI More VO re-processing = more FLOPS Can be: volume and complexity In terms of required processing power: SINFONI:MUSE = 24 Data Processing Challenge FLOPS = floating-point operations per second • Required DFO processing power • UVES: 1.0 (actual, inc. fibre) • VIMOS: 2.2 (actual) • All VLT/VLTI (2004): ~ 6 • Cam: ~ 21 • VISTA: ~ 49 • Based on Wicenec, Hanuschik, Knudstrup, & Dolensky, 2005) • Related issues • Peak loads • VO demand • 2nd generation VLT/VLTI instruments

  8. ESO Data Center: 2010 Floor space: ~100 m2 Storage volume: 1.2 PetaByte (one copy)Inflow: ~0.25 PB/year Mix of slow-access and fast-access media Need 2nd secure copy In addition: database server(s), etc. 100+ processors: 2006: 100 UVES equivalent 2010: 1000 UVES equivalent

More Related