html5-img
1 / 23

EO GRID Processing on Demand ESA grid activity report on 2007

EO GRID Processing on Demand ESA grid activity report on 2007. luigi.fusco , guoqing li ESRIN, European Space Agency (ESA) Presented for WGISS-23, 2007-5-23. Summary. Brief introduction to ESA EO satellites EO community and GRID requirements EO GRID Processing on Demand (G-POD)

thuy
Download Presentation

EO GRID Processing on Demand ESA grid activity report on 2007

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EO GRID Processing on DemandESA grid activity report on 2007 luigi.fusco, guoqing li ESRIN, European Space Agency (ESA) Presented for WGISS-23, 2007-5-23

  2. Summary • Brief introduction to ESA EO satellites • EO community and GRID requirements • EO GRID Processing on Demand (G-POD) • Operational experience • Future vision and plans

  3. ENVISAT product’s examples

  4. Key international EO programmes The International Charter on Space and Major Disasters • Initiated by CNES and ESA, joined by CSA, NOAA, ISRO, CONAE, JAXA, USGS and DMC • Unified system of space data acquisition & delivery in case of natural or human-made disasters • Data delivery to civil protection agencies, emergency & rescue services; UN cooperating body since 2003 • Operational since 2000: 24 hrs on-duty-operator, data resources from all Charter members • Recent Examples of Charter Activation: • Bam Earthquake 2003 • Darfur Crisis 2004 • Tsunami Catastrophe 2004/2005 • Hurricane Katrina 2005 • Cyclone Kyrill 2007

  5. Key international EO programmes GLOBAL MONITORING FOR ENVIRONMENT AND SECURITY (GMES) European independence in critical data sources for environmental monitoring and security and the European contribution to the Global Earth Observation System of Systems (GEOSS)

  6. The Earth Science world Atmosphere Cryosphere Biosphere Ocean Coupled and inter-disciplinary processes Complex web of sensor Optimal Trajectory First guess Field Noisy observations Complex data analysis time

  7. Some Earth Science digital data requirements • Global, regional, local applications • Alternative use of the data at different resolution • Large historical distributed archives • Long term data and knowledge preservation issues • Near real-time access to data • For processing, value adding and dissemination • Integration with models to provide long term trends and forecast • Data assimilation • Integrate different data sources • Standardisation, Virtual Organisation, … • Need to link data to technical information and scientific results • Need to keep/transfer the historical Knowledge

  8. Why GRID at ESA for EO ? • Operational issues • Envisat generate > 500GBytes of data per day… • EO archives are “scattered” • Algorithm evolutions require recurrent reprocessing • More efficient use of existing infrastructures • Scientists / user • Need near-real-time access, historical and multi-sources • have the processing algorithms • require large investments to handle the data on their site • GRID can solve the equations… • move processors close to the data • reduce dissemination costs and effort • evolutions benefit to all at once GRID as a common shared platform for collaborations

  9. EO GRID Processing on Demand (G-POD) Objectives • Provide a “user-segment” environment • Put data & processors together • Allow “on-demand” processing of the data • Offer scientists a “production” lab • Focus on algorithms • Reuse housekeeping functions (e.g. catalogue, software tools) • Bridge gap from “prototype” to “production” processor • Offer scientists a “collaboration” environment • Share tools and functions • reuse output of other processors • IPR is kept by the scientist • Open and scalable • host “any” processor • bring more resources (other grids) and data (other archives) together

  10. Computing and Storage Elements Over 200 Working Nodes Over 100 TB on-line store Middleware: LCG 2.6, GLOBUS 4, gLite3 Link to external CE and SE (e.g. campus, EGEE…) Data Interfaces GS products Rolling Archives (ENVISAT, MSG) MODIS NRT products over Europe Access to AMS SatStore Some accesses to NASA and other external data providers Network Gbit LAN 64-192 Mbps to GARR HiSEEN WAN (e.g. to PACs/PDHS) MEGALAB (soon 20-40 Gbps in Frascati area) The ESA EO GRID environment • Security User certification via ESACERT underway • Software resources on-line • IDL, Matlab, BEAT, BEAM, BEST, CQFD, Compilers, public domain image processing utilities • Catalogue queries and data provision functions • Data viewers

  11. Grid on Demand Infrastructure ESA GS Operat’s User access tools Only one environment foroperations and new dev I/F to external GRID resources via research networks

  12. EO Data on-line in GRID storage • ASAR • ASAR GM: Feb 2004 onwards • ASAR Medium Resolution: June 2005 onwards • Occasional High Resolution products • MERIS • Level 1,2 RR: full mission • Samples of FR Level1 • AATSR • Level 1: full mission • Level 2: since mid 2005 • Atmospheric • GOME Level 1,2: 1996 onwards • GOMOS, MIPAS, SCIAMACHY full archives from D-PAC • Third-party data • MSG SEVIRI: last 12 months • MODIS NRT over Europe More to come …

  13. Web Portal Interface • Temporal/spatial selection of products • Job definition, submission and live status monitoring • Customisable result visualization interfaces • Access to output products and documentation

  14. Experience so far • Wide ranging applications supported • Systematic productions of high-level products (e.g. level-3) • On-demand Data access and visualisation • Algorithm and data validation activities • software test-beds (ESA toolboxes e.g. BEAT, BEAM, BEST) • Processor test-beds • Internal Research projects • Over 40 projects supported, more to come!

  15. Routine Production • MERIS Level-3 Products NRT generation • Joint ESA collaboration with ACRI (France),JRC/Ispra (European Commission) and Brockmann Consult (BEAM) • 11 monthly products published on-line http://earth.esa.int/meris/level3 • Daily ASAR GM mapping of Antarctica • Internal Development in operations since 2005 • Daily Generation of 400-m resolution mosaics and publish to ESA Web Map Server http://ssems1.esrin.esa.int:8080/map/context.php • EarthCare Simulator • Collaboration with Estec and Univ. of Valencia (Spain) • Preparatory activity of the mission EarthCARE • 2.5 105 simulations carried out using ESA and CNR GRIDs • River and Lake Processor • ESA/Montfort University (UK) collaboration • Accurate River and Lake heights measurements in NRT from satellite altimetry (RA2) • products published online http://earth.esa.int/riverandlake • Meris True-Color Mosaics • 9km resolution global Monthly mosaics of MERIS data

  16. On demand processing • ASAR on Demand • Integrated environment for SAR processing • Binds separate functionality into higher-level applications (flood monitoring, co-registration, mosaicing, etc) • Imager • Multi-mission imaging tools • Data selection, user-defined processing, image rendering, geo-coding, visualisation • MIRAVI Geo-toolbox Geocoding of MERIS full resolution images produced by MIRAVI real-time service • Aeromeris • Fast extraction over user-area of pixels and statistics from the complete MERIS level-2 product archive • Output to Excel, Google Earth, XML

  17. Algorithm Validation • Volcanoes Monitoring by Infrared • Fast Extraction of AATSR thermal anomalies over > 200 volcanoes • Under validation, Operational service planned for early 2007 • GRIMI-2 • EADS/Astrium MIPAS Level-2 reference processor used to validate the operational PDS processor • Test data generation: 300 profiles (100GB) processed in 1hour • YAGOP • Alternative GOMOS O3 and Temp level-2 processor (internal research) • Validation against operational ESA products • RAIES Processor • Assessment of the 2000 Hz individual echoes of Envisat/RA2

  18. New Research applications • MERIS/GOME inter-comparison • Validation of Envisat/MERIS cloud fraction and water vapour measurements against ERS-2 GOME • Var2Grid • ESA/DMI cooperation (DK) • 3DVar data assimilation (T and specific humid.) with CHAMP/GPS products (Background ECMWF ERA40) • SCIA-SODIUM • Internal prototype development • Extraction of Sodium profiles in mesosphere from Sciamachy • Collaboration with ESO for the next generation telescope ground site survey • ALOS PRISM DEM Prototype Processor • Internal Research (ESA Trainee) • Image co-registration of PRISM views and height reconstruction by image matching between PRISM triplet images Mountains around Shizuoka, Japan

  19. Call for Proposals http://eopi.esa.int/G-POD Promote the access and use of EO mission data available at ESA, offering on-line access to products with attached computing infrastructure and tools to assist the generation of “scientific added value products” new earth science algorithms Build network of competences; promote e-collaboration transition from science algorithms to new user products… First call published beginning June 2006 on EOPI web 11 Proposals selected in first call, implementation ongoing First very promising results expected mid 2007 Next call in few months Earth Science G-POD

  20. YAGOP: Yet Another GOMOS Processor • Validation period: 11 day ensemble of GOMOS data (Sep., Oct., Dec. 2002) • total set of selected occ. events GOMOS: 1719 MIPAS: 4099 ECMWF: 1719 GOMOS to MIPAS temperature profile comparison (relative errors shown in Fig.) Time – Space coincidences criteria for analysis: 3h&300Km GRID is very good for such analysis: 1 generic grid service

  21. near future GRID plans • Operational Opportunities • Routine generation of “standard” high level products • Extend infrastructure to other facilities and across establishments • Consider potential use of GRID • In future EO missions preparation • GMES Service Elements • Contribute to GEOSS architecture • As a mean to get the science community more involved in ESA programmes • Earth Science presence in European e-infrastructure initiatives • Strong participation to new EC FP7 initiatives such as e-infrastructure, digital repositories, knowledge preservation, e-content... Knowledge Infrastructures

  22. GRID based RTD plans CWE Applications DL Applications … … as Knowledge Infrastructure SpaceGrid PECS GRID Grid ES Community Applications ES Grid on Demand in operation @ ESA

  23. Thank you! The ESA EO GRID Team More information at http://eogrid.esrin.esa.int

More Related