Image Information Mining for Earth Observation - ESA – ESRIN perspective

1. Image Information Miningfor Earth Observation-ESA – ESRIN perspective Sergio D’Elia Head of Research and Technology Development Section EO Ground Segment Department – EOP Directorate

2. Table of Contents • ESA activities in IIM • Wider perspective • Conclusions

3. EO Data Exploitation Chain Distributors Value Adders Data Providers Service Providers Support / Automation possible? Knowledge EO Data / Products Information / Service Archives (PBytes) Users (KBytes) Non-EO Data

4. IIM Research Pattern Probabilistic IM Primitive Features (automatic extraction and multi-dimensional clustering) Learning Systems / Training Probabilistic Search Ingestion Automation Semantic Aggregation Multi Data-type Handling Workflow Fast Databases, Scalability System / Interface Classification, Ontology KD, Mediation (basic) Prototypes Co- / absolute- registration Survey of Algorithms Space-time (geometry, time series) Further Research Impact Crater Discovery Data / Information Fusion Feature Extraction KD, Mediation

5. ESA Projects in IIM 1650 Image Information Mining – Time Series IIM-TS 525 UGEI Universal Geometry Engine for EO Images (GSTP) Registration 150 Universal Geometry Engine for EO Images - Preparation UGEIP 150 Multiple Image Registration Tools MIR 1300 Knowledge Centred Earth Observation KEO 220 MERIS Information Mining Service (EOP) MIMS IIM Services 330 KES-B Knowledge Enabled Services 200 Impact Crater Discovery (GSP) ICDY Feature Extraction Survey / classification of advanced features manipulation techniques and tools for EO 280 SURF 50 Crater Survey of algorithms for automatic recognition of impact craters (GSP) 490 KIM Validation (EOP) KIMV Probabilistic IM 300 KES Knowledge Enabled Services 200 KIM Knowledge driven information mining in remote sensing image archives 2001 2002 2003 2004 2005 2006 2007 K€

6. ESA Investments Summary • ESA has invested / booked so far 5.3 M€, from • GSP 250 K€ • TRP 4360 K€ • EOP 710 K€ • Additional 525 K€ are available from GSTP • Limited EOP funds might be available (within Oxygen) for installation of results Which results ? When ?

7. Results Expected from IIM Probabilistic IM Primitive Features (automatic extraction and multi-dimensional clustering) Learning Systems / Training KIMV Probabilistic Search Information Discovery CB Image Selection Ingestion Automation Semantic Aggregation KES Multi Data-type Handling Cooperation Environment Workflow Information Discovery CB Image Selection Feature Access (basic) Thematic Maps KEO Fast Databases, Scalability IIM Services for archives: System / Interface Classification, Ontology KD, Mediation (basic) Prototypes Co- / absolute- registration Survey of Algorithms Space-time (geometry, time series) Further Research Impact Crater Discovery Data / Information Fusion Feature Extraction KD, Mediation

8. Results expected this year • Probabilistic IM being validated on small archives for • Interactive Information Discovery (search via training) • Content Based Image Selection • CB Image Selection • Implemented for MERIS, to select images satisfying • Normal parameters (mission, sensor, …) • Spatio – temporal constrains • With cloudiness < threshold over user AOI • In principle can be extended to other features in MERIS images (e.g.: algae blooms, fires, etc.) just via additional system training (no need to re-ingest images)

9. MERIS CB Image Selection 1 MERIS level 1 Reduced Resolution products are loaded in the system - 2 KimV - + + an expert trains the system via positive and negative examples to identify clouds

10. MERIS CB Image Selection (cont.) 3 4 the system identifies clouds in all loaded images and creates related binary maps

11. MERIS CB Image Selection (cont.) individual clouds are isolated, loaded into the database and associated to the term “cloud” 5 N W E S MAP_object:{ label=”cloud 33” center = (223,332) border = (NSESWSWSWNNW) } Semantic / spatial queries for clouds can now be performed 6 Due to the very large number of cloud objects, special methods are being implemented for clouds to improve performance

12. Table of Contents • ESA activities in IIM • Wider perspective • Conclusions

13. EO Data Exploitation Chain Distributors Value Adders Data Providers Service Providers Support / Automation possible? Knowledge EO Data / Products Information / Service Archives (PBytes) Users (KBytes) Non-EO Data

14. Oxygen Objectives Oxygen is a concept(might become a programme) aimed at • Facilitating access to EO data from ESA and other missions • Increasing sustainability of EO data provision widening the range of offered data sources Oxygen has • Consolidated the historical EOP-G multi-mission approach • Extended it also to Partners (non-ESA missions and GS)

15. Oxygen Implementation Plan (2003 – 2005) • S-pillar: Developing Services • Development of Prototype Services (including Image Information Mining Services) • Creation of a new Service Support Environmentfor easing provision of Services by Service Providers

16. Oxygen Ground Segment Model Forthcoming components Quality Control User Services Help Desk Web Portal Mission Planning Catalogue Data Ordering CO-ORDINATION CENTRE SERVICES INFRASTRUCTURE Network Monitoring & Control Information Mining Services Acquisition Archive Dissemination Production FACILITY

17. Services Infrastructure - Background Data Environment User Environment Mission Planning Acquisition Order Desk Catalogue Archiving Help Desk Processing Ordering Dissemination Quality Control Service Support Simple Catalogue Service IIM Support Service Environment • Some TRP / GSTP projects are providing results, of wide interest also for Oxygen partners, which create a new“Service Environment”dimension: • Encompassing ESA &non-ESA resources • Providing a bridge towards non-EO domains • Satisfying identifiedrequirements, like: • Service Support • Simple Catalogue Service • Image Information Mining Support

18. Towards “Services Infrastructure” • Currently available “Service Support System Prototype” • Derived from MASS project (GSTP, 2.4 M€, B and I, 3 years) • For on-/off- line Services, at Service Providers’ sites • For Service publication, activation, monitoring, chaining • Becoming “Service Support Environment” by Oct. 2004 • With over 20 Prototype Services • “Neutrally managed” by ESA (requested by Service Providers) • Potentially relevant as model for GMES, TESI, INSPIRE, … • Will evolve in 2007 to “European Service Infrastructure” • Enhanced for OGC, automation, e-business, e-collaboration • Responding to new requirements (e.g.: GMES, …)

19. Prototype Architecture RFQ Portal User Order Directories Workflows Service A - Service A • Service B • Service C • … Status A1 A2 Results’ info On-line / Off-line Results A3 Internet Toolbox Toolbox Service Provider A1 Service Provider A2 Service Provider A3 On-line / Off-line Results • Handles control information • Implements service chaining

20. Protocols and Interfaces Client Web Mapping (OpenGIS) ICD (Interface Control Document) HTTP Portal Directory WSI Descriptions WSDL / XSD Workflow BPEL (OASIS) Service Provider Legacy System Toolbox SOAP / XML HTTP (80) TCP/IP ODBC, OS, FTP, SMTP, Scripts Service Providers can publish Services from their legacy system as Web Services (W3C) using • WSDL over SOAP (implementing the ICD) • The ESA TOOLBOX (provided / maintained free-of-charge), which also converts local terminology to common ICD terms

21. Workflow Example

22. Service Chaining Example User need: NPP* (service) over Belgium (AOI) in GeoTIFF (format) MASS Portal Workflow: Net Primary Productivity S1 S2 S3 NPP Processor(VITO) Format Conversion (GIM) Clipping Service (GIM) NPP EuropeHDF NPP EuropeGeoTIFF NPP BelgiumGeoTIFF Spot Végétation S10 (*) NPP or Net Primary Productivity represents repeatable estimates of the net flux of carbon.

23. Other System Functions Result: ftp://sp1.com/layer1 [click to view] Oil spill (SP1) Oil rigs (WMS1) Layer3 (WMS1) User Layer4 (WMS2) Layer5 (WMS3) WMS1 AOI Client browser WMS Viewer Portal WMS2 Internet ESRIN OpenGIS WMS AOI Server WMS3 Service Provider 1 Service Provider 2

24. Service Support Environment Contract Partners Comments • Upgrades • Services’ consolidation • Portals and rules MASS-Env KO: Nov 03; End: 1Q 05 Spacebel, GIM (B), Intecs (I) MASS-Ser-D KO: Nov 03; End: 1Q 05 Telespazio, Planetek (I), VISTA (D), ARGOSS (NL), GIM, Vito (B) • 6 new services MASS-Ser-O KO: 1Q 04; End: 1Q 05 QinetiQ, CCLRC, RSI, Comsine, NEODC, NPA – (all UK) • 11 new services • UK Regional portal ? MASS-Ser-O2 KO: 1Q 04; End: 1Q 05 Gael, GSC (F), Planetek (I), Phenix (UK), Joanneum (A), RapidEye (D) • 8 new services MIMS KO: 2Q 04; End: 4Q 07 • Installation, loading and maintenance for 3 years of MERIS IM Service (TBD) SSEM KO: 4Q 04; End: 4Q 07 • SSE Manag. for 3 years (TBD) ISAC, GEO (GSTP-4) KO: 1Q 04; End: 4Q 05 • Dedicated Client • GRID Support Spacebel (B) + TBD for GEO

25. Simple Catalogue Service • Since • The system was already permitting access to limited Service Providers’ catalogues (via AOI, time range, …) • Existing large image catalogues, if chainable, could be of interest for some services • IIMCG requested parallel access to Envisat /Spot catalogues • An attempt was made to • Access large EO catalogues as simple, chainable services, via a few parameters but with adequate performance • Demonstrate the function for Envisat and Spot

26. Simple Catalogue Service AOI Selection User Portal AOI Client browser Internet MUIS or other Catalogues Service Provider Catalogue Provider Catalogue Provider Catalogue Provider Envelop Catalogue Access as simple Service Catalogue Access can be chained with other Services • Parallel activation • Results’ recombination

27. USER AREA SPOT ENVISAT

28. Table of Contents • ESA activities in IIM • Wider perspective • Conclusions

29. “Coming soon” • Service Support Environment(SSE) by end 2004 with a core of Services • Based on standards / widely used protocols and only 2 commands (RFQ and Order) • New Service Providers can join on request, with the following benefits: • NO NEED TO MOVE services (full CONTROL and EXPLOITATION of existing infrastructure) • Can achieve SYNERGIES with other partners and reduce COSTS and TIME TO MARKET • Simple Catalogue Service (tested between Envisat and Spot) • Obtained by enveloping existing catalogues as simple Services with translation of terms • Based on two additional commands (Search, Present) for better performance • Catalogue access can be chained with other Services under Workflow control • Interactive Information Discovery Tool • Learns and applies training results to identify features / objects in ingested images • Remotely accessible for mining specific image sets ingested for the purpose (TBD) • Content Based Image Selection • Being tested for MERIS (search images with cloud cover < threshold within user area of interest) • Chainable with other SSE Services • Relevant for image catalogues (Archives, Distributors, Service Providers, …)

30. Useful Links • SSE Introduction Workshop being organised • Location / date: ESRIN / April 22, 2004 • Web site (in preparation)http://earth.esa.int/rtd/Events/SSE_2004 • Details on ESA IIM and RTD projects at • http://earth.esa.int/rtd • Details on IIMCG at site (in preparation) • http://earth.esa.int/rtd/IIMCG

31. Services Infrastructure - from … Data Environment Data Environment Data Environment End User End User Service Provider Service Provider Distri- butor Value Adder Value Adder Value Adder User Envir. User Envir. Non EO Data

32. Services Infrastructure - … To Data Environment Data Environment Data Environment End User End User IIM IIM IIM IIM IIM IIM Services Infrastructure Service Provider Distri- butor Service Provider Value Adder Value Adder Value Adder User Envir. User Envir. Non EO Data Including IIM Services