1 / 37

Web Processing Service

Web Processing Service. Models for scientific exploitation of EO Data , Session 4: State of the art technology and standards Yves Coene, Spacebel s.a. Patrick Jacques, Spacebel s.a. Peter Baumann, Jacobs University Frascati - 12 October 2012. Overview. Introduction

trina
Download Presentation

Web Processing Service

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Web Processing Service Models for scientific exploitation of EO Data, Session 4: State of the art technology and standards Yves Coene, Spacebel s.a. Patrick Jacques, Spacebel s.a. Peter Baumann, Jacobs University Frascati - 12 October 2012 Session 4: State of the art technology and standards, 12/10/2012

  2. Overview • Introduction • Web Processing Service • Case Studies • Decision Support (DREAM) • Lunar Landing (LandSAfe) • Processing On Demand (G-POD, SSEGrid, ngEO) • Open-source implementations • Issues and future work • Standardisation work • Ongoing • Planned • Conclusion Session 4: State of the art technology and standards, 12/10/2012

  3. Introduction • HMA Standardisation • Service Oriented Architectures • For GSCDA and Science • Service Taxonomy (HMA) • Identity Management Services • Discovery Services • Invoke Services • Feasibility Analysis • Ordering • Geospatial Processing: §3.4.7, §5.5.5 • Online Data Access Services Session 4: State of the art technology and standards, 12/10/2012

  4. Web Processing Service • WPS is an OGC standard for: • describing and executing a geoprocessing operation via a service, • standardizing the requests and responses for a particular geospatial processing service. • WPS is a generic model of a Web service supporting WSDL and REST, for which profiles need to be developed and standardised to support interoperability: • Syntax and semantics of input parameters • Input and output data format Session 4: State of the art technology and standards, 12/10/2012

  5. Web Processing Service • Operations: • GetCapabilities • Discover which processes are offered by the service* • DescribeProcess • Discover input/output of a specific process • Response can be used to build the user interface for capturing the input parameters**. • Execute • Execute a process • Storing response or returning response document (*)e.g. ProcessCoverages is a process that accepts WCPS request language as per OGC 09-045. (**) similar to DescribeTasking in OGC 10-135 Earth Observation Satellite Tasking Extension of SPS. Session 4: State of the art technology and standards, 12/10/2012

  6. Web Coverage Processing Service • The Web Coverage Processing Service (WCPS) specifies an expression language allowing for server-side filtering and processing of multi-dimensional raster coverages. • It defines syntax and semantics of expressions for specifying extraction and analysis of coverages containing sensor, image, and statistics data. • Not tied to a particular transmission protocol, the query paradigm can be embedded into any service framework, such as: • OGC Web Coverage Service (WCS) • OGC Web Processing Service (WPS). Session 4: State of the art technology and standards, 12/10/2012

  7. Web Coverage Processing Service Session 4: State of the art technology and standards, 12/10/2012 • Use Cases: • On-demand retrieval • “thresholded NDVI where threshold value is obtained from user input” • Ad-hoc queries • Processing • “Fourier transform of scene X” • Filtering • “Scenes with less than x% cloud coverage” • Summary data • Sensor fusion and pattern mining Slide 7

  8. WCPS By Example for $c in ( M1, M2, M3 )return encode( abs( $c.red - $c.nir ), "hdf“ ) (hdfA, hdfB, hdfC) 8 Session 4: State of the art technology and standards, 12/10/2012 "From MODIS scenes M1, M2, and M3, the absolute of the difference between red and nir, in HDF-EOS"

  9. WCPS By Example for $c in ( M1, M2, M3 )where some( $c.nir > 127 )return encode(abs( $c.red - $c.nir ), "hdf“ ) (hdfA, hdfC) 9 Session 4: State of the art technology and standards, 12/10/2012 • "From MODIS scenes M1, M2, and M3, the absolute of the difference between red and nir, in HDF-EOS" • …but only those where nir exceeds 127 somewhere

  10. WCPS By Example for $c in ( M1, M2, M3 ), $r in ( R )where some( $c.nir > 127 and $r )return encode(abs( $c.red - $c.nir ), "hdf“ ) (hdfA) 10 Session 4: State of the art technology and standards, 12/10/2012 • "From MODIS scenes M1, M2, and M3, the absolute of the difference between red and nir, in HDF-EOS" • …but only those where nir exceeds 127 somewhere • …inside region R

  11. Web Processing Service • OGC 05-007r7, Web Processing Service, OGC Implementation standard. • OGC 08-091r6, Corrigendum for ... Web Processing Service. • OGC 08-068r2, Web Coverage Processing Service (WCPS) Language Interface Standard, version 1.0.0. • OGC 12-029, Web Processing Service Best Practices, Discussion Paper, OGC Implementation standard. • OGC 08-123: WPS-T Discussion Paper • OGC 09-045, Web Processing Service – Application Profile for Coverage Processing. Version 0.2.0 (status: discussed in WPS.SWG). * (*) OGC Standards. Session 4: State of the art technology and standards, 12/10/2012

  12. Case Study 1: DREAM • DREAM Project • Dedicated to exploitation of EO data for Decision Support • Based on use cases of European Institutions: EUSC and EMSA • WPS used for Quality Assessment of ortho-images and Orthorectification of raw images: • Horizontal Accuracy Maps • Vertical Accuracy Maps Slide courtesy DREAM Project (ASTRIUM-GEO) Session 4: State of the art technology and standards, 12/10/2012

  13. O2 – Online Data Access Server O1 – Online Data Access Client …. WMS Server IF-OGC-View Viewer WMS WCS Server Q1 – Data Quality Client Web JS Plug-in IF-DREAM-Q-DataQualityProxy Q1 - Data Quality Proxy IF-DREAM-O-RegisterData [IF-HMA-Download] OGC WPS Service Interface OrthoRectify WPS Process OGC WPS Client IF-DREAM-Q-AssessQuality,IF-DREAM-Q-OrthoRectify AssessQuality WPS Process Raw Image Handler Ortho-Image Producer Data Quality Service OGC WPS Service Interface Compute AOI WPS Processing IF-DREAM-Q-FileIO Reference3D Database Compute RPCWPS Processing File System • Raw Acquisition • Ortho-image • Shift Shapefile • QualityMask Coverage Reference Orthoimages DEM Compute Quality WPS Processing Photogrammetry Library Web Application Server Slide courtesy DREAM Project (ASTRIUM-GEO) Session 4: State of the art technology and standards, 12/10/2012

  14. Case Study 1: DREAM Session 4: State of the art technology and standards, 12/10/2012 • DREAM, involving end users with massive data sets, allows to establish relevant demo scenarios. Sample WCPS use cases: • „all coverages with timestamp younger than T within bbox B where cloud percentage is below P“ • User input: T, B, P • Web client to allow input of T and P via sliders (or similar), B graphically • „Bit mask of fires detected in coverage C“ • „band ratio between B1 and B2 for given coverage C“ • User input: B1, B2, C • Scenario cases to be refined with users.

  15. Case Study 2: LandSAfe • Landing Site risk Analysis software framework (ESTEC) • ESA Lunar Lander Mission • Vision-based Navigation technologies for safe precision landing. • Hazard detection and avoidance. Session 4: State of the art technology and standards, 12/10/2012

  16. Case Study 2: LandSAfe • Software modules for mission preparation stage and use during descent stage (live footage) Session 4: State of the art technology and standards, 12/10/2012

  17. Case Study 2: LandSAfe • Processing requests sent over HTTP/SOAP to the WPS service which executes the request by running Python scripts from the LandSAfe Core. pyWPS See modules on previous slide Session 4: State of the art technology and standards, 12/10/2012

  18. Case Study 3: G-POD • ESA Grid - Processing On Demand • WPS access point of G-POD installation is http(s)://<gpod_host>/wps/ • WPS process • Refers to G-POD service • Has identifier, title and abstract • Output is RDF document referring to actual results • G-POD service parameters are LiteralData (no ComplexData). Session 4: State of the art technology and standards, 12/10/2012

  19. Case Study 3: G-POD • WPS as interface for ngEO Hosted ProcessingFacilities WPS e.g.G-POD Session 4: State of the art technology and standards, 12/10/2012

  20. Case Study 4: SSEGrid • SSEGrid Project • Introduce a Grid Processing on Demand model at VITO to be integrated with Vegetation processing chains • Portal to integrate multi-Grid processing services to be used by the Principal Investigator (PI) for the systematic processing or re‑processing of EO data in a single or multi-mission environment • On-demand Processing of EO data • Large EO data size  processing must be done where data is located • Re-processing by PI  processing chains must be deployable by PI Session 4: State of the art technology and standards, 12/10/2012

  21. Demonstration Scenario (FAPAR) for each = parallel execution data transfer Fraction of Absorbed PhotosyntheticallyActive Radiation Session 4: State of the art technology and standards, 12/10/2012

  22. One per mission Ground Segment SSEGrid WPS Hierarchy User Processing Infrastructure (UPI) Federating Processing Infrastructure (FPI) Mission Processing Infrastructure (MPI) Grid Access Infrastructure (GAI) Grid WPS-G WPS-G Grid Grid Workflow Engine Java SAGA WPS-G Federating Workflow Engine Portal PI WPS-G WPS-G Grid Workflow Engine Grid Java SAGA Hierarchy of WPS-G services Session 4: State of the art technology and standards, 12/10/2012

  23. Workflow Engine Workflow WPS-G Execute Grid Workflow Compute Elements Nodes Deployment Monitoring GRAM Meta-scheduler job queue WPS-G Execute RFT Scheduler . . . MDS Deployment Monitoring SSEGrid WPS-G Interface • Encapsulates all forms of processing • Grid Processes • Workflows Session 4: State of the art technology and standards, 12/10/2012

  24. AOI & threshold selection • Date • Threshold • AOI • Target PNG (as URL) Image thresholding • Highlight colour, fixed to (RGB=200,0,0) • URL of thresholded PNG, from previous step Image highlighting • Highlighted PNG, from previous step Display results Highlighting of thresholded data Case Study 5: SSE Session 4: State of the art technology and standards, 12/10/2012

  25. Session 4: State of the art technology and standards, 12/10/2012

  26. Open-source Implementations • FP7 GENESIS Toolbox • Evolution of SSE Toolbox • Supports WPS 1.0.0 and corrigendum • Multiple bindings + asynchr. • http://code.google.com/p/toolboxenvironment/ • PyWPS • Implementation in Python • 52°North • Supports WPS 1.0.0 • Supports WPS-T • Extended with WPS-G On-demand Processing in SSEGrid Session 4: State of the art technology and standards, 12/10/2012

  27. Open-source Implementations • SSEGrid contribution: • Asynchronous Execute operation with callback (no polling) • New WPS-G specific operations • GetStatus, Cancel • GetAudit, DeployData, UndeployData, DescribeData • Two new WPS-T backends • Apache ODE Process Manager (workflows) • JavaSAGA Process Manager (Grid processes) • Backend callback operation • Available as Open source at 52°North repository https://svn.52north.org/svn/geoprocessing/main/WPS/branches/WPS-Spacebel Session 4: State of the art technology and standards, 12/10/2012

  28. Issues and future work • Lack of support for: • Asynchronous operations (not polling) • Status reporting (Audit) • Cancel, GetAudit, … • Quota Management, • Discovery of SLA, QoS, Performance, availability • Integrating Authentication/Authorisation and security • Restrict visibility/access to deployed Processing resources • Discover access restrictions • Harmonising data model across services: • DescribeTasking (SWECommon) versus DescribeProcess Session 4: State of the art technology and standards, 12/10/2012

  29. Issues and future work • Consistency with INSPIRE Spatial Data Services and Invoke Service (Draft) Implementing rules • May or may not retain OGC WPS. • Oriented towards WSDL and SOAP. Session 4: State of the art technology and standards, 12/10/2012

  30. Issues and future work • Completion of standardisationof various interfaces defined in ESA TM-21 « HMA Cookbook » No yet OGC standards. Partly underway in WPS 2.0 SWG Session 4: State of the art technology and standards, 12/10/2012

  31. WPS-G Processing On-Demand Extension Binding Ongoing Standardisation Activities • SSEGrid Project: • Two Change Requests submitted for inclusion into WPS 2.0 • Processing On-Demand Extension (WPS-G) to be proposed when WPS 2.0 available WPS-T Asynchronous Session 4: State of the art technology and standards, 12/10/2012

  32. Ongoing Standardisation Activities • SSEGrid Project: • “Processing On-Demand WPS Extension” • Covers changes and additions to WPS 1.0 • Available at https://wiki.services.eoportal.org/tiki-download_wiki_attachment.php?attId=1645 Session 4: State of the art technology and standards, 12/10/2012

  33. Workflow 1: Execute WPS-G 2: Callback WPS with On-Demand Extension • Feature/operations • Discovery of service metadata (WPS GetCapabilities operation) • Discovery of process metadata (WPS DescribeProcess operation) • Execution of processes (WPS Execute operation) • Deployment of processes (WPS-T DeployProcess operation) • Undeployment of processes (WPS-T UndeployProcess operation) • Deployment of (auxiliary) data (WPS-G DeployData operation) • Undeployment of data (WPS-G UndeployData operation) • Discovery of data metadata (WPS-G DescribeData operation) • Query of process status (WPS-G GetStatus operation) • Cancellation of process (WPS-G Cancel operation) • Query of process execution trace (WPS-G GetAudit operation) • Process completion/failure notification (callback operation) Addressed by WPS 2.0 SWG Session 4: State of the art technology and standards, 12/10/2012

  34. Ongoing Standardisation Activities Session 4: State of the art technology and standards, 12/10/2012 • DREAM Project: • “Identify the requirements and scenarios, extend the WCPS to cover the needed multidimensional queries in particular for what concerns space/time phenomena related to the joint exploitation of S-2, optical high resolution and other data including geo-statistical data as the one routinely provided by EUROSTAT and other international organisations. “ • WCPS work items: • WCPS 1.0 based on WCS 1.1.2. Standardisation effort required to align with GMLCOV 1.0 and WCS 2.0 (Core) • ATS to be updated • ETS to be created

  35. Planned Standardisation Activities • HMA-S Project: • Revise WPS Specification taking into account: • ESA project’s technical report “SSEGrid Processing On-Demand Extension (WPS-G)” • ngEO needs in terms of process description and access to the processing outputs • Relevant third party initiatives, about cloud computing standardisation, such as IEEE P2301. • WPS 2.0 SWG ongoing work • ESE Project: • Create new WPS-G deployment profile for Map-Reduce processing environment on dynamic resources (cloud). Session 4: State of the art technology and standards, 12/10/2012

  36. Conclusion • Two main « Processing » specifications: WPS and WCPS • WPS: • Widely adopted in ESA projects • Several open-source implementations available • Many subjects to be resolved, improved or not yet a formal standard • E.g. asynchronous, WPS-T, … • Definition of OGC WPS 2.0 underway but slow process • WCPS: • Less widely adopted • To be used in DREAM and ESE projects • Various ongoing or planned ESA projects contribute to standardisation effort of both specifications. Session 4: State of the art technology and standards, 12/10/2012

  37. Conclusion • Further Reading: • HMA Cookbook: http://esamultimedia.esa.int/multimedia/ publications/TM-21/TM-21.pdf • http://www.opengeospatial.org/standards/wps • http://www.opengeospatial.org/standards/wcps • http://inspire.jrc.ec.europa.eu/documents/Network_Services/position_positionpaper_06062011pd__EN_1.pdf Session 4: State of the art technology and standards, 12/10/2012

More Related