Scientific Data Platforms for South Africa and Beyond

1. Scientific Data Platformsfor South Africa and Beyond 18 October 2011 GEOSS Interoperability Workshop Wim Hugo

2. Our Aim • It is possible, desirable, and in the public interest to: • Ensure that scientific data is described properly and discoverable; • Once discovered, its utility, quality, and scope can be understood, even if the data sets are huge; • Once understood; it can be accessed freely and openly; • Once accessed, it can be included into distributed processes, preferably automatically, and on large scales; • Once processed, the knowledge gathered can be re-used.

3. Solution • “Shared Implementation” • a framework containing architecture guidelines, abstract specifications and user requirements, recommended interoperability standards, and reference implementations. • a directed open source community that contributes to, extends, and maintains the shared implementation. • National funding provides a baseline of operations and activity. • not domain-specific, but function-specific. • syntactic and schematic interoperability Almost all knowledge portals share a base set of functions that are re-usable, and should survive individual projects

4. Solution • “Shared Meaning and Content” • meta-data standards selection • crosswalks (inter-standard translations) • controlled vocabularies, thesauri, and ontologies associated with standards – these are often region-specific • support for ‘Linked Open Data’ and semantic interoperability • support for ‘Data Democracy’, WDS, and GEO

5. Solution • “Shared Processes” • coordinated training and awareness creation on the importance of meta-data • data sharing and meta-data provision policies / guidelines to be implemented at research institutions and as a condition of funding • coordination of and assistance with contributions to national infrastructure • creation of forums for the validation of decisions and guidelines

6. Science Ecosystem ‘Science at Large’ Community Participation Content Validation Architecture Framework Reference Implementations Software Infrastructure Physical Infrastructure Earth and Environmental Observation BioInformatics and Health Human and Social Sciences

7. Earth and Environmental Observation

8. The Bigger Picture: Technology Roadmap “Directed” Open Source Local Capacity CoSAMP (SAEON, CSIR) 2005 CoGIS (SAEON, CSIR) 2007 Meta-Data I (SAEON, CSIR) 2008/9 URS I Spatial Extensions (SAEON, SAEOS, WDC, R&VA) 2009/10 Structured Data (SAEON, SAEOS, WDS) 2009/10 URS II URS III Citations Service Orientation Admin Extensions (SAEON, SAEOS) 2011/12 SensorWeb NetCDF (SAEON, SAEOS, R&VA, WDC) 2010/11 URS IV Meta-Data II (SAEON, SAEOS, WDS, R&VA) 2009/10 Process (CSIR, SAEON) 2011 Knowledge (CSIR, SAEON) 2012

9. Portals and Platform Main Stakeholders and Initiatives Main Platform Services Documents and Publications Spatial Data Other Data SAEON Nodes and Initiatives SAEON Data Analysis and Processing SAEOS Portal SAEOS World Data Centre for BHH WDC-BHHA Content Visualisation and Presentation Risk and Vulnerability Atlas RAVA Meta-Data and Data Management Functions CSIR CSIR Meta-Data and Data Services Others … Others … NSIF NSIF Portal NSIF Application

10. Main Use Cases - Phase 1: Discovery and Evaluation

11. Main Use Cases - Phase 2: Mediation and Collation

12. Main Use Cases - Phase 3: Automated Scientific Workflow ?

13. Updated Distributed Data Standards Architecture Distributed Web Folders Standardised Data Services Standardised Catalogue Services User Interface Layer NetCDF nDTM OPeNDAP Server THREDDS Server Viewers Other Objects JSON/ Atom RSS RDF Services OGC-CSW OAI-PMH Mediators Relational Spatial DB - OGC Search Interfaces O&M Schema 3DTM Sensor Observation Service Context Typical Spatial Data 2DM OGC Map Services SHP, TIFF, etc.

14. Examples: Interoperability • Distributed WxS in OpenLayers Client • Persist distributed WxS as WCD • Distributed WxS, KML, and GeoRSS in OpenLayers Client • CS/W Request as Dublin Core • CS/W Request as JSON/Atom to Search Result Presentation • CS/W Request as GeoRSS to OpenLayers Client • CS/W Request integrated with OpenLayers Client • CS/W Request integrated with Desktop Client • Search GUI to any CS/W endpoint • Search GUI to OAI-PMH endpoint • EuroGEOSS Broker Return to Search Presentation • CS/W Request to EuroGEOSS Broker • SOS in Extended OpenLayers Client • NetCDF in Extended OpenLayers Client • HDF-4 in Extended OpenLayers Client • Distributed WFS, NetCDF, and SOS to Charting Services (Google Charts)+ mediation • Citizen Science Module to WFS • WPS in Scientific Workflow + mediation

15. Multiple WxS Sources FAO, Rome, WMS Meraka Institute, Pretoria, SA, WFS

16. Persist Web Context Documents Load from server storage List of WCDs

17. Support Multiple Data Standards GeoRSS WxS Also: KML, NetCDF, HDF-4, SOS

18. Meta-Data Repository Multiple Custodians Meta-Data Entry

19. Search Interface and Result Formats Meta-Data Entry Multiple Formats

20. GeoRSS Result in Compatible Client

21. GeoRSS Piped to Map Client

22. Non-Spatial Representations of Data Data captured by Schools Program directly to WFS Mediated HTTP calls to Google Charts

23. Example Detailed Use Case: Services Integration

24. Challenges • Lack of Resources to Implement Complex Specifications (SWE being a prime example) • The branch is far from the limb • Continuity of Projects • Data and Meta-Data Programme: goodwill vs. budgeted KPA.