1 / 20

Open Solutions to Regional Observing Systems

Open Solutions to Regional Observing Systems. Outline.

dennis-mcclain
Download Presentation

Open Solutions to Regional Observing Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Open Solutions to Regional Observing Systems

  2. Outline Recent near real-time in-situ observations are aggregated to a ‘Xenia’ schema relational database(RDB) via its import XML format ‘ObsKML’ schema. From this database of recent observations a variety of file formats, web services and applications can be driven. By suggesting a minimally common observation oriented XML and RDB schema, developed scripts and products can benefit from and build around these shared schema. For more info contact jeremy.cothran@gmail.com Keywords Scripting languages Perl PHP Relational Database Sqlite Postgres Mapping Engine/Interface Mapserver OpenLayers TileCache OpenGeospatial Consortium(OGC) services WMS/WFS/SOS Google Earth/Maps KML/KMZ Google Code/Analytics

  3. Problems Xenia intended to address Grants for research instrumentation which will be collecting observation data while lacking a data management/sharing component beyond basic file storage Low-volume data(< 100,000 records per hour) in-situ observational platforms or system arrays (e.g. 1 to 1000 platforms collecting 10-20 observations per hour) collecting data at any geographic scale (local,regional,national,etc) Bridging the gap between raw data collection and the organization and sharing of data using previously developed products, services and standards(leveraging earlier work against new data providers) Fostering a standardization of products and services via a common openly shared technical infrastructure(common database schema and product support scripts)

  4. Problems Xenia not intended to address High-volume data (millions of records per hour) such as gridded model outputs, hf radar, etc. High-volume data problems at this time are better addressed using traditional file processing techniques where data management can suggest output file formats(such as images, shapefiles, etc) and metadata that are conducive to search and usage needs.

  5. Xenia Relational Database Schema Hierarchy = organization->platform->sensor->multi_obs(observation data)

  6. ObsKML XML schema for import/export

  7. Data flows – general

  8. Data flows – ObsKML ObsKML (Observations KML) A simple XML encoding of observation metadata associated with a KML Placemark. Default XML import/export format for Xenia database Instances. Postgres or Sqlite database paths

  9. Xenia data organized by time interval Julian weekly divided database files(sqlite) of all observations are available http://tinyurl.com/6ctz66 New Data Latest past several hours Recent 0-6 weeks Archival 3+ weeks to 1-2 years Possibly table separated by year,month,etc Archival file 1-2+ years Files separated by product/year/month

  10. Xenia aggregation, replication, redundancy Overlapping/redundant systems could aggregate/replicate via SQL or XML Xenia A,B,C,D,E,F Xenia Backup A,B,C,D,E,F Xenia A,B,C Xenia D,E,F Xenia Backup D,E,F Xenia A Xenia B Xenia C Xenia D Xenia E Xenia F

  11. Flow/event monitoring Around 10,000 observations are collected each hour from a variety of data providers. The data flows are monitored for both successful import and towards provided report metrics or notifications when problems or events occur. Sensor count graph showing the hourly number of successful sensor imports per provider More detailed platform/observation table showing daily counts color coded to highlight specific sensor issues

  12. Applications – National Weather Service Latest regional platform observations from Xenia database are styled to html tables and presented in fixed map interface alongside NWS forecast warnings/advisories and other map layers of interest http://forecast.weather.gov/mwp/

  13. Applications – SECOORA interactive map Latest regional observation maps are accessed via WMS(Web Mapping Service) and merged into an OpenLayers(browser javascript) map interface. Observation database requests are supported via WMS GFI(Get Feature Info) http://secoora.org/maps/dynamic

  14. Applications - CarolinasRCOOS Map image layers supported via MapServer WMS with increased map response time via TileCache which tiles and caches image requests for repeated reference. Site feature info supported via Xenia database instance styled output html table accessed via WMS GFI. http://carolinasrcoos.org

  15. KML/KMZ – Latest obs by platform KML (Keyhole Markup Language) which is the XML format used to visualize data in Google Earth/Maps and potentially other globes/maps with KML support such as NASA WorldWind and ESRI ArcExplorer Latest observation data organized by all observations per platform http://tinyurl.com/664wtx

  16. KML/KMZ – latest obs by obsType KML (Keyhole Markup Language) which is the XML format used to visualize data in Google Earth/Maps and potentially other globes/maps with KML support such as NASA WorldWind and ESRI ArcExplorer Latest observation data organized by observation type for all platforms http://tinyurl.com/664wtx All observations carry * observation type(obsType) * unit of measure(uom) * measurement value obsType+uom = measurement type(m_type) Color styled low to high value blue/green/red

  17. Development/documentation support Google Code provides very simple, free browser based access to the basic project support tools such as a subversion(svn) code/documentation sharing/versioning, wiki documentation and issue tickets. Google Analytics provides free browser based access to website usage statistics via a javascript project token included in webpage access.

  18. Lessons Learned • Web GIS interface: php/mapscript -> openlayers/javascript/ajax, tilecache • Latest in-situ obs: RDB – sensor/obs hard-coded/table approach -> obs non-specific infrastructure(xenia,obskml) • Sensor/observation specific metadata: loosely link a variety of more specific metadata files/schemas to common infrastructure • Hourly gridded data: RDB quickscat, model layers -> ogr/sql+shapefiles • Full RDB to file-based RDB: postgresql+postgis -> sqlite+ogr, sqlitegeo? • Self hosted to third party hosted: third party website hosting for static content/context, common processing/storage. Google code/analytics for basic shared development support/documentation. • ROA(Resource Oriented Architecture)/REST style simpler declarative alternatives to complex query SOA(Service Oriented Architecture)

  19. File formats and web services • File formats (CSV,shapefile,netCDF,KML/KMZ,Sqlite DB) http://secoora.org/data Free desktop analysis tools MapWindows(shapefile/GIS) http://www.mapwindow.com ncBrowse(netCDF) http://www.epic.noaa.gov/java/ncBrowse ODV(ctd/csv) http://odv.awi.de Sqlite DB http://sqlite.org • Web services (OGC WMS/WFS/SOS) http://secoora.org/data http://code.google.com/p/xenia/wiki/XeniaSOS

  20. Credits 2C.Calloway,2J.Cleary,1J.Cothran,4J.Donovan, 3J.Dorton,1M.Fletcher,C.Galvarino,1S.King, 2S.Haines,3L.Leonard,1D.Porter,3X.Qi, 1D.Ramage,2H.Seim,4V.Subramanian,1S.Walker,4R.Weisberg 1University of South Carolina, 2University of North Carolina at Chapel Hill, 3University of North Carolina at Wilmington, 4University of South Florida

More Related