How to join the environmental data revolution - PowerPoint PPT Presentation

How to join the environmental data revolution
1 / 27

  • Uploaded on
  • Presentation posted in: General

How to join the environmental data revolution. Jon Blower Reading e-Science Centre Environmental Systems Science Centre University of Reading United Kingdom. Pre-revolutionary times. Technical difficulties stand in the way of effective data sharing

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

How to join the environmental data revolution

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

How to join the environmental data revolution

How to join the environmental data revolution

Jon Blower

Reading e-Science Centre

Environmental Systems Science Centre

University of Reading

United Kingdom

Pre revolutionary times

Pre-revolutionary times

  • Technical difficulties stand in the way of effective data sharing

    • plethora of file formats, coordinate systems etc

    • expensive GIS software plus vendor lock-in

  • Inhibits collaboration and effective publication of research results

  • In day-to-day collaboration and publication, scientists often share static images or plots of data, not the data

    • very inflexible, hard to combine with new data

    • if data is provided, often in home-grown file format

What is the environmental data revolution

What is the environmental data revolution?

  • Common standards and simple, cheap or free software now available for data sharing

  • A “revolution” in both senses:

    • enables a big change in behaviour

    • “power to the people”: don’t have to be a technical guru

  • You can all do this!

Extensible markup language xml

eXtensible Markup Language (XML)

  • A structure for holding data in plain text files

  • Everything is defined between tags like HTML:

    <name>Fred Bloggs</name>

  • Tags can be nested and have attributes:

    <Person gender=“male”>

    <name>Fred Bloggs</name>


  • Can be generated and read by all languages on all platforms

    • and is (just) human-readable

    • lots of software libraries available

  • But it’s verbose: not suitable for large datasets

Netcdf common data format

NetCDF (Common Data Format)

  • Binary format suitable for lots of different data types

  • Stores data and metadata (description of data)

    • hence self-contained

  • Climate and Forecast conventions should be applied to environmental data if possible

    • for correct labelling of axes (latitude, longitude, time, pressure, etc)

    • for standard naming of variables (sea_water_potential_temperature etc)

  • Software libraries available for most major languages

  • Use it if you can!

    • Don’t invent your own binary format if possible

How easy can it be to share data

How easy can it be to share data?

  • Simplest way is probably GeoRSS

    • RSS = “Really Simple Syndication”

  • Adds geo-referencing information to RSS:

    <rss version="2.0">


    <title>Some stuff</title>


    <title>My House</title>

    <georss:point>51.7 -1.0</georss:point>




  • Not very sophisticated but not a bad start

  • Items can be time-stamped

Displaying georss on your website

Displaying GeoRSS on your website

  • Very easy (honest)!

    • Download WorldKit (

    • Unzip into your webspace

    • Change config file to point at a GeoRSS file

Keyhole markup language kml

Keyhole Markup Language (KML)

The format of Google Earth

Much richer than GeoRSS, encodes lots of types of geographic data

Understood by many clients:

Google Maps

NASA World Wind

Other GIS software

Now on the “proper” standards track

KMZ = zipped KML

for large datasets

Can package up imagery and icons

Kml ground overlays

KML ground overlays

Kml storm track katrina

KML storm track (Katrina)

  • Points styled according to intensity

  • Can be animated

  • KML contains just as much information as original TRACK output file!

3 d model output in kml

3-D Model output in KML

  • Eruption of Cleveland volcano modelled by PUFF (Alaska VO)

  • KML contains “altitude” attribute so can display things in the atmosphere

  • (but doesn’t contain “depth” attribute for ocean!”)

Web map service wms

Web Map Service (WMS)

  • Generates map images

  • Advertises its capabilities through XML document

  • Clients can request data in a number of formats:

    • e.g. JPEG, GIF, PNG, KML!

  • … and map projections, styles, etc

  • Map request is encoded in one big long Web address

    • This address is a unique reference for the image

    • Can be passed around, emailed etc

  • Designed for overlaying of data from different sources

Wms in action

WMS in action

Background imagery from NASA

Ocean data from ESSC

Lots of GIS software supports WMS, including WorldKit





Could use

images from

many other








WMS = OGC-compliant

Web Map Service





Web server and WMS

could be co-located

Exposing your data through wms

Exposing your data through WMS

  • ReSC has developed a WMS for NetCDF data (imaginatively entitled “ncWMS”)

  • We’re encouraging other groups to use the software to share their own data

  • See me if you want to share your data in this way

    • Can run your own server or add data to ours

Non geographic data svg

Non-geographic data: SVG

  • Timeseries, depth profiles etc do not plot on a map

  • Scalable Vector Graphics (SVG)

    • Describe data in XML format

  • Renders to a graphical plot

    • Support increasing in modern browsers (e.g. Firefox, right)

  • Can be interactive

    • Zoom in, pan around

    • Show values on the graph

  • Tools to create SVG are growing

  • NERC e-Minerals project used this heavily

The next level

The next level

  • So far we’ve focussed on quick and easy ways to get data “out there” in a reusable way

  • With more effort, we can do even better

  • Geography Markup Language (GML)

    • Complex, but can create complete descriptions of very complex features, e.g. weather fronts

  • Climate Science Markup Language (CSML)

    • Produced by NERC Data Grid project

    • Describes features of interest in meteorology and oceanography

  • Web Feature Service (WFS)

    • Serves features (simple and complex) in GML

    • Better representation of semantics

  • Web Coverage Service (WCS)

    • Serves data (usually gridded)

    • We have developed a WCS in DEWS project

    • Standard is not yet quite adequate for our needs (doesn’t cope with large data well)

  • Currently difficult for scientists to expose their data in these ways without technical help

    • But these things will get easier with better software tools and wider adoption

Summary so far

Summary so far

  • XML is a general data format

    • But verbose: not suitable for large datasets

  • GeoRSS and KML are flavours of XML for geographic data

    • Both easy to create and visualize

    • KML is “easy GML”, designed for simple end-user use

  • For larger datasets use NetCDF

    • CF-compliant if possible

  • SVG for non-geographic data

  • Web Map Service + Godiva2 are good for creating visualizations of gridded data

    • We have developed a WMS you can use

  • All the above are easy to use but important to remember their limitations

    • the “next level” requires more technical help

What it s all for mashups

What it’s all for: Mashups

What is a mashup

What is a mashup?

  • Taking two or more “things” that were independently produced and putting them together such that the resulting whole is greater than the sum of the parts

  • Requires adherence to common standards

  • E.g. Online map of apartments for rent, merged with local crime statistics

  • Also applies to music…

    • with “hilarious” consequences

Mashup 1 ocean science

Mashup 1: Ocean science

DAMOCLES (Arctic ice)



Java program





Google Earth

ARGO float data

DRAKKAR model data (NEMO code)

Useful for checking model results against assimilated observations to look for anomalies

Mashup 2 hurricane katrina

Mashup 2: Hurricane Katrina

SST data from Met Office FOAM model

Output from TRACK program

plain text



Python script






Google Earth

vorticity data from ECMWF reanalysis

Can check positioning of storm tracks and view effect of storms on ocean (e.g. cooling of sea surface as Katrina passes)

Summary of key technologies

Summary of key technologies


cheap or free

client software)

Google Earth

NASA World Wind


Web Mapping Service (WMS)


(Standard Web Services)

(XML languages

for geographic data)




(general-purpose data formats,

platform and language




The World Wide Web

(common communication protocol,

means to uniquely identify resources)

(common infrastructure)

The Internet

Not an exhaustive list!



  • Mostly concerned with simple visualization

    • Data analysis is a harder problem

  • Specs are all very well but not everyone implements the specs properly!

    • Especially true for WMS z and t dimensions

    • Specs often focus on 2-D (land surface)

  • Google Earth doesn’t show proper ocean bathymetry

  • Chicken-and-egg situation

    • Scientists won’t learn these new technologies until there is enough data out there, but needs scientists to make data available

The future

The Future

  • Full semantic interoperability

    • i.e. ability to search and compare data based on its meaning

    • Data described fully, together with operations that are possible on the data

  • Ability to perform data analysis operations

    • e.g. calculate difference of two remote datasets efficiently

  • NERC Data Grid

    • Discovery, visualization, download of NERC data

    • focus on atmosphere/ocean for the moment

    • strictly standards-based


    • European Spatial Data Infrastructure

Plug for event

Plug for event!

  • 2-day workshop on Google Earth and other internet mapping tools

  • Focus on scientific applications for geobrowsers and geo-websites

  • 2-3 April 2007, Cambridge, UK

  • Registration closes Feb 25th (ish)

  • Presentations and coding tutorials

  • Please email or see me if you are interested



  • Please expose your data and results (not just static plots) – it’s easy!

    • GeoRSS/Atom

    • KML (recommended)

    • Web Map Service

    • Scalable Vector Graphics

    • (WCS and WFS for more advanced)

  • For larger datasets (even intermediate results):

    • CF-compliant NetCDF

  • Experiment!

  • We’re happy to help

  • Login