MONITORING
Download
1 / 48

MONITORING - PowerPoint PPT Presentation


  • 87 Views
  • Uploaded on

MONITORING. WEATHER. CLIMATE. SEISMIC ACTIVITY. TO. A SUSTAINABLE DEVELOPMENT AND A SAFER WORLD. EUMeTrain LSA SAF Week Ricardo Silva, IM. Session 2 – Messing around with the data. Who am I?. Ricardo Silva. Where do you come from?. How often do you use LSA SAF products?. Regularly.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' MONITORING' - kolton


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

MONITORING

WEATHER

CLIMATE

SEISMIC ACTIVITY

TO

A SUSTAINABLE DEVELOPMENT AND A SAFER WORLD

EUMeTrain LSA SAF Week

Ricardo Silva, IM

Session 2 – Messing around with the data


Who am i
Who am I?

Ricardo Silva



How often do you use lsa saf products
How often do you use LSA SAF products?

Regularly

Occasionally

Neverused, but plan

to start


Session 2 summary
Session 2 Summary

  • Obtaining LSA SAF products

  • General Product Information

    • Documentation

    • File structure

  • Quick Visualization using HDFView

  • GIS tools for visualization and analysis

    • Quantum GIS

    • GRASS GIS

    • ILWIS

  • Tool for georeferencing LSA SAF products

  • Using programming tools – Python


Obtaining lsa saf products
Obtaining LSA SAF products

  • Near Real Time Dissemination via EUMETCast

  • Dissemination on-demand

    • http:landsaf.meteo.pt

    • Products are available for registered users

      • Registration is free of charge, but may take up to 3 working days for approval

    • Downloads use the FTP protocol

    • Several download methods according to desired temporal extent and download settings


How do you obtain lsa saf products
How do you obtain LSA SAF products?

EUMETCast

LSA SAF

(website, FTP, helpdesk)

Other?


Obtaining lsa saf products1
Obtaining LSA SAF products

  • Dissemination options

    • Ordering methods

      • Random slot ordering

        • The user can select individual time slots for several products

        • Automatic dissemination

      • Long series ordering

        • Ordering of long time series of products.

        • Requires human intervention and may take up to one week

    • Dissemination methods

      • LSA SAF FTP server

        • The order is placed in the LandSAF server for the user to download

      • User's personal FTP server

        • LSA SAF's system transfers the order directly to the user's FTP server


Product documentation
Product Documentation

  • Product User Manual (PUM)

    • Main documentation source

      • Physical phenomena

      • Algorithm basis

      • Output data specifications

      • Etc.

  • Product Output Format (POF)

    • Details concerning description of metadata, file naming formats, etc.

  • Validation Report (VR)

  • Algorithm Theoretical Basis Document (ATBD)


Product file structure
Product file structure

  • LSA SAF products use the HDF5 format

    • http://www.hdfgroup.org/

  • Hierarchical Data Format (HDF)

    • Groups – Containers for datasets and other groups

    • Datasets – Multidimensional arrays of a homogeneous type

    • Attributes – Metadata on groups and datasets

  • HDF files:

    • Are self-describing. Complex data relationships and dependencies

    • Can contain binary data

    • Support n-dimensional datasets

  • Each element in the dataset may itself be a complex object


Product file structure1
Product file structure

  • LSA SAF HDF5 files:

    • A common set of attributes on the root group

      • general information

    • A dataset for the main product (LST, DSLF, etc)

    • Additional datasets for cell metadata (Quality flags, etc.)

    • Each dataset may have:

      • Specific attributes (Scaling factor, missing value, etc.)

      • Different dimensions (Most datasets are 2D)

      • Different data types (Most datasets are 16bit signed integers)


Quick visualization of products
Quick visualization of products

  • HDFView software

    http://www.hdfgroup.org/hdf-java-html/hdfview/index.html#download_hdfview

    • Java-based HDF Viewer

    • Inspection of a HDF5's structure and data

    • Visualize data in spreadsheet or image mode

    • Basic statistics, line plotting and histogram generation

    • Export to image formats



Hdfview
HDFView

  • Datasets

  • Attributes

  • Table View

  • Table options

  • Image view

  • Image options

  • Pixel values


Hdfview1
HDFView

Inspecting file and dataset attributes


Hdfview2
HDFView

  • When visualizing products take note:

    • Cell values are unscaled

    • The 'missing value' values have no physical meaning

    • Product is not georeferenced (line, column coordinates only)


Gis tools for visualization and analysis
GIS tools for visualization and analysis

  • Quantum GIS

    • User friendly interface

    • Many plugins created by the community

  • GRASS GIS

    • Many raster analysis modules

    • Python scripting

  • ILWIS

    • Direct import of LSASAF products through the GEONETCast toolbox

  • Other tools (ArcGIS, Mapinfo, SAGA, etc.)


Quantum gis
Quantum GIS

  • http://www.qgis.org/

  • Open source multi-platform GIS software

  • User-friendly interface

  • Reads many input formats

    • Raster files, Vector files, Web services, etc

  • On-the-fly reprojection for vector and raster data

  • Large and friendly community of users and developers


Quantum gis1
Quantum GIS

Integration of different data sources


Quantum gis2
Quantum GIS

Quick inspection of relationships between data


Grass gis
GRASS GIS

  • Geographic Resources Analysis Support System

    • http://grass.fbk.eu

  • Open source multi-platform GIS software

  • Hundreds of modules for raster and vector processing

  • Reads many input formats

    • Can read HDF5 directly (won't be georeferenced)

    • It is preferable to georeference the HDF5 file first


Grass gis1
GRASS GIS

Modular and flexible user interface


Grass gis2
GRASS GIS

  • Raster calculator

    • Allows for computing a combination of logical and algebraic expressions

    • Example: Calculate the arithmetic mean LST values for a specific day (using hourly sample)

      • 24 raster files each with the LSA SAF LST product for the hourly timeslots

Approach #1: Using only cells where there is enough data for the whole day

(In GRASS, any math performed with NULL cells always results in a NULL value for these cells

Approach #2: Using all the cells where there is at least one value in the day


Grass gis3
GRASS GIS

Approach #1


Grass gis4
GRASS GIS

Approach #2


Grass gis5
GRASS GIS

  • Most commands are available in two interfaces:

    • Graphical – Good for visualization and exploratory analysis

    • Command-line – Nice for automating repetitive tasks


Are you used to creating small scripts for automating tasks
Are you used to creating small scripts for automating tasks?

Yes

Not really

I might start using them

In the future...


Grass gis6
GRASS GIS

GRASS 6.4.1 (world4326):~ > python

>>> # example importing all the LST geotiffs into GRASS

>>> import grass.script as grass

>>> from glob import glob

>>> import os

>>> import datetime as dt

>>>

>>> for path in glob('/home/geo2/test_data/lsasaf/modificados/georefs/*LST*.tif'):

... fName = os.path.basename(path)

... nameFields=fName.split('_')

... product, area = nameFields[3:5]

... timeslot = dt.datetime.strptime(nameFields[5], '%Y%m%d%H%M')

... newName = '_'.join((product, area, timeslot.strftime('%Y_%m_%d_%H_%M')))

... grass.run_command('r.in.gdal', flags='oe', overwrite=True, input=path, output=newName)

  • Python scripting example:

    • Automatically import and rename 191 LST files for a given day


Ilwis
ILWIS

  • Integrated Land and Water Information System

    • http://52north.org/communities/ilwis/

  • Open Source GIS software

  • Available only for MS Windows (as of now)

  • Imports LSA SAF products, using the GEONETCast Toolbox

  • Several modules for raster and vector processing


Geonetcast toolbox
GEONETCast Toolbox

  • GEONETCast Toolbox

    • http://52north.org/communities/earth-observation/geonetcast-toolbox-software

    • Available as a toolbox for ILWIS

    • Developed by ITC – University of Twente

    • Enables easy access to and management of GEONETCast data

      • various satellite and environmental data and/or resulting products

    • Tailored for users running GEONETCast reception stations


Ilwis geonetcast
ILWIS/GEONETCast

  • Installation

    • Follow the GEONETCast Toolbox's user manual

    • Read section 1.1 and 1.2 CAREFULLY

  • Importing LSA SAF products

    • Beware of file names:

      • Files obtained through GEONETCast (EUMETCast)

        • S-LSA_-HDF5_LSASAF_MSG_LST_Euro_201111060000.bz2

        • Everything will work OK

      • Files obtained through LSA SAF's on-demand dissemination

        • HDF5_LSASAF_MSG_LST_Euro_201111060000.bz2

        • GEONETCast Toolbox expects the 'S-LSA_-' prefix and will throw errors

        • Solutions:

          • Uncompress → rename → recompress (time consuming and boring)

          • Modify the GEONETCast's batch import scripts

          • Contact LSA SAF's helpdesk and they'll send you the modified scripts


Ilwis geonetcast1
ILWIS/GEONETCast

Modular user interface, integrating the GEONETCast Toolbox


Ilwis geonetcast2
ILWIS/GEONETCast

Use native NGP projection of the LSA SAF products


Have you used these gis tools before
Have you used these GIS tools before?

Yes, some of them

No

I might start using them

In the future...


Lsa saf product coordinate system
LSA SAF Product Coordinate System

  • Normalized Geostationary Projection (NGP)

    • Projection of the underlying MSG LRIT source data

    • Important parameters for MSG satellite:

      • Satellite height above ellipsoid: 35785831 m

      • Central meridian: 0º

    • Documented on EUMETSAT's LRIT and HRIT Global Specification

  • http://www.eumetsat.int/groups/cps/documents/document/pdf_cgms_03.pdf


Lsa saf product coordinate system1
LSA SAF Product Coordinate System

  • LSA SAF products hold coordinate system information in the attributes of the root group

  • But HDF5 files have no standard way of defining coordinate systems

    • It's hard to automatically import LSA SAF products into a Geographical Information System (GIS) software AND have the data correctly georeferenced


Visualization of lsa saf products
Visualization of LSA SAF products

GIS tools are nice for

visualization, analysis

map making, etc.

HDFView is nice

for a quick inspection

Most GIS tools cannot

recognize LSA SAF

coordinate information

Some GIS tools

can open HDF5 directly

BUT ...

The files will not be

projected

It won't be possible to

overlay different data

UNLESS …


Lsasaf georeferencer tool
LSASAF Georeferencer Tool

  • A graphical tool for georeferencing LSA SAF products

  • Still in-development, but already usable

  • Open development

    • Source code hosted on a public repository

    • https://github.com/ricardogsilva/LSASAF-georeferencer

  • Uses open source tools

    • Python, GDAL, Proj, Qt4

  • Multi-platform


Lsasaf georeferencer tool1
LSASAF Georeferencer Tool

  • Convert LSASAF HDF5 files to GeoTiff

    • GeoTiff is a Tiff based interchange format for georeferenced raster imagery

  • Reproject from the NGP projection to other coordinate systems

  • Apply scaling factor and missing value information to the output file

  • Output GeoTiff files are suitable for further processing in GIS software

  • Process multiple files simultaneously

  • Also has a CLI interface, useful for scripting


Lsasaf georeferencer tool2
LSASAF Georeferencer Tool

  • Graphical Window usage example

  • Select the file(s) to process

  • Load file information

  • Choose the dataset(s) to be processed

  • Choose the output projection

  • Choose the output directory

  • Process the files


Lsasaf georeferencer tool3
LSASAF Georeferencer Tool

# display usage information

[email protected]:~$ ./georef_hdf5.py -h

# process all the LST files in ~/input_dir

[email protected]:~$ ./georef_hdf5.py -x -o ~/my_output_dir ~/input_dir/*LST*

  • Command line usage example


Guess who is developing the lsa saf georeferencer tool
Guess who is developing the LSA SAF Georeferencer Tool?

Ricardo Silva

(me) :)

Some other dude...

  • Contributions, bug reports, etc. are very much welcome!


Programming tools python
Programming tools – Python

  • Python(x, y)

    • http://code.google.com/p/pythonxy/

    • A (free) scientific and engineering development software for

      • numerical computations,

      • data analysis

      • data visualization

    • Bundles Python and a big set of extra modules in a single package

      • Includes modules for manipulating HDF5 files


Programming tools python1
Programming tools – Python

  • Some useful python modules for working with LSA SAF products:

    • NumPy

      • The fundamental package needed for scientific computing with Python

      • Provides an N-dimensional array object and specialized data types and functions

    • Scipy

      • A library built to work with NumPy arrays

      • Provides efficient numerical routines for interpolation, statistics, linear algebra, etc.

    • Matplotlib

      • 2D plotting library which produces publication quality figures

    • PyTables

      • Package for managing HDF5 files

      • Designed to efficiently and easily cope with extremely large amounts of data

    • Gdal

      • A translator library for raster geospatial data formats.

      • Has read/write functionality for many raster formats (HDF4/5, GeoTiff, GRIB, netCDF, etc.)

      • Allows assigning georeferencing information


Programming tools python2
Programming tools – Python

[email protected]:~$ python

>>> import tables

>>> import numpy as np

>>> import matplotlib.pyplot as plt

>>> product = tables.openFile('path_to_HDF5_file')

# list attributes of the root group

>>> product.root._v_attrs

# list attributes of the LST dataset

>>> product.root.LST.attrs

# import the LST dataset into a 32bit float numpy array

>>> lst = np.array(product.root.LST, dtype=np.float32)

# convert missing values to nan and apply scaling factor

>>> lst[lst == product.root.LST.attrs['MISSING_VALUE']] = np.nan

>>> lst = lst / product.root.LST.attrs['SCALING_FACTOR']

# plot the data

>>> plt.imshow(lst, interpolation='nearest')

>>> plt.colorbar()

>>> plt.show()

  • Example interactive session for manipulating a LST product


Programming tools python3
Programming tools – Python

  • The script's output:


End of session 2
END of session 2

Thank you

for attending!

Enjoy the rest of

the LSA SAF week


ad