HDF-EOS Maintenance, Current Development and Tools Abe Taaheri, Raytheon Information Systems Larry Klein, RS Information Systems HDF-EOS Workshop X, November 2006
HDF-EOS Support • Maintenance support is funded on EMD Evolution through May 2008 • Current staff - One full time developer for maintaining: SDP/MTD Toolkit, HDF-EOS2 & 5, HDF-EOS plug-in for HDFView, HEG, other Toolkit/HDF-EOS related software/maintenance: - Users Guides - Web sites - User help - Training another developer for HEG maintenance
ECS Support of HDF-EOS • Both HDF4 and HDF5 based flavors of HDF-EOS are part of ECS baseline. • OS's: Solaris (8, 9, 10), Irix6.5, HP 11, AIX, DEC, Windows NT/2000/XP, Linux (including 64-bit Opteron and Itanium), Mac OS X, Cygwin (MTD Toolkit, HDF-EOS2 & 5) • Adding Cygwinsupport for SDP Toolkit • Possibly supporting Intel based Mac for next release • Compilers: Fortran 77/90 & g77/pgf90 , C, C++, gcc, g++ • Support for SGI 7.4.2 besides SGI 7.3.1
Software Releases SDP Toolkit/HDF-EOS: • Last release – March 2006 - TOOLKIT 5.2.14, HDF-EOS 2.14, HDF-EOS5.1.10 - with HDF4.2r1, HDF5-1.6.5-post5, Szip2.0 • Next release – Early 2007 - TOOLKIT 5.2.15, HDF-EOS 2.15, HDF-EOS5.1.11 - with new releases of HDF4, and HDF5 MTD Toolkit:Last Release – Nov. 2006 HDF-EOS Plug-in for HDFView: • Last release – May 2006 • Version 2.3 • Next release – ? • After HDFView2.4 release by The HDF Group possibly in early 2007 HEG: • Last release – March 2006 • Version 2.7 • Next release – Early 2007
Downloadable Applications • HE5View* (HDF5 - based browser) • EOSView* (HDF4 - based browser) • HDFView Plug-in, Java-based browser (HDF4, HDF5, HDF-EOS 2 and 5 access) • heconvert (converts HDF4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents) • Compliments HDF4 -> HDF5 conversion tool • Not tested on all products • HEG (HDF-EOS2 to GeoTIFF converter, subsetting, reprojection, stitching, sub-sampling, etc.) * Dropped support for OS updates since July 2006
SDP/MTD Toolkit Science Data Production (SDP) Toolkit is a collection of tools used by - Science software developers who produce code to process instrument data. - HDF-EOS userswho need to • Access metadata in HDF-EOS files • Perform Time/Date conversion • Access Digital Elevation Model Data • Access Ancillary Data, such as Digital Chart of the World database (DCW), Olson World Ecosystem files, DEM data files, etc.
SDP Toolkit (cont.) • Tools provided in (SDP, MTD) Toolkit: - AA (Ancillary Data Access) - CBP(CELESTIAL Body Position) - CSC (Coordinate System Conversion) - CUC (Constant and Unit Conversions) - DEM (Digital Elevation Model Access) - EPH (Ephemeris Data Access) - GCT (Geo Coordinate Transformation) - IO (Input Output (File I/O)) - MEM (Memory Management) - MET (Metadata Access) - PC (Process Control) - SMF (Status Message File (Error/Status)) - TD (Time Date Conversion)
What is HDF-EOS ? • HDF-EOSis an extension to HDF which standardizes storage of and access to common Earth Sciences data structures • By providing a single interface to data structures common to earth science • By providing a container for EOS inventory, archive and product specific metadata • By establishing a consistent relationship between geolocation and science data.
HDF-EOS 2 • HDF4 - based, storage format for EOS standard products. • Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments • Support for Grid/Point/Swath structures • HDF4 and HDF-EOS 2 will be supported by THG/NASA for the fore-seeable future.
HDF-EOS 5 • Based on HDF5, a complete rewrite of HDF-EOS2 with a different user interface. • First released in 2000. • Format used by EOS Aura instruments • Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. • Supports same data structures • Added prefix ‘HE5_’ to HDF-EOS 2 functions.
HDF-EOS Grid Structure • Usage - Data which is organized by regular geographic spacing, specified by projection parameters. • Structure • Any number of 2-D to 8-D data arrays per structure • Geolocation information contained in projection formula, coupled by structural metadata. • Any number of Grid structures per file allowed.
HDF-EOS Swath Structure • Data which is organized by time, or other track parameter. • Spacing can be irregular. • Structure • Geolocation information stored explicitly in Geolocation Field (2-D array) • Data stored in 2-D or 3-D arrays • Time stored in 1-D or 2-D array, TAI is standard. • Geolocation/science data connected by structural metadata
HDF-EOS Point Structure Time Temp(C) Station Lat Lon 0800 -3 Chicago 41.49 -87.37 0900 -2 Los Angeles 34.03 -118.14 1000 -1 Washington 38.50 -77.00 0800 20 Miami 25.45 -80.11 0900 21 1000 22 1100 24 1000 6 1100 8 1200 9 1300 11 1400 12 060015 0700 16 • Data is specified temporally and/or spatially, but with no particular organization • Structure • Tables used to store science data at a particular Lat/Long/Height • Up to eight levels of data allowed. Structural metadata specifies relationship between levels.
Zonal Average Structure • Generalized array structure with no geolocation linkage • Data can be organized by time or track parameter • Data spacing can be irregular • Structure • Data stored in multidimensional arrays • Time stored in 1-D or 2-D array
Major Development/Fixes Last Release: HDF-EOS5: • POINT object fixes of: - linkage between levels - updating levels - writing Forward and Backward Linkage data - char field inquiry • Added a routine to return list of aliases for a given field • Used HDF5’s H5T_C_S1 datatype for string type • Added a few more Fortran wrappers • Modified symbol “H5E_NONE_MAJOR” to other symbols for better description of errors • Ported to Solaris 9, Solaris 10,and Cygwin • C++ fixes for SGI 7.4.2 compiler support
Major Development/Fixes (cont.) • Fixed subsetting problem in HDF-EOS5 Grid,resulting from faulty conversion of DMS degree to Decimal degree. • Setting LD_LIBRARY_PATH for SZIP/HDF5 sharedlibraries in installation and environment setting scripts TOOLKIT/HDF-EOS2: • Ported to Solaris 9 and Solaris 10 • Keeping more significant digits after the decimal point in projection parameters if needed. • chkeph utility fixes for reading binary ephemeris files (Problem in reading EPH files with different endiannes than the machine ) • SGI 7.4.2 compiler support • orbsim fixes for Linux to create files similar to those in UNIX platforms
Future Development/Fixes Next Release: • Default installation of SZIP • Choice for installing Toolkit/HDF-EOS in 32 or 64 bit modes in 64-bit Linux platforms • New HDF4/HDF5 Release support • Lat/Lon conversions to pixel number in DEM tools in Linux for getting results similar to those in Unix platforms Future Plans: • Support for Cygwin • Support for HDF4.2r2 and HDF5-1.8 • Performance improvement, if possible, for reading HDF format ephemeris/attitude files in Toolkit • Performance improvement in HDF-EOS5 by redesigning some functions
HDFView • HDFView: A Java-based browser from THG for HDF4 and HDF5 – based data. • HDF-EOS plug-in extends HDFView to browse HDF-EOS data in both versions. • The extension is a plug-in module and does not change THG HDFView functionality. All copy, save, editing, image display, etc. for HDF4 and 5 objects remains. • Identical look and feel when displaying HDF and HDF-EOS objects. • Users can access a complete breakdown of HDF-EOS objects. e.g. grid information, projection info., dimensions, datafields, and attributes are displayed. • Current Support: SUN, SGI, LINUX, WINDOWS platforms
HDFView (Cont.) • Release 2.2 October 2005 - A separate HDF-EOS plug-in: Users install HDFView 2.2 and integrates HDF-EOS plug-in into it. - Ability to do "Save as" for HDF-EOS5: modify individual values in grid or swath data. Delete entire grids or swaths. Save the result. - Separate help window for HDF-EOS objects • Release 2.3 January 2006 • Display Point Objects • Scientific notation in Table data • Geolocating pixels for swath and grid images
HDFView (Cont.) • Next Release: Version 2.4 • Automatic installation of plug-in into HDFView running self-extracting archives. • Display image for selected pixels in the viewed tables • Geolocating cells in the data field tables for swath and grid • Mac OSX porting • Better display of Plug-in User’s Guide • New icons for activating Lat/Lon display for Swath & Grid - Future Plans: - Ability to modify input HDF-EOS2 granules. - Ability to cut/paste objects, modify/delete attributes. - Ability to convert HDF-EOS2 to HDF-EOS5 - Ability to create new grids/swaths
AMSR-E Swath New Features in Next Release Version 2.4
HDF-EOS to GeoTIFF Converter (HEG) HEG: Utility that converts EOSDIS data from HDF-EOS format to common Geographical Information System (GIS)-compatible formats. • Two versions available: • A downloadable desktop version • Access through NASA archive online storage (Data Pools) • Implemented also in prototypes for Stand-alone Data Pool, OGC (Open Geospatial Consortium) for Web Services Chaining, and DOWS (Deploy OGC Web Services) on the DAACs
HEG Functionality • Conversion: • HDF-EOS to a single-band or multi-band GeoTIFF’s • HDF-EOS Swath to HDF-EOS Grid and Swath • HDF-EOS Swath or Grid to GeoTIFF • HDF-EOS Swath or Grid to generic Binary (with metadata file) • Subsetting (spatial, field, band) • Stitching (mosaicing) + subsetting + Reprojecting • Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ, GEO, SIN) • Subsampling (Subsample stacks) • Running from Command Line (batch jobs) or with Java-based GUI
HEG Functionality • Metadata preservation/creation. • Currently supports MODIS, MISR, ASTER, AIRS and AMSR-E products on TERRA and AQUA (>110 products), • Allows file selection from users local storage • Operable on Sun, SGI, Win, Linux, MAC • Integrated into ECS Data Pool • Reduces the transfer time of HDF-EOS data sets (if subset is requested) • Provides the end-user with the exact file required by their application • User Interface • Portable, written in Java. • Not dependent on COTS (eg. IDL).
HEG Releases • Support for: AMSR-E L3 GRID products (in Polar Stereographic and EASE-Grid) AMSR-E L3 GRID products (in Geographic projection) MISR L3 products (in Geographic projection). AMSR-E L2 SWATH products AIRS L3 products (in Geographic projection) ASTER DEM products: Correction for missing data in the internal geolocation fields of the ASTER DEMs. MOD43C1, MOD43C2, MOD43C3, MCD43C1, MCD43C2, MCD43C3, MOD12C1, MYD12C1: Corner points of GRID in DD instead of DMS units. MOD11_L2: fillvalues (-999.0) in the Lat/Lon geolocations while (-999.9) in internal metadata. • Added Albers Projection • New "Subsetter" functionality for Parameters (aka Fields). This results in: • Output to HDF-EOS SWATH from input HDF-EOS SWATH • No alteration of input data because of projection operation.
MODIS L1B: La Plata Tornado Reprojected, Co-Registered with ASTER
2004 Hurricane Francis, MODIS MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5
AST_L1B, 2003 California FiresGeo. projection, Multi-Band GeoTIFF, (3 bands: Visibile Near Infrared (VNIR)) ….. and Subsetted Two Swaths Stitched …..
MODIS Land Products: Reprojected, Co-registered, Processed in ArcInfo/ArcMap Vegetation Index Leaf Area Index Surface Reflectance
Availability • Access to libraries and applications: TOOLKIT: http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html HDFView: http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/HDFView_hdfeos_plugin.html HEG: http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html • Email • Abe_Taaheri@raytheon.com • Landover_PGSTLKIT@raytheon.com