HDF-EOS APIs, tools, etc.

1. HDF-EOS APIs, tools, etc. Larry Klein, Abe Taaheri, and Cid Praderas L-3 Communications Government Services, Inc. November 30, 2005

2. What is HDF-EOS ? • HDF (Hierarchical Data Format) is the format chosen by NASA as the standard for the EOS program • HDF-EOS is an extension to HDF which standardizes storage of and access to common Earth Sciences data structures • By providing a single interface to data structures common to earth science • By providing a container for EOS inventory, archive and product specific metadata • By establishing a consistent relationship between geolocation and science data.

3. HDF-EOS Grid Structure • Usage - Data which is organized by regular geographic spacing, specified by projection parameters. • Structure • Any number of 2-D to 8-D data arrays per structure, one per data type (e.g. temperature) • Geolocation information contained in projection formula, coupled by structural metadata. • Any number of Grid structures per file allowed.

4. HDF-EOS Swath Structure • Data which is organized by time, or other track parameter. • Spacing can be irregular. • Structure • Geolocation information stored explicitly in Geolocation Field (2-D array) • Data stored in 2-D or 3-D arrays • Time stored in 1-D or 2-D array, TAI is standard. • Geolocation/science data connected by structural metadata

5. Swath Data Example

6. HDF-EOS Point Structure • Data is specified temporally and/or spatially, but with no particular organization • Structure • Tables used to store science data at a particular Lat/Long/Height • Up to eight levels of data allowed. Structural metadata specifies relationship between levels.

7. Zonal Average Structure • Generalized array structure with no geolocation linkage • Data can be organized by time or track parameter • Data spacing can be irregular • Structure • Data stored in multidimensional arrays • Time stored in 1-D or 2-D array

8. HDF-EOS 2 • HDF4 - based, storage format for EOS standard products. • Used operationally by MODIS, MISR, ASTER, Landsat, AIRS and other EOS instruments • Support for Grid/Point/Swath structures • HDF4 and HDF-EOS 2 will be supported by NCSA/NASA for the fore-seeable future.

9. HDF-EOS 5 • Based on HDF5, a complete rewrite of HDF4 with a different user interface. • First released in 2000. • Format used by EOS Aura instruments • Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. • Supports same data structures • Added prefix ‘HE5_’ to HDF-EOS 2 functions. • Doesn’t preclude HDF5 functionality.

10. Top Level of HDF-EOS 5 File Root -- “/” HDFEOS INFORMATION HDFEOS ECS METADATA ADDITIONAL STRUC. METADATA ARRAY SWATH GRID POINT Global (File) Attributes

11. Basic HDF-EOS Functions • Access - Open, create, close objects/files • Definition - Set key parameters of data sets • Input/Output - Read/Write data • Inquiry - Return information about data • Subset - Read data from specified region

12. File/Object Access GRID: • HE5_GDopen/he5_gdopen • HE5_GDcreate/he5_gdcreate • HE5_GDattach/he5_gdattach • HE5_GDdetach/he5_gddetach • HE5_GDclose/he5_gdclose

13. File/Object Access POINT: • HE5_PTopen/he5_ptopen • HE5_PTcreate/he5_ptcreate • HE5_PTattach/he5_ptattach • HE5_PTdetach/he5_ptdetach • HE5_PTclose/he5_ptclose

14. File/Object Access SWATH: • HE5_SWopen/he5_swopen • HE5_SWcreate/he5_swcreate • HE5_SWattach/he5_swattach • HE5_SWdetach/he5_swdetach • HE5_SWclose/he5_swclose

15. File/Object Access • ZA: • HE5_ZAopen/he5_zaopen • HE5_ZAcreate/he5_zacreate • HE5_ZAattach/he5_zaattach • HE5_ZAdetach/he5_zadetach • HE5_ZAclose/he5_zaclose

16. Definition GRID: • HE5_GDdefdim/he5_gddefdim • HE5_GDdefproj/he5_gddefproj • HE5_GDdeforigin/he5_gddeforigin • HE5_GDdefpixreg/he5_gd_defpreg

17. Definition POINT: • HE5_PTdeflevel/he5_ptdeflevel • HE5_PTdeflinkage/he5_ptdeflinkage

18. Definition SWATH: • HE5_SWdefdim/he5_swdefdim • HE5_SWdefdimmap/he5_swdefmap • HE5_SWdefidxmap/he5_swdefimap • HE5_SWdefgeofield/he5_swdefgfld • HE5_SWdefdatafield/he5_swdefdfld

19. Definition • ZA: • HE5_ZAdefdim/he5_zadefdim • HE5_ZAdefine/he5_zadefine • HE5_ZAdefcomp/he5_zadefcomp • HE5_ZAdefchunk/he5_zadefchunk

20. Input/Output GRID: • HE5_GDwritefield/he5_gdwrfld • HE5_GDreadfield/he5_gdrdfld • HE5_GDwriteattr/he5_gdwrattr • HE5_GDreadattr/he5_gdrdattr

21. Input/Output POINT: • HE5_PTwritelevel/he5_ptwritelevel • HE5_PTreadlevel/he5_ptreadlevel • HE5_PTwriteattr/he5_ptwrattr • HE5_PTreadattr/he5_ptrdattr

22. Input/Output SWATH: • HE5_SWwritefield/he5_swwrfld • HE5_SWreadfield/he5_swrdfld • HE5_SWwriteattr/he5_swwrattr • HE5_SWreadattr/he5_swrdattr

23. Input/Output • ZA: • HE5_ZAwrite/he5_zawrite • HE5_ZAread/he5_zaread • HE5_ZAwriteattr/he5_zawrattr • HE5_ZAreadattr/he5_zardattr

24. Inquiry GRID: • HE5_GDinqdims/he5_gdinqdims • HE5_GDdiminfo/he5_gddiminfo • HE5_GDprojinfo/he5_gdprojinfo • HE5_GDorigininfo/he5_gdorigininfo • HE5_GDpixreginfo/he5_gdpreginfo

25. Inquiry POINT: • HE5_PTlevelinfo/he5_ptlevelinfo • HE5_PTinqattrs/he5_ptinqattrs • HE5_PTattrinfo/he5_ptattrinfo

26. Inquiry SWATH: • HE5_SWinqdatafields/he5_swinwdflds • HE5_SWinqgeofields/he5_swinqgflds • HE5_SWinqattrs/he5_swinqattrs • HE5_SWattrinfo/he5_swattrinfo

27. Inquiry • ZA: • HE5_ZAinquire/he5_zainquire • HE5_ZAinfo/he5_zainfo • HE5_ZAinqattrs/he5_zainqattrs • HE5_ZAattrinfo/he5_zaattrinfo

28. Programming Model • Open the HDF-EOS5 file • Create or Attach to a specified Grid/Swath/Point/ZA object • Define key object features and/or perform necessary operations (with the data sets, attributes, etc) • Detach from the Grid/Swath/Point/ZA object • Close the HDF-EOS5 file

29. HDF-EOS Functionality • Basic File I/O • Swath Interface • Grid Interface • Point Interface • Profile Interface • Generalized Array Interface (Zonal Avg. (HDF5)) • Subsetting • Fill Values • Compression (gzip, szip, shuffling) • Chunking/Tiling • Aliases • Global (File), Group & Local Attributes • Threadsafe Version

30. Applications • HE5View (HDF5 - based browser) • EOSView (HDF4 - based browser) • HDFView (with HDF-EOS Plug-in), Java EOS Browser (HDF-EOS 2 and 5 access) • heconvert (converts HDF4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents) • Compliments HDF4 -> HDF5 conversion tool • Not tested on all products • HEG (HDF-EOS to GeoTIFF Converter/Subsetter)

31. HDFView • HDFView: A Java-based browser from NCSA for HDF4 and HDF5 – based data. • HDF-EOS plug-in extends HDFView to browse HDF-EOS data in both versions. • One reason: EOSView is becoming obsolete • The extension is a plug-in module and does not change NCSA HDFView functionality. All copy, save, editing, image display, etc. for HDF4 and 5 objects remains. • Identical look and feel when displaying HDF and HDF-EOS objects. • Users can access a complete breakdown of HDF-EOS objects. e.g. grid information, projection information, dimensions, datafields, and attributes are displayed. 

32. HDFView (Cont.) • Release 2.2 October 2005 - A separate HDF-EOS plug-in: Users install HDFView 2.2 and integrates HDF-EOS plug-in into it. - Ability to do "Save as" for HDF-EOS5: modify individual values in grid or swath data. Delete entire grids or swaths. Save the result. - Separate help window for HDF-EOS objects • Release 2.3 January 2006 - Point Object viewing capability - Geolocating pixels for swath and grid images • Future Plans: - Ability to modify input HDF-EOS2 granules. - Ability to cut/paste objects, modify/delete attributes. - Ability to create new grids/swaths

33. HDFView showing ASTER L1B swath

34. HDF-EOS to GeoTIFF Converter (HEG) • A utility that converts EOSDIS data from HDF-EOS format to common Geographical Information System (GIS)-compatible formats. • Two versions available: • A downloadable desktop version • Access through NASA archive online storage (Data Pools) • Implemented also in prototypes for Stand-alone Data Pool, OGC (Open Geospatial Consortium) for Web Services Chaining, and DOWS (Deploy OGC Web Services) on the DAACs • Details of internal file structures are transparent to users.

35. HEG Functionality • Subsetting (spatial, field, band) • Performs conversion to a single-band or multi-band GeoTIFF’s • HDF-EOS Swath to HDF-EOS Grid and Swath • HDF-EOS Swath or Grid to GeoTIFF • HDF-EOS Swath or Grid to generic Binary (with metadata file) • Stitching (mosaicing) + subsetting + Reprojecting • Running from Command Line (batch jobs) or with Java-based GUI • Reprojection ( to UTM, PS, TM, STP, LCC, LAMAZ, GEO, SIN) • Subsampling (Subsample stacks)

36. HEG Functionality • Metadata preservation/creation. • Currently supports MODIS, MISR, ASTER, AIRS and AMSR-E products on TERRA and AQUA (>110 products), AURA/HDF-EOS5 support in progress • Allows file selection from users local storage • Operable on Sun, SGI, Win, Linux (64-bit soon), MAC • Integrated into ECS Data Pool • Reduces the transfer time of HDF-EOS data sets (if subset is requested) • Provides the end-user with the exact file required by their application • User Interface • Portable, written in Java. • Not dependent on COTS (eg. IDL).

37. Availability • Access to libraries and applications: http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html • Email • larry.klein@l-3com.com • Abe_Taaheri-NR@raytheon.com • Cid_Praderas-NR@raytheon.com • Landover_PGSTLKIT@raytheon.com

38. Mt. Etna Eruption: MISR L1B Swaths: Stitched

39. Mt. Etna: Stitched, Subsetted ASTER L1B

40. MODIS L1B: Reprojected, Co-Registered with ASTER, La Plata Tornado

41. La Plata Tornado Damage: ASTER L1B reprojected/stitched/subsetted

42. MLS L2GP-H2O Product (2004, day 243), Geographic Projection, Full Globe

43. MLS L2GP-H2O Product, PS Projection, Northern Hemisphere

44. 2004 Hurricane Francis, MODIS (MOD021KM, Multi-Band GeoTIFF, 3 bands: EV_1KM_RefSB, Bands 1,2,5)

45. AST_L1B, 2003 California FiresGeo. projection, Multi-Band GeoTIFF, (3 bands: Visibile Near Infrared (VNIR)) ….. and Subsetted Two Swaths Stitched …..