1 / 21

An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format

An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format. Ken Stone, Center for Lower Atmospheric Studies - University of Colorado, Boulder with help from. Acknowledgements. Larry L. Gordley - GATS, Inc., Newport News, Virginia

uttara
Download Presentation

An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format Ken Stone, Center for Lower Atmospheric Studies - University of Colorado, Boulder with help from...

  2. Acknowledgements • Larry L. Gordley - GATS, Inc., Newport News, Virginia • The HALogen Occultation Experiment (HALOE) Team • Cheryl Craig, NCAR • Joe McInerney, University of Colorado

  3. Overview • Overview of the toolkit • History • Motivation • Some examples • HDF-EOS extensions • Problems & Limitations • Strengths • Lessons learned • Summary

  4. Motivation • Problem: It takes considerable time and effort to make the many multiple comparisons, internal and external, needed during validation phase. • Goal: A simple set of tools that allow validation staff to select, line-up, compare and plot profiles and cross sections. • Latest twist: HDF-EOS files are now the archive format of choice.

  5. History • LIMS - Early 80’s • Fortran routine for comparisons and plotting • Fast, low memory usage • Input: multiple files • Output: text summaries and plots • HALOE - Early 90’s • Perl Scripts and Standard I/O file format (bsel format) • Fortran routine ported to IDL • New functionality and visualization tools. • HIRDLS • HDF-EOS Swath capability • Additional Tools (dot-plots, map differences)

  6. Toolkit Overview • IDL procedures with Perl scripts for control • Complex scenarios developed both with interactive use (storing results in “journal” file) and stringing scripts together with Unix shell. • Analytical Visualization tools • Mapping, cross sections, line plots. • Focus on quality and simplicity rather than complex visualizations. • Analysis tools • Coincidence finder, statistical computations. • Standardized file format(s) for I/O • “Profile” based format hinges on vectors of geo-referenced quantities.

  7. Toolkit Detail • Prior to HDF-EOS extension the Toolkit used a single file type for input (BSEL format) • Similar in some ways to HDF 4 SDS but limited to sets of 1-d arrays stored as records. • User input for any given IDL scenario can be recorded in a journal (text) file for later use. • Perl “wrapper” scripts create this file as output. • Keyword substitutions in a “mask” file. • Depending on the mode (mood?) the user is in, they could modify the journal file directly, or change the inputs on the script.

  8. Toolkit Script Methodology User Input: -f filename -I id, -o outputfile -g -h, etc Mask File Perl Script (e.g., 2dlat) Perl helper lib Input File Postscript IDL procedure or C, or Fortran... Journal file creation from IDL scripts BSEL File Analysis routines

  9. Toolkit Data Flow: Archive and Working Formats Archive D a t a S o u r c e Selection Analysis Display “working” data format Bsel HDF-EOS “Standard” Data Format Toolkit Extension

  10. Toolkit Data Model (BSEL Format) BSEL File . . . Event (1) Event (i) Event Header Real(8) Long(8) char*60(6) Plot Info Range, axis Label, legend Label, Scale, Logflag Data Header Data order (=2) J (Profile count) K (levels/event) Data (j,k)

  11. HIRDLS HDF-EOS Swath HDF-EOS Swath File Dimensions: nTimes, nLevels etc. Metadata Global Attributes Inst. Name, Ranges, Labels, Scales, Missing Values, Profile count Geolocation Fields Data Fields Time Pressure Ozone Mixing Ratio nLevels nTimes

  12. HDF-EOS Extension Strengths • Fast! Ten times faster than native (Bsel) format • Easier to identify field names (and more flexible) • Standard way of identifying missing data • IDL/Fortran/C I/O interfaces • Potentially more tools available

  13. HDF-EOS Extension Difficulties • Lots of different ways to store data and still meet the standard. • Aura standard format guidelines helps here. • HDF-EOS Swath isn’t “packet-like” • You can’t just “cat” files together! • IDL/Fortran/C I/O interfaces • Some trouble with stability of certain I/Fs. • Point Interface (and raw Vdata) failed from the start due to lack of “id” resources.

  14. Lessons Learned • Developed general purpose swath reader in IDL • usage: data = get_sw_data(filename, swathname, varname) • Generally reads any variable or attribute. • http://www.eos.ucar.edu/hirdls/get_sw_data.pro • Working out the best way to “easily select” variables from swaths is a work-in-progress. • To map HDF to the internal data structure a small external data table was needed...

  15. Mapping Table HDF-EOS - Internal SwathName VarName Label ID Min Max Log Scale Unit HIRDLS H2O H2O Mixing Ratio 2 0.0 10.0 0 1.0e-06 ppmv HIRDLS NO2 NO2 Mixing Ratio 3 0.0 10.0 0 1.0e-09 ppbv HIRDLS O3 O3 Mixing Ratio 4 0.0 10.0 0 1.0e-06 ppmv HIRDLS CH4 CH4 Mixing Ratio 6 0.0 2.0 0 1.0e-06 ppmv HIRDLS CFCl3 CFCl3 Mixing Ratio 151 0.0 0.5 0 1.0e-09 ppbv CH4 L2gpValue CH4 Mixing Ratio 6 0.0 2.0 0 1.0e-06 ppmv O3 L2gpValue O3 Mixing Ratio 4 0.0 10.0 0 1.0e-06 ppmv Internal to toolkit Internal to HDF-EOS Designations for MLS Files

  16. Summary • HDF-EOS Swath format provides a good storage facility for atmospheric profile data. • Current swath implementation (a la Aura) provides solid groundwork for future sophisticated software development. • Interface instability can contribute to major headaches. • Ancillary tools to “view and edit” or simply compute additional fields from existing ones could prove quite useful.

  17. Future • Extend data base table for mapping fields. • Develop interface for Grid format. • Already tested in limited cases. • Testing, testing, testing… • Perhaps incorporate new IDL features • Improved color contouring • Develop an easier to use interface for the scripting tools.

More Related