1 / 29

GES DISC Services

This document explores the complexities and challenges associated with data quality and usability in the context of the Mirador and AIRS services. It discusses the balance between simplicity and features, the importance of external search capabilities, and the operational status of various data services like the Data Quality Screening Service (DQSS). Users are encouraged to engage with documentation, understand quality implications, and utilize tools efficiently. Emphasizing user experience, the text outlines plans for evolving systems, improving access, and managing data subsets.

fay
Download Presentation

GES DISC Services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GES DISC Services Push Harder? Be Careful? Change Direction? What about adding ______?

  2. Discovery Services • Mirador • Development scaled back to sustaining engineering level • External Search (in Test mode TS1) • Technically successful, but... • Usability-challenged • Start and stop date/time • Total number of hits • Uniform sort order • Duplicates • Usability: Simplicity vs. Features (esp. Services) • Mirador Usability Sounding Board? • mail list for queries on usability quandaries

  3. Data Services

  4. Number of Users* - March 2011 *OK, not really. It’s the number of distinct IP addresses

  5. Number of Users*: Sep 2010 – Apr 2011

  6. Data Quality Screening Service

  7. The quality of AIRS data varies considerably Version 5 Level 2 Standard Retrieval Statistics

  8. Quality Schemes can be complicated Hurricane Ike, viewed by the Atmospheric Infrared Sounder (AIRS) Air Temperatureat 300 mbar PBest : Maximum pressure for which quality value is “Best” in temperature profiles

  9. Current user scenarios... • Nominal scenario • Search for and download data • Locate documentation on handling quality • Read & understand documentation on quality • Write custom routine to filter out bad pixels • Equally likely scenario (especially in user communities not familiar with satellite data) • Search for and download data • Assume that quality has a negligible effect Repeat for each user

  10. The effect of bad qualitydata is often not negligible Hurricane Ike, 9/10/2008 Total Column Precipitable Water Quality Best Good Do Not Use kg/m2

  11. DQSS replaces bad-quality pixels with fill values Original data array (Total column precipitable water) Mask based on user criteria (Quality level < 2) Good quality data pixels retained Output file has the same format and structure as the input file (except for extra mask and original_data fields)

  12. DQSS Demo

  13. DQSS Status + Plans • Operational for AIRS L2 Standard Retrieval • Nearly operational for MODIS Water Vapor • Next: MODIS Aerosols, MLS Water Vapor • Next: ??? • Also, OPeNDAP Gateway nearly reader to front-end DQSS • Allow OPeNDAP access to DQSS-served data.

  14. OPeNDAP* • Remote access to data: no need to download • Access at fine granularity • Variable • Array regions • Stride • Present HDF data as netCDF/CF • Enhances Tool Usability • Reformatting: ASCII, netCDF *OPeNDAP = OpenSource Project for a Network Data Access Protocol

  15. Who Uses OPeNDAP? • Industrial-strength scripters looking for subsets • Thick client users • GrADS, Panoply, IDV, McIDAS-V, Ferret • Internal Systems • Giovanni • MapServer • Simple Subset Wizard

  16. OPeNDAP Demo

  17. OGC* Standards - WMS • Web Map Service (WMS) • URL request: returns map image • Implemented with open-source MapServer • Giovanni also supports WMS • Consumers: • AIRS NRT page • Google Earth • GIS programs • IDV • Giovanni *OGC = Open Geospatial Consortium

  18. OGC - WCS • Returns “coverages”: data variables in NetCDF/CF1 • Used by other systems • DataFed • Giovanni • Atmospheric Composition Portal • Simple Subset Wizard

  19. Subsetting • Semi-custom tools for some products • Reuse HSE libraries from UAH • Reuse Lats4D from A. DaSilva • Usually HDF in -> HDF out • Implemented as REST* URLs • Subsetting at time of download • Subsets are implemented as user requests come in • Areas where we should proactively develop subsetters?

  20. ~100 Subsettable Datasets • AIRS Radiances (channel), L2 Retrievals (variable), L3 (spatial+variable via SSW) • MLS L2 (spatial+variable) • TOMS L3, OMI L2-L3 (spatial+variable), OMI L2 • TRMM L3 (spatial+variable) • Models (spatial+variable) • Did we miss any (that shouldn’t be missed)? • Should all SSW subsets be offered in Mirador?

  21. Format Conversion • Custom code for some L3 and L2 datasets • HDF -> netCDF/CF • Improves usability in tools • Moving toward external tools where possible • OPeNDAP • Lats4d: based on GrADS

  22. Simple Subset Wizard • Desired: “Just give me the data from time 1 to time 2 for this spatial box”. • Current: “search for granules, view granules, select granules, select subset option, re-enter spatial box...” • ESDIS-funded technology infusion effort • DEMO

  23. Giovanni Evolution

  24. G3 Evolution to Agile Giovanni (G4) • Factors driving evolution • G3 architecture was never completed • No workflow engine • Cost of adding significant features is too high • Architecture is too brittle

  25. Key G4 Goals • Reduce cost and time to add new features • Improve performance over G3 • Support external maintenance of external data

  26. Evolution Plan • Implement new projects in Agile Giovanni (G4) • Aerostat ACCESS project • Point data in database, bias corrections • Year of Tropical Convection (YOTC) • Level 2 data • Community-based Giovanni • Externally maintained portals and data • Implement G4 features to meet existing G3 functionality • Migrate G3 instances to G4 portals

  27. Roads Not Taken Not Taken • Giovanni 3 enhancements • ISO 19115 Metadata • Document architecture • Mirador features and usability revamp • Persistent locators • Unique identifiers • Giovanni Evolution • DQSS • Atmospheric Composition Portal • Simple Subset Wizard • Community-based Initiatives • Mirador External Search • Expanding data services

  28. Backup Slides

  29. Agile Giovanni Architectural Features • Model-view-controller • Semantic Web underpinnings • Variable-centric, not dataset-centric • Code reuse: Kepler, YUI, JCache, MapServer

More Related