1 / 10

GENI I&M Workshop NetCDF and Local Data Manager (LDM) Mike Zink

GENI I&M Workshop NetCDF and Local Data Manager (LDM) Mike Zink. November 4, 2010 http://geni.cs.umass.edu/vise http://geni.cs.umass.edu/dicloud http://www.geni.net. Overview. What is NetCDF.

fallon
Download Presentation

GENI I&M Workshop NetCDF and Local Data Manager (LDM) Mike Zink

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GENI I&M WorkshopNetCDF and Local Data Manager (LDM)Mike Zink November 4, 2010 http://geni.cs.umass.edu/vise http://geni.cs.umass.edu/dicloud http://www.geni.net

  2. Overview

  3. What is NetCDF • NetCDF is a set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. • Uses HDF5 as data storage layer. • Also provides read-only access to some HDF4, HDF5 archives. • Parallel I/O for high performance computing. • Developed and maintained by Unidata at UCAR

  4. Who uses NetCDF? • NetCDF is widely used in University Earth Science community. • Used for Intergovernmental Panel on Climate Change (IPCC) data sets. • Used by NASA and other large data producers. • Used for NEXRAD and CASA radar data (moments)

  5. NetCDF Example     float Reflectivity(Radial, Gate) ;         Reflectivity:Units = "dBz" ;     float Velocity(Radial, Gate) ;         Velocity:Units = "MetersPerSecond" ;     float SpectralWidth(Radial, Gate) ;         SpectralWidth:Units = "MetersPerSecond" ;     float DifferentialReflectivity(Radial, Gate) ;         DifferentialReflectivity:Units = "dB" ;     float DifferentialPhase(Radial, Gate) ;         DifferentialPhase:Units = "Degrees" ;     float CrossPolCorrelation(Radial, Gate) ;         CrossPolCorrelation:Units = "Unitless" ;     float NormalizedCoherentPower(Radial, Gate) ;         NormalizedCoherentPower:Units = "Unitless" ;     float SpecificPhase(Radial, Gate) ;         SpecificPhase:Units = "DegreePerKm" ;     float HPropagationPhase(Radial, Gate) ;         HPropagationPhase:Units = "Radians" ;     float VPropagationPhase(Radial, Gate) ;         VPropagationPhase:Units = "Radians" ;     int GateFlags(Radial, Gate) ;         GateFlags:Units = "BitField" ;     float CorrectedReflectivity(Radial, Gate) ;         CorrectedReflectivity:Units = "dBZ" ;     float CorrectedDifferentialReflectivity(Radial, Gate) ;         CorrectedDifferentialReflectivity:Units = "dB" ; 5

  6. OpeNDAP Client for NetCDF • OPenDAP (http://www.opendap.org/) is a widely supported protocol for access to remote data • Defined and maintained by the OPenDAP organization • Designed to serve as intermediate format for accessing a wide variety of data sources. • Client is now built into netCDF C library.

  7. Local Data Manager (LDM) • LDM software acquires data and shares them with other nodes • Data product is treated as a opaque unit, thus nearly any data can be relayed • LDM can handle data from: • National Weather Service "NOAAport channel 3" data streams • NEXRAD radar data • lightning data from the National Lightning Detection Network • GOES satellite imagery

  8. LDM Features • User configurable • execute an arbitrary program with the data product as input • Supports distributed processing • Data ingest functions can be separated from storage and use functions • LDM servers may be configured to relay subsets of data among themselves, distributing the storage and processing around the network. • Extensible • New decoders can be added easily • Event-driven • Handles data as it arrives

  9. Demo Data Flow archived netcdf data aggregated multi-radar data Nowcast images for display • Dynamic end-to-end Nowcasting • Mapping Nowcast Workflows onto GENI “raw” live data Radar Nodes Archival Storage Upstream LDM feed Nowcast Processing Post to Web NetCDF over LDM LDM server NetCDF over LDM LDM server NetCDF over LDM LDM server Storage Nowcasting Merge

  10. Generate “raw” live data ViSE/CASA radar nodes http://stb.ece.uprm.edu/current.jsp Ingest mulit-radar data feeds Merge and grid multi-radar data Generate 1min, 5min, and 10min Nowcasts Send results over NLR to Umass Repeat Use proxy to track usage-based spending on Amazon and enforce quotas and limits http://geni.cs.umass.edu/vise/dicloud.php ViSE views steerable radars as shared, virtualized resources http://geni.cs.umass.edu/vise “raw” live data Nowcast images for display DiCloud Archival Service (S3) LDM Data Feed (EC2) Multi-radar NetCDF Data Nowcast Processing

More Related