granule metadata management l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Granule Metadata Management PowerPoint Presentation
Download Presentation
Granule Metadata Management

Loading in 2 Seconds...

play fullscreen
1 / 19

Granule Metadata Management - PowerPoint PPT Presentation


  • 90 Views
  • Uploaded on

Granule Metadata Management. ted.habermann@noaa.gov mark.ohrenschall@noaa.gov. NCDC (MI3), NWS (MIRS), and others in all NOAA Lines. CLASS: File System Rich Inventory. NESDIS Metadata. CoRIS, NMMR, MERMAid, NMFS. NVDS, CLASS, CoRIS, NMMR, MERMAid NOS: Data Explorer NOAA Library

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

Granule Metadata Management


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
granule metadata management

Granule Metadata Management

ted.habermann@noaa.gov

mark.ohrenschall@noaa.gov

nesdis metadata

NCDC (MI3), NWS (MIRS), and others

in all NOAA Lines

CLASS:

File System

Rich Inventory

NESDIS Metadata

CoRIS, NMMR, MERMAid, NMFS

NVDS, CLASS, CoRIS, NMMR, MERMAid

NOS: Data Explorer

NOAA Library

NOAAServer(??)

types of granule metadata
Types of Granule Metadata

Granules are the smallest items that can be retrieved from an archive without invoking a subsetting operation. They are usually files. The metadata that describes characteristics of these files is termed Granule Metadata. In NPOESS Land this is called Dynamic Metadata. These metadata tend to be closely related to the data in the granules and many times are not consistent with national or international standards (which is fine).

  • There are several types of granule metadata:
  • File Names (Product Type, Date)
  • File Headers (format information, statistics, quality information)
  • Descriptive Statistics from parameters in the granule
present inventory
Present Inventory
  • Files come to archive and filename metadata is ingested into inventory.
  • Fileheader metadata is stored and is not available to data discovery system.
  • Descriptive Statistics are not calculated.
  • Users need to develop their own data discovery systems.
rich inventory
Rich Inventory
  • Files come to Archive
  • Filename and fileheader metadata are added to inventory.
  • Descriptive Statistics are calculated and added to inventory.
  • All metadata is available to the data discovery system and users get the data they need without secondary data discovery.
types of metadata
Types of Metadata

File headers include constants (format information), systematics (dates) and variables.

Parameters include constants (land masks, spatial covariances) and variables (science parameters). In the aerosol case 30% of the grids presently archived with each granule are constants.

All of these metadata types can be described as a series of time segments with constant values. The shortest time segment may be one file interval long.

examples
Examples

Systematic Header Parameters (dates)

Variable Parameters

(aerosol thickness mean and standard deviation)

segment model
Segment Model

Constant (Static)

Slow Variation (Quasi-static)

Fast Variation (Dynamic)

Time (File Number)

metadata ingest
Metadata Ingest

File

raw values

sum(x),

sum(x ),

mean, std,

count

2

Create

segment

yes

New

value?

Add to last

segment

no

current status prototype
Current Status - Prototype
  • Prototype Implementations:
  • NESDIS Sea Surface Temperatures and Atmospheric Aerosols
  • AMSU-B and ATOVS Vertical Statistics
  • NOAA Observing Systems
  • Pathfinder High Resolution Sea Surface Temperature
  • AVHRR GAC level 1b Orbital Data
monitoring and graphics
Monitoring and Graphics

CWOP

ENSO

GOES

GPS

Other

NOAA Observing

Systems

Simple Observing System Characteristics (num, min/max lon) recorded after each update then monitored

Regular Automatic Updates

parameter series
Parameter Series

The Parameter Series table holds information about granule metadata values that are constant across multiple granules. This decreases the size of the database significantly (~90%) in some cases and improves performance.

Tools now exist to populate the table retroactively or on ingest.

java api documentation
Java API Documentation

http://gdsg.ngdc.noaa.gov/projects/richInventory/apidocs/

tiered rich inventory
Tiered Rich Inventory

...

...

Level 3

SST Rich Inventory

GVI Rich Inventory

Aerosol Rich Inventory

...

...

Level 2

L1B Rich Inventory

distributed ingest
Distributed Ingest
  • Product generation algorithms write all metadata to inventory directly instead of file headers.
  • Files are archived somewhere with pointers from Inventory.
  • Users get the data they need from distributed system without secondary data discovery.