icoads archive practices at ncar
Download
Skip this Video
Download Presentation
ICOADS Archive Practices at NCAR

Loading in 2 Seconds...

play fullscreen
1 / 30

ICOADS Archive Practices at NCAR - PowerPoint PPT Presentation


  • 95 Views
  • Uploaded on

ICOADS Archive Practices at NCAR. JCOMM ETMC-III 9-12 February 2010 Steven Worley. Topics. Environment setting Data management tools and principles ICOADS NCAR Release 2.5 contributions Background Collections Future Challenges. Environment Setting.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'ICOADS Archive Practices at NCAR' - sumi


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
icoads archive practices at ncar

ICOADS Archive Practices at NCAR

JCOMM ETMC-III

9-12 February 2010

Steven Worley

topics
Topics
  • Environment setting
  • Data management tools and principles
  • ICOADS NCAR Release 2.5 contributions
  • Background Collections
  • Future Challenges
environment setting
Environment Setting
  • ICOADS is part of a larger collection called the Research Data Archive (RDA)
  • RDA – briefly
    • 600+ datasets (atmosphere, ocean, geosciences)
    • 4.3M files, 462 TB (primary data)
    • 6000+ unique users annually, including ICOADS
    • Staff, 7 scientific programmers (M.S. degrees), me, and administrative assistant
data management principles
Data management principles
  • Always archive 2 copies of observational data
    • 3rd copy at a partner center (disaster recovery)
  • Free and open data access world-wide
    • Internet
    • Past – other media, cd-roms, tapes, etc.
  • Share what we have to build archives
    • E.g. Digitization of Maury data in China in exchange for global land surface data
data management tools
Data Management Tools
  • New System: Common RDA tools that homogenize data management.
    • Efficient
    • Scalable
  • Old System: Specialized Software to manage each data input.
    • Inefficient
    • Difficult to Scale

RDA

Metadata

Database

GCMD

Metadata

Server

NWP

Server

RDA Data Server

Online Disk

Specialized Software Package 1

RDA Data Management Common Tool Set

University Server

Specialized Software Package 2

Tape Storage

Specialized Software Package 3

Unidata Server

data management tools a few details
Data Management tools – a few details
  • Common scripting structure to do routine dataset updates (dsupdt)
    • Very tunable
      • Frequency, multiple server priority list, validation
    • Fully integrated with RDADB
      • Users view is automatically update and therefore always current
  • Common single archiving function (dsarch)
      • location and copy control (MSS/HPSS storage, and online disk)
      • Fills all DB entries (e.g. file and dataset relationships)
data management tools1
Data management tools
  • Harvest file level metadata (gatherxml)
    • Handle various formats (GRIB1, GRIB2, netCDF, BUFR, IMMA, ON29, etc.)
    • Save as and populate DB
    • Benefits
      • Problem detection
        • Versioning, replacement, extension
      • Inventory information
      • Drive better data service for users
data management tools2
Data management tools
  • Provide access to data in tape storage archive (dsrqst)
    • Relatively new, not universally available across RDA - yet
    • Delayed mode – with DB control (many details)
    • Why – RDA holds 462 TB
      • 40 TB online – most popular small scale products
      • Access to more products for greater community
icoads release 2 5 contributions @ ncar
ICOADS Release 2.5 contributions @ NCAR
  • Data Preparation – format evaluations, translate native formats to IMMA format
    • Moored research buoy delayed mode archives
      • TOA, PIRATA (PMEL, JAMSTEC)
    • World Ocean Database 2005
      • Multiple ocean profile types (NODC)
  • Receive/archive ICOADS data processing results
    • NOAA/ESRL does processing - source merging, duplicate elimination, preconditioning deletion and fixes, etc.
icoads release 2 5 contributions @ ncar1
ICOADS Release 2.5 contributions @ NCAR
  • Create and maintain user data access interfaces
    • File access
      • IMMA and binary (observations, monthly summary statistics)
    • Sub-selection (time, space, parameter)
      • Example coming.
      • Output is ASCII tabular format
      • Runs automatically – nearly all requests completed in 10 minutes
    • Keep user metrics
icoads release 2 5 contributions @ ncar2
ICOADS Release 2.5 contributions @ NCAR
  • Near-term preliminary extensions to R2.5
    • Beginning with data in 2008 and forward
    • Based on NCEP GTS compilation/merge
    • Runs on day 2 of each month – processes previous month.
    • Create IMMA observations and binary monthly summary statistics
    • Harvest file level metadata
    • Do all archiving of original and processed files
    • Automatically, update user interfaces
background collections
Background Collections
  • Historical
    • Most complete set of ALL source data used to create ALL ICOADS Releases
      • Beginning in mid-1980s
    • Copies of ALL ICOADS Releases
      • We do not delete any files
background collections1
Background Collections
  • Ongoing / Routine data receipts
    • Format conversions are done at NCDC
future challenges
Future Challenges
  • Eliminate user interface dependency on java applets – deploy java script instead.
  • Support “advanced” ICOADS initiative
    • Bias adjusted / corrected observations
    • Serve as a central DB / handle data ingest
    • Build a user interface
  • Continue as a full U.S. partner.
ad