Skip this Video
Download Presentation
Envisat Validation Workshop Atmospheric Chemistry Validation Team

Loading in 2 Seconds...

play fullscreen
1 / 21

Envisat Validation Workshop Atmospheric Chemistry Validation Team - PowerPoint PPT Presentation

  • Uploaded on

Envisat Validation Workshop Atmospheric Chemistry Validation Team Ground-Based Measurements and Campaign Database Subgroup Campaign Database. Terje.Krognes , [email protected] Aasmund.Fahre.Vik, [email protected] Norwegian Institute for Air Research (NILU). Presentation Outline. Data Centre Status

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Envisat Validation Workshop Atmospheric Chemistry Validation Team' - raiden

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Envisat Validation Workshop

Atmospheric Chemistry Validation Team

Ground-Based Measurements and Campaign Database Subgroup

Campaign Database

Terje.Krognes, [email protected]

Aasmund.Fahre.Vik, [email protected]

Norwegian Institute for Air Research (NILU)

presentation outline
  • Data Centre Status
  • Tools
    • Local end-user tools
    • WWW search tools
  • Commissioning Phase Activities
data flow
nadir web








http or ftp


Error email





ftp to zardoz

Data flow
  • Signed protocol required for access to tools, docs and files
  • ECMWF data requires additional protocol
data centre status 1 3 hardware








Data Centre Status (1/3): Hardware
  • The machines have been swapped between production and spare servers
  • The hardware has not been changed
data centre status 2 3 software
Data Centre Status (2/3): Software
  • Web-server (nadir) runs Linux Redhat 7.2 with up2date ~weekly (6.2 at 2nd rehearsal)
  • File server (zardoz) runs Solaris 7, as before
  • Nusphere MySQL
    • Apache 1.3.22 (1.3.14)
    • MySQL 3.23.44 (3.23.32)
    • PHP 4.0.6 (4.0.4pl1)
  • IDL 5.4, as before
  • HDF 4.1r3, as before
  • No re-boot on nadir since April 2002
  • Uptime on zardoz since April 2002 is approximately 99,98% (2 half-hour maintenance + re-boot)
data centre status 3 3 metadata
Data Centre Status (3/3): Metadata
  • Metadata Guidelines updated by metadata board on 24 April 2002
  • During commissioning phase, updates of various metadata table entries, without affecting structure:
    • Project-related information
    • Actual descriptors
  • Guidelines still applicable, but more clarifications to be added.
new user tools
New user tools
  • Major re-write of the ASC2HDF file generation and checking tool around launch
  • 6 smaller upgrades of the ASC2HDF package after launch
  • Entire ASC2HDF source code (FORTRAN) published to users. HDF output routines may be combined with user’s own input and processing modules
  • IDL-tool to generate HDF data files (without error checking): IDL2HDF
  • IDL tool for reading and plotting HDF data files (HDF2IDL), to be downloaded and used locally
  • IDL subroutine to read HDF Cal/Val data, for integration into user-developed software (
new web functionality ecmwf
New web functionality - ECMWF
  • ECMWF plots (NH 2,5º*2,5º and T106)
  • ECMWF 10-day back trajectory plots
  • T106 extractions to HDF files
new web functionality search and plot
New web functionality - search and plot
  • Use of defaults
  • Facility to save and retrieve a search
  • Provision for downloading multiple files in tarball via web interface
  • Improved on-line plotting (also for multidimensional data)
  • Corrected display of comments to variables
file searching 1 4
File searching (1/4)
  • Defaults are marked with ***
  • Page initially opens without defaults
  • Page may be reset or set to saved defaults bys selecting a previously saved query
  • Files may be filtered by a large number of metadata parameters
file searching 2 4
File searching (2/4)
  • The default criteria page lists all criteria sets that this user has previously saved
  • Entries may easily be removed
  • The main search page will be loaded with the selected defaults when launched from this page
  • Defaults may include 4D space-time parameters and variable name wild-cards
file searching 3 4
File searching (3/4)
  • The NUMERICAL SIMULATION entry is now marked with *** since we selected it as default through the saved query page
  • Variable name / mode / descriptor have wildcard entries automatically added to the drop down lists.
  • Variable name and variable mode accept multiple selections
  • 4D space-time criteria may be entered as a single point, or as a max/min box
  • All files that include or touch the 4D criteria are listed as matches
file searching 4 4
File searching (4/4)
  • Files may be selected by posting date
  • Files updated after a given date may be selected
  • The file list may be sorted by file name, by submitter or PI, by posting date, or by AOID
  • If no criteria are selected, the entire file list will be requested
file list
File List
  • The file list displays the current selection criteria
  • A single file may be downloaded immediately
  • Files may be marked and downloaded in a compressed tarball
  • Search criteria may be saved for later re-use (type a meaningful description before saving)
  • The Variables link brings up a variable list and other documentation contained in the file
variable list
Variable List
  • The variable list page displays all parameters contained in a file
  • Variable name, mode and descriptor are displayed
  • The measurement unit and corresponding SI unit are displayed
  • All documentation fields attached to the file are displayed
  • Additional documentation fields attached to a single variable may be displayed through the Comments link
  • Variables may be selected to generate an on-line plot (next slide)
  • Observe the dependencies listed for each variable – not all variable combinations will produce meaningful plots
administrative tools
Administrative tools
  • Web tools to report uploads/downloads
  • Web tools to archive/retrieve outdated data files
  • Web tools to check and maintain index database consistency and integrity
user activities
User activities
  • Users have sent approximately 200 emails with requests for accounts or updates to metadata, or with questions to HDF file generation
  • After launch focus has slowly shifted to requests for help with HDF file formatting issues and removal of outdated files
  • From May to November 2002 users have submitted 5880 HDF data files (570 of these were rejected due to formatting problems, and 238 have later been archived).
  • From May to November 2002 users have downloaded 4000 HDF data files (totalling 215 MB) (ftp downloads via zardoz not included)
  • There are currently 4741 ACVT files open to users (22 active AOIDs)
  • There are currently 462 MAVT files open to users (5 active AOIDs)
user activities up and downloads
User activities: Up and Downloads

The download statistics only covers WWW downloads from nadir, not FTP downloads from zardoz

nadir team activities
Nadir-team activities
  • 300+ interactions with users (of which 50% during Commissioning Phase)
  • 250+ interactions with ESA and the Metadata Board (10% during Commissioning Phase)
  • One major re-write and 6 updates of the ASC2HDF and HDF2ASC programs
  • Maintained and improved shell scripts for automatic file checking and storage
  • Created additional IDL and FORTRAN tools
  • Performed extensive web program development
  • Performed user administration updates
  • Added metadata entries and updated the table.dat file 23 times since launch
  • Developed and maintained scripts and data flow to provide pre-processed ECMWF data extractions and plots on the web
  • Monitored the automatic file reception system (now containing nearly 5000 files totalling 275 MB + index database 20 MB)
metadata issues
Metadata issues

Although many quality checks are performed on the data upon submission to the database:

  • The metadata definition is currently not complete enough to perform automatic checks of the validity of reported measurement units
  • Some components and units may still be inconsistently named or mis-used

Several files have indeed been submitted containing such errors due to misinterpretation of the guidelines. The data submittors will be contacted to correct these files.