Envisat Validation Workshop
1 / 21

Envisat Validation Workshop Atmospheric Chemistry Validation Team - PowerPoint PPT Presentation

  • Uploaded on

Envisat Validation Workshop Atmospheric Chemistry Validation Team Ground-Based Measurements and Campaign Database Subgroup Campaign Database. Terje.Krognes , [email protected] Aasmund.Fahre.Vik, [email protected] Norwegian Institute for Air Research (NILU). Presentation Outline. Data Centre Status

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Envisat Validation Workshop Atmospheric Chemistry Validation Team' - raiden

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Envisat Validation Workshop

Atmospheric Chemistry Validation Team

Ground-Based Measurements and Campaign Database Subgroup

Campaign Database

Terje.Krognes, [email protected]

Aasmund.Fahre.Vik, [email protected]

Norwegian Institute for Air Research (NILU)

Presentation outline

  • Data Centre Status

  • Tools

    • Local end-user tools

    • WWW search tools

  • Commissioning Phase Activities

Data flow

nadir web








http or ftp


Error email





ftp to zardoz

Data flow

  • Signed protocol required for access to tools, docs and files

  • ECMWF data requires additional protocol

Data centre status 1 3 hardware









Data Centre Status (1/3): Hardware

  • The machines have been swapped between production and spare servers

  • The hardware has not been changed

Data centre status 2 3 software
Data Centre Status (2/3): Software

  • Web-server (nadir) runs Linux Redhat 7.2 with up2date ~weekly (6.2 at 2nd rehearsal)

  • File server (zardoz) runs Solaris 7, as before

  • Nusphere MySQL

    • Apache 1.3.22 (1.3.14)

    • MySQL 3.23.44 (3.23.32)

    • PHP 4.0.6 (4.0.4pl1)

  • IDL 5.4, as before

  • HDF 4.1r3, as before

  • No re-boot on nadir since April 2002

  • Uptime on zardoz since April 2002 is approximately 99,98% (2 half-hour maintenance + re-boot)

Data centre status 3 3 metadata
Data Centre Status (3/3): Metadata

  • Metadata Guidelines updated by metadata board on 24 April 2002

  • During commissioning phase, updates of various metadata table entries, without affecting structure:

    • Project-related information

    • Actual descriptors

  • Guidelines still applicable, but more clarifications to be added.

New user tools
New user tools

  • Major re-write of the ASC2HDF file generation and checking tool around launch

  • 6 smaller upgrades of the ASC2HDF package after launch

  • Entire ASC2HDF source code (FORTRAN) published to users. HDF output routines may be combined with user’s own input and processing modules

  • IDL-tool to generate HDF data files (without error checking): IDL2HDF

  • IDL tool for reading and plotting HDF data files (HDF2IDL), to be downloaded and used locally

  • IDL subroutine to read HDF Cal/Val data, for integration into user-developed software (nilu_hdf_read_user.pro)

New web functionality ecmwf
New web functionality - ECMWF

  • ECMWF plots (NH 2,5º*2,5º and T106)

  • ECMWF 10-day back trajectory plots

  • T106 extractions to HDF files

New web functionality search and plot
New web functionality - search and plot

  • Use of defaults

  • Facility to save and retrieve a search

  • Provision for downloading multiple files in tarball via web interface

  • Improved on-line plotting (also for multidimensional data)

  • Corrected display of comments to variables

File searching 1 4
File searching (1/4)

  • Defaults are marked with ***

  • Page initially opens without defaults

  • Page may be reset or set to saved defaults bys selecting a previously saved query

  • Files may be filtered by a large number of metadata parameters

File searching 2 4
File searching (2/4)

  • The default criteria page lists all criteria sets that this user has previously saved

  • Entries may easily be removed

  • The main search page will be loaded with the selected defaults when launched from this page

  • Defaults may include 4D space-time parameters and variable name wild-cards

File searching 3 4
File searching (3/4)

  • The NUMERICAL SIMULATION entry is now marked with *** since we selected it as default through the saved query page

  • Variable name / mode / descriptor have wildcard entries automatically added to the drop down lists.

  • Variable name and variable mode accept multiple selections

  • 4D space-time criteria may be entered as a single point, or as a max/min box

  • All files that include or touch the 4D criteria are listed as matches

File searching 4 4
File searching (4/4)

  • Files may be selected by posting date

  • Files updated after a given date may be selected

  • The file list may be sorted by file name, by submitter or PI, by posting date, or by AOID

  • If no criteria are selected, the entire file list will be requested

File list
File List

  • The file list displays the current selection criteria

  • A single file may be downloaded immediately

  • Files may be marked and downloaded in a compressed tarball

  • Search criteria may be saved for later re-use (type a meaningful description before saving)

  • The Variables link brings up a variable list and other documentation contained in the file

Variable list
Variable List

  • The variable list page displays all parameters contained in a file

  • Variable name, mode and descriptor are displayed

  • The measurement unit and corresponding SI unit are displayed

  • All documentation fields attached to the file are displayed

  • Additional documentation fields attached to a single variable may be displayed through the Comments link

  • Variables may be selected to generate an on-line plot (next slide)

  • Observe the dependencies listed for each variable – not all variable combinations will produce meaningful plots

Administrative tools
Administrative tools

  • Web tools to report uploads/downloads

  • Web tools to archive/retrieve outdated data files

  • Web tools to check and maintain index database consistency and integrity

User activities
User activities

  • Users have sent approximately 200 emails with requests for accounts or updates to metadata, or with questions to HDF file generation

  • After launch focus has slowly shifted to requests for help with HDF file formatting issues and removal of outdated files

  • From May to November 2002 users have submitted 5880 HDF data files (570 of these were rejected due to formatting problems, and 238 have later been archived).

  • From May to November 2002 users have downloaded 4000 HDF data files (totalling 215 MB) (ftp downloads via zardoz not included)

  • There are currently 4741 ACVT files open to users (22 active AOIDs)

  • There are currently 462 MAVT files open to users (5 active AOIDs)

User activities up and downloads
User activities: Up and Downloads

The download statistics only covers WWW downloads from nadir, not FTP downloads from zardoz

Nadir team activities
Nadir-team activities

  • 300+ interactions with users (of which 50% during Commissioning Phase)

  • 250+ interactions with ESA and the Metadata Board (10% during Commissioning Phase)

  • One major re-write and 6 updates of the ASC2HDF and HDF2ASC programs

  • Maintained and improved shell scripts for automatic file checking and storage

  • Created additional IDL and FORTRAN tools

  • Performed extensive web program development

  • Performed user administration updates

  • Added metadata entries and updated the table.dat file 23 times since launch

  • Developed and maintained scripts and data flow to provide pre-processed ECMWF data extractions and plots on the web

  • Monitored the automatic file reception system (now containing nearly 5000 files totalling 275 MB + index database 20 MB)

Metadata issues
Metadata issues

Although many quality checks are performed on the data upon submission to the database:

  • The metadata definition is currently not complete enough to perform automatic checks of the validity of reported measurement units

  • Some components and units may still be inconsistently named or mis-used

    Several files have indeed been submitted containing such errors due to misinterpretation of the guidelines. The data submittors will be contacted to correct these files.