Envisat Validation Workshop
This presentation is the property of its rightful owner.
Sponsored Links
1 / 21

Envisat Validation Workshop Atmospheric Chemistry Validation Team PowerPoint PPT Presentation


  • 55 Views
  • Uploaded on
  • Presentation posted in: General

Envisat Validation Workshop Atmospheric Chemistry Validation Team Ground-Based Measurements and Campaign Database Subgroup Campaign Database. Terje.Krognes , [email protected] Aasmund.Fahre.Vik, [email protected] Norwegian Institute for Air Research (NILU). Presentation Outline. Data Centre Status

Download Presentation

Envisat Validation Workshop Atmospheric Chemistry Validation Team

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Envisat validation workshop atmospheric chemistry validation team

Envisat Validation Workshop

Atmospheric Chemistry Validation Team

Ground-Based Measurements and Campaign Database Subgroup

Campaign Database

Terje.Krognes, [email protected]

Aasmund.Fahre.Vik, [email protected]

Norwegian Institute for Air Research (NILU)


Presentation outline

PresentationOutline

  • Data Centre Status

  • Tools

    • Local end-user tools

    • WWW search tools

  • Commissioning Phase Activities


Data flow

nadir web

DOCS

DATA ORIGINATOR

DATA

table.dat

ASC2HDF

HDF2ASC

FILES

http or ftp

DATABASE

Error email

OK?

http

SEARCH

DATA USER

ftp to zardoz

Data flow

  • Signed protocol required for access to tools, docs and files

  • ECMWF data requires additional protocol


Data centre status 1 3 hardware

spare

Internet

nadir

http

Storage

Firewall

zardoz

ftp,telnet

Data Centre Status (1/3): Hardware

  • The machines have been swapped between production and spare servers

  • The hardware has not been changed


Data centre status 2 3 software

Data Centre Status (2/3): Software

  • Web-server (nadir) runs Linux Redhat 7.2 with up2date ~weekly (6.2 at 2nd rehearsal)

  • File server (zardoz) runs Solaris 7, as before

  • Nusphere MySQL

    • Apache 1.3.22 (1.3.14)

    • MySQL 3.23.44 (3.23.32)

    • PHP 4.0.6 (4.0.4pl1)

  • IDL 5.4, as before

  • HDF 4.1r3, as before

  • No re-boot on nadir since April 2002

  • Uptime on zardoz since April 2002 is approximately 99,98% (2 half-hour maintenance + re-boot)


Data centre status 3 3 metadata

Data Centre Status (3/3): Metadata

  • Metadata Guidelines updated by metadata board on 24 April 2002

  • During commissioning phase, updates of various metadata table entries, without affecting structure:

    • Project-related information

    • Actual descriptors

  • Guidelines still applicable, but more clarifications to be added.


New user tools

New user tools

  • Major re-write of the ASC2HDF file generation and checking tool around launch

  • 6 smaller upgrades of the ASC2HDF package after launch

  • Entire ASC2HDF source code (FORTRAN) published to users. HDF output routines may be combined with user’s own input and processing modules

  • IDL-tool to generate HDF data files (without error checking): IDL2HDF

  • IDL tool for reading and plotting HDF data files (HDF2IDL), to be downloaded and used locally

  • IDL subroutine to read HDF Cal/Val data, for integration into user-developed software (nilu_hdf_read_user.pro)


New web functionality ecmwf

New web functionality - ECMWF

  • ECMWF plots (NH 2,5º*2,5º and T106)

  • ECMWF 10-day back trajectory plots

  • T106 extractions to HDF files


New web functionality search and plot

New web functionality - search and plot

  • Use of defaults

  • Facility to save and retrieve a search

  • Provision for downloading multiple files in tarball via web interface

  • Improved on-line plotting (also for multidimensional data)

  • Corrected display of comments to variables


File searching 1 4

File searching (1/4)

  • Defaults are marked with ***

  • Page initially opens without defaults

  • Page may be reset or set to saved defaults bys selecting a previously saved query

  • Files may be filtered by a large number of metadata parameters


File searching 2 4

File searching (2/4)

  • The default criteria page lists all criteria sets that this user has previously saved

  • Entries may easily be removed

  • The main search page will be loaded with the selected defaults when launched from this page

  • Defaults may include 4D space-time parameters and variable name wild-cards


File searching 3 4

File searching (3/4)

  • The NUMERICAL SIMULATION entry is now marked with *** since we selected it as default through the saved query page

  • Variable name / mode / descriptor have wildcard entries automatically added to the drop down lists.

  • Variable name and variable mode accept multiple selections

  • 4D space-time criteria may be entered as a single point, or as a max/min box

  • All files that include or touch the 4D criteria are listed as matches


File searching 4 4

File searching (4/4)

  • Files may be selected by posting date

  • Files updated after a given date may be selected

  • The file list may be sorted by file name, by submitter or PI, by posting date, or by AOID

  • If no criteria are selected, the entire file list will be requested


File list

File List

  • The file list displays the current selection criteria

  • A single file may be downloaded immediately

  • Files may be marked and downloaded in a compressed tarball

  • Search criteria may be saved for later re-use (type a meaningful description before saving)

  • The Variables link brings up a variable list and other documentation contained in the file


Variable list

Variable List

  • The variable list page displays all parameters contained in a file

  • Variable name, mode and descriptor are displayed

  • The measurement unit and corresponding SI unit are displayed

  • All documentation fields attached to the file are displayed

  • Additional documentation fields attached to a single variable may be displayed through the Comments link

  • Variables may be selected to generate an on-line plot (next slide)

  • Observe the dependencies listed for each variable – not all variable combinations will produce meaningful plots


Variable plotting on line browse

Variable plotting (on-line browse)


Administrative tools

Administrative tools

  • Web tools to report uploads/downloads

  • Web tools to archive/retrieve outdated data files

  • Web tools to check and maintain index database consistency and integrity


User activities

User activities

  • Users have sent approximately 200 emails with requests for accounts or updates to metadata, or with questions to HDF file generation

  • After launch focus has slowly shifted to requests for help with HDF file formatting issues and removal of outdated files

  • From May to November 2002 users have submitted 5880 HDF data files (570 of these were rejected due to formatting problems, and 238 have later been archived).

  • From May to November 2002 users have downloaded 4000 HDF data files (totalling 215 MB) (ftp downloads via zardoz not included)

  • There are currently 4741 ACVT files open to users (22 active AOIDs)

  • There are currently 462 MAVT files open to users (5 active AOIDs)


User activities up and downloads

User activities: Up and Downloads

The download statistics only covers WWW downloads from nadir, not FTP downloads from zardoz


Nadir team activities

Nadir-team activities

  • 300+ interactions with users (of which 50% during Commissioning Phase)

  • 250+ interactions with ESA and the Metadata Board (10% during Commissioning Phase)

  • One major re-write and 6 updates of the ASC2HDF and HDF2ASC programs

  • Maintained and improved shell scripts for automatic file checking and storage

  • Created additional IDL and FORTRAN tools

  • Performed extensive web program development

  • Performed user administration updates

  • Added metadata entries and updated the table.dat file 23 times since launch

  • Developed and maintained scripts and data flow to provide pre-processed ECMWF data extractions and plots on the web

  • Monitored the automatic file reception system (now containing nearly 5000 files totalling 275 MB + index database 20 MB)


Metadata issues

Metadata issues

Although many quality checks are performed on the data upon submission to the database:

  • The metadata definition is currently not complete enough to perform automatic checks of the validity of reported measurement units

  • Some components and units may still be inconsistently named or mis-used

    Several files have indeed been submitted containing such errors due to misinterpretation of the guidelines. The data submittors will be contacted to correct these files.


  • Login