19 th egows meeting ljubljana june 9 12 2008 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
WDB Weather and Water Database PowerPoint Presentation
Download Presentation
WDB Weather and Water Database

Loading in 2 Seconds...

play fullscreen
1 / 36

WDB Weather and Water Database - PowerPoint PPT Presentation


  • 492 Views
  • Uploaded on

19 th EGOWS Meeting, Ljubljana, June 9-12, 2008. WDB Weather and Water Database Outline Introduction Background of WDB Overview of WDB WDB Components Future of WDB Who are we? Michael Akinde Ph.D., Database Architect, IT, Met.no Systems architect on ROAD system at SMHI(2003-2006)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'WDB Weather and Water Database' - Patman


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
outline
Outline
  • Introduction
  • Background of WDB
  • Overview of WDB
  • WDB Components
  • Future of WDB
who are we
Who are we?
  • Michael Akinde
    • Ph.D., Database Architect, IT, Met.no
    • Systems architect on ROAD system at SMHI(2003-2006)
  • Vegard Bønes
    • M.Sc., Senior Engineer, IT, Met.no
  • Børge Moe
    • M.Sc., Senior Engineer, IT, Met.no
who are you
Who are you?
  • Survey: please raise your hands...
    • Who has coded programs?
    • Who has written SQL queries?
    • Who has worked with database APIs (JDBC,ODBC,etc)?
    • Who knows what database normalization is?
    • Who knows what a database index is?
    • Who has written database extensions?
what is this about
What is this about?
  • We need to store weather/water data
  • We need indexed (fast) access to time series retrieved from multiple files or sources
  • We need access from a variety of applications
  • We need products to be consistent
  • We want to be able to track changes
  • We want programs to be independent of the data format
  • We have limited ressources
what is wdb
What is WDB?
  • WDB is a storage system for meteorological, hydrological, and oceanographic data
    • Developed by the PROFF program at met.no
    • Based on the ROAD (SMHI) architecture
history of road
History of ROAD
  • Operational meteorological database, developed in the late 90’s at SMHI; operational since 2001
  • Responsible for 90% of SMHI’s automatic production
  • One of the world’s fastest meteorological databases
  • Mature and well-established production system; 3rd generation in development 2006-2008
problems of road
Problems of ROAD
  • Like most met systems, ROAD is a legacy system
    • Prior to ROAD3G, the loading chain of the ROAD system included Fortran code developed in the 1970s
  • SMHI is in the process of a major rehaul of the production systems in terms of migrating software from OpenVMS to Linux
  • IBM Informix platform
    • Geodetic datablade (geospatial extension)
    • Licenses are expensive
wdb is
WDB is…
  • Open source collaboration between met.no and SMHI
  • Can be considered the 4th generation of the ROAD system
    • Built from the ground up without legacy code
    • Developed at met.no
  • Released under the GNU GPL2 license

Fast, Free, Flexible

wdb open source alternative
WDB: Open Source Alternative
  • Should be robust enough to handle high volume, high availability production
    • e.g., as the backend engine of yr.no
  • Should be flexible
    • e.g., to easily handle new types of data
  • Should support quality and consistency of data
  • Should be simple to use
  • Should be easy to maintain and operate
wdb provides
WDB provides...
  • Quick and easy to deploy data storage system
  • Modular loading programs
    • Easy to add more…
  • Data-independent application programmers interface (API)
    • Usable, in theory, from any environment
  • Fast performance on queries
installing wdb
Installing WDB
  • Download the tar package, and...
  • Or request the debian package

server:wdb-x.x% ./configure --prefix=/home/myuser/local

server:wdb-x.x% make

server:wdb-x.x% make install

putting wdb through its paces
Putting WDB through its paces
  • WDB implements all the standard GNU targets
  • “make check” runs the WDB unit tests
    • tests that can be run without being installed
  • “make installcheck” runs the WDB install tests
    • tests that require an installed database
  • 300+ tests and counting
    • CPPunit and TAP protocol
    • Fully automated test framework on CruiseControl
enabling technology
Enabling Technology
  • Linux (Debian, Fedora Core 5)
  • PostgreSQL version 8.1.x or later
  • PostGIS version 1.1.x or later
  • Proj.4 version 4.6.0
  • ECMWF GRIB API
  • GNU Autotools
  • C++ open source libraries
    • Boost Std Libraries, log4cpp, cppunit, etc.
loading forecast data
Loading forecast data
  • Loading data into the database, is as simple as:

server:mydata% gribLoad myforecast.grib

server:mydata%

server:mydata% gribLoad /opdata/hirlam10/*.grib

server:mydata%

loading programs
Loading Programs
  • GRIBLoad is the design pattern for the WDB loading programs
  • Loads one or more GRIB1 files into the database
    • Uses ECMWF GRIB API, so theoretically, GRIB2 works as well
    • Stand-alone client program that connects to any WDB database
  • Loaders for other grid formats are easily built in the same way by reusing code
  • Loading of BUFR will be ported from ROAD
wdb call interface wci
WDB Call Interface (WCI)
  • WCI is the data retrieval API for WDB
  • Based on a function interface in SQL

Why not stick to “simple” SQL?

- SQL isn't really that simple

- Requires knowledge of the underlying data model (tables/views)

- Lacks useful functionality; e.g., interpolation, point-in-field, etc.

wci versus pure sql
WCI versus pure SQL
  • SQL results in tight coupling between application and database
    • Table names
    • Table structure (attributes)
    • Really strange database queries
wdb call interface wci22
WDB Call Interface (WCI)
  • WCI is the data retrieval API for WDB
  • Based on a function interface in SQL

Why not a lib file?

- Changes in a library require that the applications using the

library are recompiled

- New libraries need to be pushed to the user (application)

- Each operating system, machine and language = its own library

wci versus compiled software library
WCI versus compiled software library
  • How many combinations do we need?

Python

C

Perl

C++

Java

Fortran

Debian

Fedora

Ubuntu

Sun Solaris

Windows XP

Windows Vista

Don't forget 32 vs 64 bit architectures...

wdb call interface features
WDB Call Interface Features
  • Stable Interface
  • Performance
    • 100-1000 values/s in single point retrieval (to be improved)
    • 10 million floating points/s in field retrieval (about 100 fields/s)
  • Functionality:
    • Pointwise, Timeseries, 2d-volumes, Quality information
    • Coming: Paths, Interpolation in space and time, 3D volumes?
  • Easy to program
    • 1/2 page of code to retrieve MHO-data
  • Platforms: C/C++, Java, Perl, Python, Fortran, ODBC…
  • Operating systems: Pretty much anything...
wci functions
WCI functions
  • wci.begin
  • wci.end
  • wci.read
  • wci.write
  • wci.browse
  • wci.info
    • Programmer doesn’t need any knowledge of the data format or the database model
the wci data model
The WCI Data Model

Time (Valid)

Parameter

Time (Ref)

Level

Place

Version

Data Value

Data Provider

Quality

??

Value ID

wdb2ts
WDB2TS
  • TS2XML for WDB
  • Web service to retrieve time series of data from a WDB database
    • delivers data as csv or xml
    • REST-like interface

http://host/path?lat=10;lon=10;parameterspec

(parameterspec = data specification)

the road ahead for wdb
The Road ahead for WDB
  • WDB version 0.6.0
    • First production version of WDB
    • Hindcast archive (12 TBs of grids)
  • WDB version 0.7.4
    • To be released next week
    • We currently release a new version about once a month
    • Implements a new loading program
      • FELT: met.no field format
      • Approximate time to develop: 2 weeks x 2 persons
wdb roadmap
WDB Roadmap
  • Version 0.8.0 is scheduled for July 2008
    • Official beta version of WDB
    • Limited scale real-time production
  • Version 0.9.0 is scheduled for October 2008
    • Release candidate version (full-scale testing)
  • Version 1.0 scheduled for December 2008
    • Full-scale production version
  • WDB is a part of the PROFF framework; minimum expected life-time is at least15 years
future thoughts
Future thoughts?
  • Complete integration with Diana
  • More data types
    • Fields (NetCDF, GRIB2, etc.)
    • Observations (BUFR, XML)
  • Better administration tools
  • Verified Stability
    • 24x7 hands-free operation (have this in ROAD…)
  • Improved Performance
    • We have not started optimizations
future thoughts32
Future thoughts?
  • Scalability and fail-over
    • Want more load? Add another server
    • Want to guarantee up-time? Add more servers
  • Consistency
    • Easy on one server; we want multiple servers
    • Distributed data/replication is a well-known problem in the database community
    • Problems: huge data, geographically distant regions
  • Tracking of changes
    • Metadata can be used to track history…
recommended reading
Recommended Reading
  • WDB website (wdb.met.no)
    • WDB User Manual
    • WCI User Manual
    • Man Pages for all WDB utilities
    • Source code browser (SVN repository and doxygen)
    • Developer documentation (design docs)
    • Eclipse setup instructions
  • Subscribe to WDB project on Freshmeat.net
summary
Summary
  • WDB is designed as a generic weather/water database system
    • Perspective: one core system that can be leveraged for many different uses
    • Functionality that isn't implemented can probably be added
    • For users: fast access to data
    • For developers: a simple, generic interface to all kinds of different data
    • For operators: a system that is easy to maintain
concluding thoughts
Concluding thoughts…
  • Many, many more WDB systems in the future
    • WDB is intended as the backbone of the production systems at met.no and SMHI
  • Open source is a different model of collaboration which can be of great benefit to meteorological institutions
    • We all require broadly the same functionality
    • Good communications = better systems
    • Free as in Freedom