Observation Screening
1 / 37

Observation Screening - PowerPoint PPT Presentation

  • Uploaded on

Observation Screening. Kertész Sándor 3D-VAR/ODB Training Budapest, 7 June, 2006. OUTLINE. ODB primer Observation data flow Screening QC and decisions in ODB Screening inputs How is screening working? Output listings Exercises. ODB primer. ODB is Observational Database

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Observation Screening' - louie

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Observation Screening

Kertész Sándor

3D-VAR/ODB Training

Budapest, 7 June, 2006


  • ODB primer

  • Observation data flow

  • Screening QC and decisions in ODB

  • Screening inputs

  • How is screening working?

  • Output listings

  • Exercises

ODB primer

  • ODB is Observational Database

  • Basic data structure is called table. Tables is made up of data columns

  • Set of tables (with relations) forms a database type. E.g. ECMA that contains all the observations to be used in a DA system

  • Information on tables, columns etc.

    • http://www.cnrm.meteo.fr/gmapdoc/meshtml/DOC_odb/odb.html

    • In „Doc” directory on regatta: Assimilation_Chapter9_v2.3.pdf

    • ODB training!

ODB primer

  • An ODB database for a given analyis date is a set of files organized into a directory structure (check your BATOR output!)

  • ODB data is divided into pools for parallel runs (to ensure load balance)

  • To look into ODB you need to run the viewer with your view!

  • The view define the retrieval!

ODB primer

  • View definition (e.g. to retrieve AIREP data)

  • CREATE VIEW myview AS


  • obstype, lat, lon, varno, obsvalue, [email protected]

  • FROM hdr, body

  • WHERE obstype == 2

View name

Retrieved columns

Involved tables

Selection criteria

ODB primer

  • The concept of sub-bases was invented to have the choice for handling observation types separately. E.g.: ECMA.conv, ECMA.amsua, ECMA.amsub etc…

  • These ODB sub-bases must be merged into a new ODB

  • Many applications are working on this kind of ODB that is called virtual basis

Observation Screening

  • The observation screening is the last step in the pre-processing chain to produce observation inputs for the variational analysis

  • Screening performs various quality control checks on data

  • ALADIN screening is very similar to the global (ARPEGE) one

  • Screening uses an ECMA ODB as input

  • The production of this ODB is rather complicated, we have to step back to BATOR to see the complete data flow …

Obs dataflow: BATOR

  • BATOR is producing separate ECMA ODB sub-bases containing the required number of ODB pools

  • We need: #pools = # CPUs for further applications!

  • These ODB sub-bases must be merged into a new ECMA ODB using programmeSHUFFLE

  • Further applications are working on this ODB virtual basis

Obs dataflow: LAMFLAG

  • The resulting ODB is not suitable forALADIN screening since it is not working with observations outside the C+I zoneAbort in GEFGER!

  • Solution: the input ODB must be restricted to the C+I zoneprogrammeLAMFLAG


LAMFLAG writes status flags into ODB


[email protected]=0

[email protected]=1

Obs dataflow: LAMFLAG namelists

Observation filter

If .TRUE.: selection is based only on observation filter!






























Model geometry settings

Used for screening

Obs dataflow: SHUFFLE reduction

  • LAMFLAG puts flags into ODB but does not reduce it!

  • SHUFFLE must run with setting TO_ODB_REDUC=1 for each sub-bases in the ODB separately to produce a reduced ODB

  • Then another SHUFFLE run is needed to put sub-bases together again (merging)

  • This results in a virtual base ECMA ODB again that is now ready for used as screening input

Obs dataflow: Screening

  • Screening puts its quality control flags and decisions into the input ECMA ODB

  • QC flags, decisions for reports (in table hdr):

    • statusstatus_t

    • event1report_event1_t

    • blacklistreport_blacklist_t

report_event1_t     no_data                  bit1     all_rejected             bit1     bad_practice           bit1     rdb_rejected            bit1     rdb_activated          bit1     whitelist_activated   bit1     horipos_outrange    bit1     vertpos_outrange    bit1     time_outrange         bit1     redundant                bit1     land                     bit1     sea                      bit1…..

report_blacklist_t     obstype      bit1     statid       bit1     codetype     bit1     instype      bit1     date         bit1     time         bit1     lat          bit1     lon          bit1     stalt        bit1     scanpos      bit1     retrtype     bit1 …..


activebit1 passivebit1rejectedbit1blacklistedbit1


Passive and blacklisted are a subset of rejected!

Obs dataflow: Screening

  • QC flags, decisions and additional infromation for observed values (in table body):

    • status status_t

    • event1 report_event1_t

    • blacklist datum_blacklist_t

    • fg_depar = obs-guess


vertco_missing        bit1      obsvalue_missing    bit1      fg_missing              bit1      rdb_rejected            bit1      rdb_activated           bit1      whitelist_activated    bit1      bad_practice            bit1      vertpos_outrange     bit1      reflevel_outrange      bit1      fg2big                  bit1      depar2big              bit1      obs_error2big           bit1      datum_redundant      bit1      level_redundant         bit1 ….

datum_blacklist_t     obstype      bit1     statid       bit1     codetype     bit1     instype      bit1     date         bit1     time         bit1     lat          bit1     lon          bit1 ….


activebit1 passivebit1rejectedbit1blacklistedbit1


Obs dataflow: Compression

  • After Screening another SHUFFLE is running, this time to perform an ECMA CCMA conversion

  • A CCMA ODB consists of only those tables and column that are needed for the analysis

  • The resulting CCMA ODB after SHUFFLE contains only the active reports and observations!

  • This CCMA ODB is the input of the minimization (e131)

  • The observation data flow is finished!

ECMA (Full)





ECMA.sb2 …


ECMA.sb2 …







Obs dataflow summary

How to run screening?

  • It is configuration 002:

  • MASTER –c002 –maladin –vmeteo –t001 –aeul –eMIN1

  • Inputs:

    • ECMA ODB (observation, sigma-o, blacklisting)

    • First guess file under 5 different names:


    • Constants, statistics

    • Namelists


  • errgrib: GRIB file containing grid-point space sigma-B values (lat-lon grid, several levels recieved from MF)

  • rt_coef_atovs_neepred_ieee.dat: coefficients for the RTM (received from MF), binary file

  • rszcoef_fmt.dat: the same the previous file but in ASCII format

  • bcor_noaa.dat: bias correction file

  • chanspec_noaa: obsolete?

  • rmtberr_noaa.dat: obsolete?

  • cstlim_noaa.dat: obsolete?

Screening specific namelists

  • These parameters should be set like this for 3D-VAR:

    • NUPTRA = -1 , number of updates of the trajectory (NAMVAR)

    • NOBSHOR = 201, bilinear horizontal interpolation in H (NAMVAR)

    • LOBS = .TRUE., real obs (NAMCT0)

    • LOBSC1 = .FALSE., (NAMCT0), traj. is not computed

    • LSIMOB = .FALSE., no simelated obs (NAMCT0)

    • LSCREEN = .TRUE., (NAMCT0), screening is on!

    • LSCRE4D = .FALSE. , 3D screening (NAMSCC)

  • Tunable parameters are in NAMSCC and NAMOBS (discussed later)!

How is screening working?

Decisions are taken in subroutine:

Independent decisions: performed separately for each report, RDB flag is assigned to reports


[email protected]

[email protected]

Update of flags (FLGTST). Flags are converted into status: active, passive, blacklisted, rejected

Dependent decisions: dependency on other reports or obs, or the previous decisions

Independent decisions

  • Preliminary checks (PRECH): Completeness of report missing station altitude for SYNOP, TEMP and PILOT…

  • Blacklisting (BLACKCLN, BALCKSAT)

  • Background quality control (FIRST)


  • Blacklisting information is placed into ODB by BATOR

  • In the screening a separate selection is applied for satellite data (BLACKSAT)

  • Blacklisting is also performed in BLACKCLN:

    • Blacklisted obs will be rejected

    • Various blacklisting for SATOB (e.g. QI < 85)

    • Land-sea rejection (SYNOP, TEMP, PILOT). Controlled via NAMOBS LSLREJ (default is .T.)

      • LSLRW10: Wind 10m, LSLRT2: T 2m, LSLRRH2: THU 2m (default is .T. meaning rejection over land. Will get a passive status!)


  • Also in BLACKCLN:

    • Height-based rejection. Controlled via NAMOBS LHDLREJ (default is .T., rejected obs will get passive status!)

      • Orographic rejection limit is applied:

        • LHDRW10: Wind 10m, LHDRT2: T 2m, LHDRRH2: RHU 2m (default is .T.)

      • AIREP data and significant levels for TEMP and PILOT are not used below the model surface!

      • RHU and Q is rejected above 300 hPa

      • AIREP is not used below a certain pressure

Background QC (FIRST)

  • From the BLUE theory we know:

  • For a given location we expect:

Values are read from file errgrib and interpolated to obs locations

Values are read from ODB

Background QC (FIRST)

  • Flags are assigned to observations by checking:

Based on predefined flaglimits 4 flag values can be assigned to obs

0: correct

1: probably correct

2: probably incorrect

3: incorrect

Flaglimits are defined in DEFRUN

Dependent decisions

  • Remove redundant surface levels from TEMP/PILOT (REDSL)

  • Vertical consistency of profiles (VERCO)

  • Removal of duplicated reports (DUPLI): two reports with the same ID, time an location …

  • Redundancy check (REDUN)

  • Thinning for AIREP (THIAIR)

  • Specific scatterometer quality control

  • Thinning of satellite data (THINN)

Redundancy check (REDUN)

  • Applied for all active reports which are co-located and originate from the same station. Separate treatment for:

    • Land SYNOP: more reports for the same place but with different time, the one with closest to the analysis date with the most active obs are selected

    • Ship SYNOP, DRIBU: redundant if moving platforms are within a circle of 1° radius

    • TEMP and PILOT

    • SYNOP mass observation is redundant if TEMP obs is available and the diff < 50 hPa

Thinning of AIREP data (THIAIR)

  • One flight consist of a set of reports

  • Thinning is performed for each flight separately

    • 3D boxes are constructed around model levels

    • In each box the report closest to the analysis date with the most active observations are selected

    • Box size is set in metres via RFIND_AIREP in NAMSCC, the default is ~70 km!

  • By default box size less than 50 km cannot be used in ARPEGE, code modification is needed

Thinning of satellite data (THINNER)

  • For radiances a horizontal thinning is performed in two steps:

    • First a minimum distance is enforced (default ~ 70 km, can be changed in NAMSCC via RMIND_RAD1C(sensor))

    • Then a repeated thinning is performed to reach the final separation (default ~ 140 km, can be changed in NAMSCC via RFIND_RAD1C(sensor))

  • Where sensor is: 0=HIRS, 1=MSU, 2=SSU, 3=AMSU-A, 4=AMSU-B, 6=SSM/I, 20=METEOSAT, 21=MSG_HR, …

  • Selection criterion:

    • Sea is preferred over land

    • Clear sky pixel is preferred over a cloudy one

    • Obs closer to the analysis date is preferred

Thinning of satellite data (THINNER)

  • For SATOB a the thinning is performed:

    • Horizontally in two steps similarly to radiances: controlled in NAMSCC via RMIND_SATOB, RFIND_SATOB (default is ~70 and ~140 km)

    • Vertically around model levels

  • In each box the report closest to the analysis date with the most active observations are selected

  • QI values are also used in the selection

Sreening output

  • Valuable summary about screening decisions can be found in the NODE file:

  • Try „grep SCREENING STATISTICS” to get:

    • STATUS summary

    • EVENT summary

    • Number of variables, departures and missing departures

    • Diagnostic JO-table

Sreening output

  • Summary at „grep SCREENING STATISTICS”

    • Report and data status summary

OB.TYP REPORTS ACTIVE PASSIVE REJECTED BLACKLISTED 1 725 44 0 681 0 2 0 0 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0 5 61 60 0 1 0 6 61 59 0 2 0 7 0 0 0 0 0 8 0 0 0 0 0 9 0 0 0 0 0 10 0 0 0 0 0 ---------------------------------------------------------------------------------------- TOT 847 163 0 684 0

OB.TYP DATA ACTIVE PASSIVE REJECTED BLACKLISTED 1 4576 164 1846 4412 3862 2 0 0 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0 5 9186 7271 1218 1915 705 6 3438 2050 64 1388 72 7 0 0 0 0 0 8 0 0 0 0 0 9 0 0 0 0 0 10 0 0 0 0 0 ------------------------------------------------------------------------- TOT 17200 9485 3128 7715 4639

Sreening output

  • Summary at „grepSCREENING STATISTICS”

    • Diagnostic Jo table

Obstype 1 === SYNOP, Land stations and ships -------------------------------------------------- Codetype 11 === SYNOP Land Manual Report Variable DataCount Jo_Costfunction JO/n ObsErr BgErr U10 1380 2250.937197524 1.63 0.200E+01 0.000E+00 H2 721 834.1196058351 1.16 0.100E+00 0.000E+00 Z 567 892.5077244762 1.57 0.785E+02 0.000E+00 T2 721 2138.950482967 2.97 0.140E+01 0.000E+00 Q 721 1577.382611689 2.19 0.998E-03 0.000E+00---------- --------------------------- -------- ObsType 1 Total: 4110 7693.897622492 1.87

Sreening output

  • Observation monitoring available at:

  • http://pc2088.met.hu/monitor/start.php


  • Run screening for all the prepared observations with script ~/workdir/3dvar/Assim and study the output listing! (You have to put an exit after screening in the script).


  • Examine in detail the SYNOP and TEMP report for Budapest, see if surface obs (except Z) are really rejected, and find the reason for it. Try this view first:


    • SELECT

    • obstype, [email protected],varno,

  • obsvalue, fg_depar, press, [email protected],

  • [email protected], [email protected], [email protected],

  • [email protected], [email protected]

    • FROM hdr, body

    • WHERE (obstype == 1 OR obstype == 5) && statid = ‘12843 ‘

  • Guidance: A prepared viewer par file and sql file can be found in ~wshop01/workdir/OdbViewer as bp.par bp.sql. Copy them to your directory, set the sql path fo your file and type „viewer –p bp.par” to run viewer.

  • If you are brave enough try to write the same view for AIREP (obstype == 2) and explore bit fields of [email protected]!


  • Re-run screenig with a 10 km thinning distance for AIREP. To do this you have to use the exe you created yesterday (by modifying arp/obs_preporc/thiair.F90to reduce ISCALE to 1000!) Modify PROC_LAM_ODB and d_RUN in include.inc!

Compare the default and the new airep thinning in the output listings or in the ODB!