slide1 n.
Skip this Video
Download Presentation
Observation Screening

Loading in 2 Seconds...

play fullscreen
1 / 37

Observation Screening - PowerPoint PPT Presentation

  • Uploaded on

Observation Screening. Kertész Sándor 3D-VAR/ODB Training Budapest, 7 June, 2006. OUTLINE. ODB primer Observation data flow Screening QC and decisions in ODB Screening inputs How is screening working? Output listings Exercises. ODB primer. ODB is Observational Database

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Observation Screening' - louie

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Observation Screening

Kertész Sándor

3D-VAR/ODB Training

Budapest, 7 June, 2006



  • ODB primer
  • Observation data flow
  • Screening QC and decisions in ODB
  • Screening inputs
  • How is screening working?
  • Output listings
  • Exercises

ODB primer

  • ODB is Observational Database
  • Basic data structure is called table. Tables is made up of data columns
  • Set of tables (with relations) forms a database type. E.g. ECMA that contains all the observations to be used in a DA system
  • Information on tables, columns etc.
    • In „Doc” directory on regatta: Assimilation_Chapter9_v2.3.pdf
    • ODB training!

ODB primer

  • An ODB database for a given analyis date is a set of files organized into a directory structure (check your BATOR output!)
  • ODB data is divided into pools for parallel runs (to ensure load balance)
  • To look into ODB you need to run the viewer with your view!
  • The view define the retrieval!

ODB primer

  • View definition (e.g. to retrieve AIREP data)
  • CREATE VIEW myview AS
  • obstype, lat, lon, varno, obsvalue, status.blacklist@body
  • FROM hdr, body
  • WHERE obstype == 2

View name

Retrieved columns

Involved tables

Selection criteria


ODB primer

  • The concept of sub-bases was invented to have the choice for handling observation types separately. E.g.: ECMA.conv, ECMA.amsua, ECMA.amsub etc…
  • These ODB sub-bases must be merged into a new ODB
  • Many applications are working on this kind of ODB that is called virtual basis

Observation Screening

  • The observation screening is the last step in the pre-processing chain to produce observation inputs for the variational analysis
  • Screening performs various quality control checks on data
  • ALADIN screening is very similar to the global (ARPEGE) one
  • Screening uses an ECMA ODB as input
  • The production of this ODB is rather complicated, we have to step back to BATOR to see the complete data flow …

Obs dataflow: BATOR

  • BATOR is producing separate ECMA ODB sub-bases containing the required number of ODB pools
  • We need: #pools = # CPUs for further applications!
  • These ODB sub-bases must be merged into a new ECMA ODB using programmeSHUFFLE
  • Further applications are working on this ODB virtual basis

Obs dataflow: LAMFLAG

  • The resulting ODB is not suitable forALADIN screening since it is not working with observations outside the C+I zoneAbort in GEFGER!
  • Solution: the input ODB must be restricted to the C+I zoneprogrammeLAMFLAG


LAMFLAG writes status flags into ODB





Obs dataflow: LAMFLAG namelists

Observation filter

If .TRUE.: selection is based only on observation filter!






























Model geometry settings

Used for screening


Obs dataflow: SHUFFLE reduction

  • LAMFLAG puts flags into ODB but does not reduce it!
  • SHUFFLE must run with setting TO_ODB_REDUC=1 for each sub-bases in the ODB separately to produce a reduced ODB
  • Then another SHUFFLE run is needed to put sub-bases together again (merging)
  • This results in a virtual base ECMA ODB again that is now ready for used as screening input

Obs dataflow: Screening

  • Screening puts its quality control flags and decisions into the input ECMA ODB
  • QC flags, decisions for reports (in table hdr):
    • statusstatus_t
    • event1report_event1_t
    • blacklistreport_blacklist_t

report_event1_t     no_data                  bit1     all_rejected             bit1     bad_practice           bit1     rdb_rejected            bit1     rdb_activated          bit1     whitelist_activated   bit1     horipos_outrange    bit1     vertpos_outrange    bit1     time_outrange         bit1     redundant                bit1     land                     bit1     sea                      bit1…..

report_blacklist_t     obstype      bit1     statid       bit1     codetype     bit1     instype      bit1     date         bit1     time         bit1     lat          bit1     lon          bit1     stalt        bit1     scanpos      bit1     retrtype     bit1 …..


activebit1 passivebit1rejectedbit1blacklistedbit1


Passive and blacklisted are a subset of rejected!


Obs dataflow: Screening

  • QC flags, decisions and additional infromation for observed values (in table body):
    • status status_t
    • event1 report_event1_t
    • blacklist datum_blacklist_t
    • fg_depar = obs-guess


vertco_missing        bit1      obsvalue_missing    bit1      fg_missing              bit1      rdb_rejected            bit1      rdb_activated           bit1      whitelist_activated    bit1      bad_practice            bit1      vertpos_outrange     bit1      reflevel_outrange      bit1      fg2big                  bit1      depar2big              bit1      obs_error2big           bit1      datum_redundant      bit1      level_redundant         bit1 ….

datum_blacklist_t     obstype      bit1     statid       bit1     codetype     bit1     instype      bit1     date         bit1     time         bit1     lat          bit1     lon          bit1 ….


activebit1 passivebit1rejectedbit1blacklistedbit1



Obs dataflow: Compression

  • After Screening another SHUFFLE is running, this time to perform an ECMA CCMA conversion
  • A CCMA ODB consists of only those tables and column that are needed for the analysis
  • The resulting CCMA ODB after SHUFFLE contains only the active reports and observations!
  • This CCMA ODB is the input of the minimization (e131)
  • The observation data flow is finished!

ECMA (Full)





ECMA.sb2 …


ECMA.sb2 …







Obs dataflow summary


How to run screening?

  • It is configuration 002:
  • MASTER –c002 –maladin –vmeteo –t001 –aeul –eMIN1
  • Inputs:
    • ECMA ODB (observation, sigma-o, blacklisting)
    • First guess file under 5 different names:
    • Constants, statistics
    • Namelists


  • errgrib: GRIB file containing grid-point space sigma-B values (lat-lon grid, several levels recieved from MF)
  • rt_coef_atovs_neepred_ieee.dat: coefficients for the RTM (received from MF), binary file
  • rszcoef_fmt.dat: the same the previous file but in ASCII format
  • bcor_noaa.dat: bias correction file
  • chanspec_noaa: obsolete?
  • rmtberr_noaa.dat: obsolete?
  • cstlim_noaa.dat: obsolete?

Screening specific namelists

  • These parameters should be set like this for 3D-VAR:
    • NUPTRA = -1 , number of updates of the trajectory (NAMVAR)
    • NOBSHOR = 201, bilinear horizontal interpolation in H (NAMVAR)
    • LOBS = .TRUE., real obs (NAMCT0)
    • LOBSC1 = .FALSE., (NAMCT0), traj. is not computed
    • LSIMOB = .FALSE., no simelated obs (NAMCT0)
    • LSCREEN = .TRUE., (NAMCT0), screening is on!
    • LSCRE4D = .FALSE. , 3D screening (NAMSCC)
  • Tunable parameters are in NAMSCC and NAMOBS (discussed later)!

How is screening working?

Decisions are taken in subroutine:

Independent decisions: performed separately for each report, RDB flag is assigned to reports




Update of flags (FLGTST). Flags are converted into status: active, passive, blacklisted, rejected

Dependent decisions: dependency on other reports or obs, or the previous decisions


Independent decisions

  • Preliminary checks (PRECH): Completeness of report missing station altitude for SYNOP, TEMP and PILOT…
  • Blacklisting (BLACKCLN, BALCKSAT)
  • Background quality control (FIRST)


  • Blacklisting information is placed into ODB by BATOR
  • In the screening a separate selection is applied for satellite data (BLACKSAT)
  • Blacklisting is also performed in BLACKCLN:
    • Blacklisted obs will be rejected
    • Various blacklisting for SATOB (e.g. QI < 85)
    • Land-sea rejection (SYNOP, TEMP, PILOT). Controlled via NAMOBS LSLREJ (default is .T.)
      • LSLRW10: Wind 10m, LSLRT2: T 2m, LSLRRH2: THU 2m (default is .T. meaning rejection over land. Will get a passive status!)


  • Also in BLACKCLN:
    • Height-based rejection. Controlled via NAMOBS LHDLREJ (default is .T., rejected obs will get passive status!)
      • Orographic rejection limit is applied:
        • LHDRW10: Wind 10m, LHDRT2: T 2m, LHDRRH2: RHU 2m (default is .T.)
      • AIREP data and significant levels for TEMP and PILOT are not used below the model surface!
      • RHU and Q is rejected above 300 hPa
      • AIREP is not used below a certain pressure

Background QC (FIRST)

  • From the BLUE theory we know:
  • For a given location we expect:

Values are read from file errgrib and interpolated to obs locations

Values are read from ODB


Background QC (FIRST)

  • Flags are assigned to observations by checking:

Based on predefined flaglimits 4 flag values can be assigned to obs

0: correct

1: probably correct

2: probably incorrect

3: incorrect

Flaglimits are defined in DEFRUN


Dependent decisions

  • Remove redundant surface levels from TEMP/PILOT (REDSL)
  • Vertical consistency of profiles (VERCO)
  • Removal of duplicated reports (DUPLI): two reports with the same ID, time an location …
  • Redundancy check (REDUN)
  • Thinning for AIREP (THIAIR)
  • Specific scatterometer quality control
  • Thinning of satellite data (THINN)

Redundancy check (REDUN)

  • Applied for all active reports which are co-located and originate from the same station. Separate treatment for:
    • Land SYNOP: more reports for the same place but with different time, the one with closest to the analysis date with the most active obs are selected
    • Ship SYNOP, DRIBU: redundant if moving platforms are within a circle of 1° radius
    • TEMP and PILOT
    • SYNOP mass observation is redundant if TEMP obs is available and the diff < 50 hPa

Thinning of AIREP data (THIAIR)

  • One flight consist of a set of reports
  • Thinning is performed for each flight separately
    • 3D boxes are constructed around model levels
    • In each box the report closest to the analysis date with the most active observations are selected
    • Box size is set in metres via RFIND_AIREP in NAMSCC, the default is ~70 km!
  • By default box size less than 50 km cannot be used in ARPEGE, code modification is needed

Thinning of satellite data (THINNER)

  • For radiances a horizontal thinning is performed in two steps:
    • First a minimum distance is enforced (default ~ 70 km, can be changed in NAMSCC via RMIND_RAD1C(sensor))
    • Then a repeated thinning is performed to reach the final separation (default ~ 140 km, can be changed in NAMSCC via RFIND_RAD1C(sensor))
  • Where sensor is: 0=HIRS, 1=MSU, 2=SSU, 3=AMSU-A, 4=AMSU-B, 6=SSM/I, 20=METEOSAT, 21=MSG_HR, …
  • Selection criterion:
    • Sea is preferred over land
    • Clear sky pixel is preferred over a cloudy one
    • Obs closer to the analysis date is preferred

Thinning of satellite data (THINNER)

  • For SATOB a the thinning is performed:
    • Horizontally in two steps similarly to radiances: controlled in NAMSCC via RMIND_SATOB, RFIND_SATOB (default is ~70 and ~140 km)
    • Vertically around model levels
  • In each box the report closest to the analysis date with the most active observations are selected
  • QI values are also used in the selection

Sreening output

  • Valuable summary about screening decisions can be found in the NODE file:
  • Try „grep SCREENING STATISTICS” to get:
    • STATUS summary
    • EVENT summary
    • Number of variables, departures and missing departures
    • Diagnostic JO-table

Sreening output

  • Summary at „grep SCREENING STATISTICS”
    • Report and data status summary

OB.TYP REPORTS ACTIVE PASSIVE REJECTED BLACKLISTED 1 725 44 0 681 0 2 0 0 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0 5 61 60 0 1 0 6 61 59 0 2 0 7 0 0 0 0 0 8 0 0 0 0 0 9 0 0 0 0 0 10 0 0 0 0 0 ---------------------------------------------------------------------------------------- TOT 847 163 0 684 0

OB.TYP DATA ACTIVE PASSIVE REJECTED BLACKLISTED 1 4576 164 1846 4412 3862 2 0 0 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0 5 9186 7271 1218 1915 705 6 3438 2050 64 1388 72 7 0 0 0 0 0 8 0 0 0 0 0 9 0 0 0 0 0 10 0 0 0 0 0 ------------------------------------------------------------------------- TOT 17200 9485 3128 7715 4639


Sreening output

  • Summary at „grepSCREENING STATISTICS”
    • Diagnostic Jo table

Obstype 1 === SYNOP, Land stations and ships -------------------------------------------------- Codetype 11 === SYNOP Land Manual Report Variable DataCount Jo_Costfunction JO/n ObsErr BgErr U10 1380 2250.937197524 1.63 0.200E+01 0.000E+00 H2 721 834.1196058351 1.16 0.100E+00 0.000E+00 Z 567 892.5077244762 1.57 0.785E+02 0.000E+00 T2 721 2138.950482967 2.97 0.140E+01 0.000E+00 Q 721 1577.382611689 2.19 0.998E-03 0.000E+00---------- --------------------------- -------- ObsType 1 Total: 4110 7693.897622492 1.87


Sreening output

  • Observation monitoring available at:


  • Run screening for all the prepared observations with script ~/workdir/3dvar/Assim and study the output listing! (You have to put an exit after screening in the script).


  • Examine in detail the SYNOP and TEMP report for Budapest, see if surface obs (except Z) are really rejected, and find the reason for it. Try this view first:
    • SELECT
    • obstype,,varno,
  • obsvalue, fg_depar, press,,
  • status.passive@body, status.blacklisted@body, status.rejected@body,
  •, event1.datum_redundant@body
    • FROM hdr, body
    • WHERE (obstype == 1 OR obstype == 5) && statid = ‘12843 ‘
  • Guidance: A prepared viewer par file and sql file can be found in ~wshop01/workdir/OdbViewer as bp.par bp.sql. Copy them to your directory, set the sql path fo your file and type „viewer –p bp.par” to run viewer.
  • If you are brave enough try to write the same view for AIREP (obstype == 2) and explore bit fields of event@body!


  • Re-run screenig with a 10 km thinning distance for AIREP. To do this you have to use the exe you created yesterday (by modifying arp/obs_preporc/thiair.F90to reduce ISCALE to 1000!) Modify PROC_LAM_ODB and d_RUN in!

Compare the default and the new airep thinning in the output listings or in the ODB!