Observation Screening
Download
1 / 37

Observation Screening - PowerPoint PPT Presentation


  • 134 Views
  • Uploaded on

Observation Screening. Kertész Sándor 3D-VAR/ODB Training Budapest, 7 June, 2006. OUTLINE. ODB primer Observation data flow Screening QC and decisions in ODB Screening inputs How is screening working? Output listings Exercises. ODB primer. ODB is Observational Database

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Observation Screening' - louie


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Observation Screening

Kertész Sándor

3D-VAR/ODB Training

Budapest, 7 June, 2006


OUTLINE

  • ODB primer

  • Observation data flow

  • Screening QC and decisions in ODB

  • Screening inputs

  • How is screening working?

  • Output listings

  • Exercises


ODB primer

  • ODB is Observational Database

  • Basic data structure is called table. Tables is made up of data columns

  • Set of tables (with relations) forms a database type. E.g. ECMA that contains all the observations to be used in a DA system

  • Information on tables, columns etc.

    • http://www.cnrm.meteo.fr/gmapdoc/meshtml/DOC_odb/odb.html

    • In „Doc” directory on regatta: Assimilation_Chapter9_v2.3.pdf

    • ODB training!


ODB primer

  • An ODB database for a given analyis date is a set of files organized into a directory structure (check your BATOR output!)

  • ODB data is divided into pools for parallel runs (to ensure load balance)

  • To look into ODB you need to run the viewer with your view!

  • The view define the retrieval!


ODB primer

  • View definition (e.g. to retrieve AIREP data)

  • CREATE VIEW myview AS

  • SELECT

  • obstype, lat, lon, varno, obsvalue, [email protected]

  • FROM hdr, body

  • WHERE obstype == 2

View name

Retrieved columns

Involved tables

Selection criteria


ODB primer

  • The concept of sub-bases was invented to have the choice for handling observation types separately. E.g.: ECMA.conv, ECMA.amsua, ECMA.amsub etc…

  • These ODB sub-bases must be merged into a new ODB

  • Many applications are working on this kind of ODB that is called virtual basis


Observation Screening

  • The observation screening is the last step in the pre-processing chain to produce observation inputs for the variational analysis

  • Screening performs various quality control checks on data

  • ALADIN screening is very similar to the global (ARPEGE) one

  • Screening uses an ECMA ODB as input

  • The production of this ODB is rather complicated, we have to step back to BATOR to see the complete data flow …


Obs dataflow: BATOR

  • BATOR is producing separate ECMA ODB sub-bases containing the required number of ODB pools

  • We need: #pools = # CPUs for further applications!

  • These ODB sub-bases must be merged into a new ECMA ODB using programmeSHUFFLE

  • Further applications are working on this ODB virtual basis


Obs dataflow: LAMFLAG

  • The resulting ODB is not suitable forALADIN screening since it is not working with observations outside the C+I zoneAbort in GEFGER!

  • Solution: the input ODB must be restricted to the C+I zoneprogrammeLAMFLAG

E

LAMFLAG writes status flags into ODB

C+I

[email protected]=0

[email protected]=1


Obs dataflow: LAMFLAG namelists

Observation filter

If .TRUE.: selection is based only on observation filter!

&NAMFCNT

LOBSONLY=.FALSE.

/END

&NAMFGEOM

ELAT0=46.2447006399999978,

ELON0=17.0000000000000000,

ELATC=46.2447005948822678,

ELONC=16.9999999982550491,

EDELX=12176.1563281414165,

EDELY=12176.1563281414165,

NDLUN=1,

NDGUN=1,

NDLUX=229,

NDGUX=205,

Z_CANZONE=1500.,

LVAR=.T.,

LNEWGEOM=.F.,

/END

&NAMFOBS

LSYNOP=.T.,

LAIREP=.T.,

LSATOB=.T.,

LDRIBU=.F.,

LTEMP=.T.,

LPILOT=.T.,

LSATEM=.T.

LPAOB=.F.,

LSCATT=.F.,

/END

Model geometry settings

Used for screening


Obs dataflow: SHUFFLE reduction

  • LAMFLAG puts flags into ODB but does not reduce it!

  • SHUFFLE must run with setting TO_ODB_REDUC=1 for each sub-bases in the ODB separately to produce a reduced ODB

  • Then another SHUFFLE run is needed to put sub-bases together again (merging)

  • This results in a virtual base ECMA ODB again that is now ready for used as screening input


Obs dataflow: Screening

  • Screening puts its quality control flags and decisions into the input ECMA ODB

  • QC flags, decisions for reports (in table hdr):

    • statusstatus_t

    • event1report_event1_t

    • blacklistreport_blacklist_t

report_event1_t     no_data                  bit1     all_rejected             bit1     bad_practice           bit1     rdb_rejected            bit1     rdb_activated          bit1     whitelist_activated   bit1     horipos_outrange    bit1     vertpos_outrange    bit1     time_outrange         bit1     redundant                bit1     land                     bit1     sea                      bit1…..

report_blacklist_t     obstype      bit1     statid       bit1     codetype     bit1     instype      bit1     date         bit1     time         bit1     lat          bit1     lon          bit1     stalt        bit1     scanpos      bit1     retrtype     bit1 …..

status_t

activebit1 passivebit1rejectedbit1blacklistedbit1

….

Passive and blacklisted are a subset of rejected!


Obs dataflow: Screening

  • QC flags, decisions and additional infromation for observed values (in table body):

    • status status_t

    • event1 report_event1_t

    • blacklist datum_blacklist_t

    • fg_depar = obs-guess

datum_event1_t

vertco_missing        bit1      obsvalue_missing    bit1      fg_missing              bit1      rdb_rejected            bit1      rdb_activated           bit1      whitelist_activated    bit1      bad_practice            bit1      vertpos_outrange     bit1      reflevel_outrange      bit1      fg2big                  bit1      depar2big              bit1      obs_error2big           bit1      datum_redundant      bit1      level_redundant         bit1 ….

datum_blacklist_t     obstype      bit1     statid       bit1     codetype     bit1     instype      bit1     date         bit1     time         bit1     lat          bit1     lon          bit1 ….

status_t

activebit1 passivebit1rejectedbit1blacklistedbit1

….


Obs dataflow: Compression

  • After Screening another SHUFFLE is running, this time to perform an ECMA CCMA conversion

  • A CCMA ODB consists of only those tables and column that are needed for the analysis

  • The resulting CCMA ODB after SHUFFLE contains only the active reports and observations!

  • This CCMA ODB is the input of the minimization (e131)

  • The observation data flow is finished!


ECMA (Full)

LAMFLAG

BATOR

SHUFFLE (Reduc)

ECMA.sb1

ECMA.sb2 …

ECMA.sb1

ECMA.sb2 …

SHUFFLE (Merge)

SHUFFLE (Merge)

ECMA (C+I)

Screening

SHUFFLE (ECMA->CCMA)

CCMA

Obs dataflow summary


How to run screening?

  • It is configuration 002:

  • MASTER –c002 –maladin –vmeteo –t001 –aeul –eMIN1

  • Inputs:

    • ECMA ODB (observation, sigma-o, blacklisting)

    • First guess file under 5 different names:

    • ICMSHMIN1INIT,ICMSHMIN1IMIN,ICMRFMIN10000,ELSCFMIN1ALBC000,ELSCFMIN1ALBC

    • Constants, statistics

    • Namelists


Constants

  • errgrib: GRIB file containing grid-point space sigma-B values (lat-lon grid, several levels recieved from MF)

  • rt_coef_atovs_neepred_ieee.dat: coefficients for the RTM (received from MF), binary file

  • rszcoef_fmt.dat: the same the previous file but in ASCII format

  • bcor_noaa.dat: bias correction file

  • chanspec_noaa: obsolete?

  • rmtberr_noaa.dat: obsolete?

  • cstlim_noaa.dat: obsolete?


Screening specific namelists

  • These parameters should be set like this for 3D-VAR:

    • NUPTRA = -1 , number of updates of the trajectory (NAMVAR)

    • NOBSHOR = 201, bilinear horizontal interpolation in H (NAMVAR)

    • LOBS = .TRUE., real obs (NAMCT0)

    • LOBSC1 = .FALSE., (NAMCT0), traj. is not computed

    • LSIMOB = .FALSE., no simelated obs (NAMCT0)

    • LSCREEN = .TRUE., (NAMCT0), screening is on!

    • LSCRE4D = .FALSE. , 3D screening (NAMSCC)

  • Tunable parameters are in NAMSCC and NAMOBS (discussed later)!


How is screening working?

Decisions are taken in subroutine:

Independent decisions: performed separately for each report, RDB flag is assigned to reports

DECIS

[email protected]

[email protected]

Update of flags (FLGTST). Flags are converted into status: active, passive, blacklisted, rejected

Dependent decisions: dependency on other reports or obs, or the previous decisions


Independent decisions

  • Preliminary checks (PRECH): Completeness of report missing station altitude for SYNOP, TEMP and PILOT…

  • Blacklisting (BLACKCLN, BALCKSAT)

  • Background quality control (FIRST)


Blacklisting

  • Blacklisting information is placed into ODB by BATOR

  • In the screening a separate selection is applied for satellite data (BLACKSAT)

  • Blacklisting is also performed in BLACKCLN:

    • Blacklisted obs will be rejected

    • Various blacklisting for SATOB (e.g. QI < 85)

    • Land-sea rejection (SYNOP, TEMP, PILOT). Controlled via NAMOBS LSLREJ (default is .T.)

      • LSLRW10: Wind 10m, LSLRT2: T 2m, LSLRRH2: THU 2m (default is .T. meaning rejection over land. Will get a passive status!)


Blacklisting

  • Also in BLACKCLN:

    • Height-based rejection. Controlled via NAMOBS LHDLREJ (default is .T., rejected obs will get passive status!)

      • Orographic rejection limit is applied:

        • LHDRW10: Wind 10m, LHDRT2: T 2m, LHDRRH2: RHU 2m (default is .T.)

      • AIREP data and significant levels for TEMP and PILOT are not used below the model surface!

      • RHU and Q is rejected above 300 hPa

      • AIREP is not used below a certain pressure


Background QC (FIRST)

  • From the BLUE theory we know:

  • For a given location we expect:

Values are read from file errgrib and interpolated to obs locations

Values are read from ODB


Background QC (FIRST)

  • Flags are assigned to observations by checking:

Based on predefined flaglimits 4 flag values can be assigned to obs

0: correct

1: probably correct

2: probably incorrect

3: incorrect

Flaglimits are defined in DEFRUN


Dependent decisions

  • Remove redundant surface levels from TEMP/PILOT (REDSL)

  • Vertical consistency of profiles (VERCO)

  • Removal of duplicated reports (DUPLI): two reports with the same ID, time an location …

  • Redundancy check (REDUN)

  • Thinning for AIREP (THIAIR)

  • Specific scatterometer quality control

  • Thinning of satellite data (THINN)


Redundancy check (REDUN)

  • Applied for all active reports which are co-located and originate from the same station. Separate treatment for:

    • Land SYNOP: more reports for the same place but with different time, the one with closest to the analysis date with the most active obs are selected

    • Ship SYNOP, DRIBU: redundant if moving platforms are within a circle of 1° radius

    • TEMP and PILOT

    • SYNOP mass observation is redundant if TEMP obs is available and the diff < 50 hPa


Thinning of AIREP data (THIAIR)

  • One flight consist of a set of reports

  • Thinning is performed for each flight separately

    • 3D boxes are constructed around model levels

    • In each box the report closest to the analysis date with the most active observations are selected

    • Box size is set in metres via RFIND_AIREP in NAMSCC, the default is ~70 km!

  • By default box size less than 50 km cannot be used in ARPEGE, code modification is needed


Thinning of satellite data (THINNER)

  • For radiances a horizontal thinning is performed in two steps:

    • First a minimum distance is enforced (default ~ 70 km, can be changed in NAMSCC via RMIND_RAD1C(sensor))

    • Then a repeated thinning is performed to reach the final separation (default ~ 140 km, can be changed in NAMSCC via RFIND_RAD1C(sensor))

  • Where sensor is: 0=HIRS, 1=MSU, 2=SSU, 3=AMSU-A, 4=AMSU-B, 6=SSM/I, 20=METEOSAT, 21=MSG_HR, …

  • Selection criterion:

    • Sea is preferred over land

    • Clear sky pixel is preferred over a cloudy one

    • Obs closer to the analysis date is preferred


Thinning of satellite data (THINNER)

  • For SATOB a the thinning is performed:

    • Horizontally in two steps similarly to radiances: controlled in NAMSCC via RMIND_SATOB, RFIND_SATOB (default is ~70 and ~140 km)

    • Vertically around model levels

  • In each box the report closest to the analysis date with the most active observations are selected

  • QI values are also used in the selection


Sreening output

  • Valuable summary about screening decisions can be found in the NODE file:

  • Try „grep SCREENING STATISTICS” to get:

    • STATUS summary

    • EVENT summary

    • Number of variables, departures and missing departures

    • Diagnostic JO-table


Sreening output

  • Summary at „grep SCREENING STATISTICS”

    • Report and data status summary

OB.TYP REPORTS ACTIVE PASSIVE REJECTED BLACKLISTED 1 725 44 0 681 0 2 0 0 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0 5 61 60 0 1 0 6 61 59 0 2 0 7 0 0 0 0 0 8 0 0 0 0 0 9 0 0 0 0 0 10 0 0 0 0 0 ---------------------------------------------------------------------------------------- TOT 847 163 0 684 0

OB.TYP DATA ACTIVE PASSIVE REJECTED BLACKLISTED 1 4576 164 1846 4412 3862 2 0 0 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0 5 9186 7271 1218 1915 705 6 3438 2050 64 1388 72 7 0 0 0 0 0 8 0 0 0 0 0 9 0 0 0 0 0 10 0 0 0 0 0 ------------------------------------------------------------------------- TOT 17200 9485 3128 7715 4639


Sreening output

  • Summary at „grepSCREENING STATISTICS”

    • Diagnostic Jo table

Obstype 1 === SYNOP, Land stations and ships -------------------------------------------------- Codetype 11 === SYNOP Land Manual Report Variable DataCount Jo_Costfunction JO/n ObsErr BgErr U10 1380 2250.937197524 1.63 0.200E+01 0.000E+00 H2 721 834.1196058351 1.16 0.100E+00 0.000E+00 Z 567 892.5077244762 1.57 0.785E+02 0.000E+00 T2 721 2138.950482967 2.97 0.140E+01 0.000E+00 Q 721 1577.382611689 2.19 0.998E-03 0.000E+00---------- --------------------------- -------- ObsType 1 Total: 4110 7693.897622492 1.87


Sreening output

  • Observation monitoring available at:

  • http://pc2088.met.hu/monitor/start.php



Exercises

  • Run screening for all the prepared observations with script ~/workdir/3dvar/Assim and study the output listing! (You have to put an exit after screening in the script).


Exercises

  • Examine in detail the SYNOP and TEMP report for Budapest, see if surface obs (except Z) are really rejected, and find the reason for it. Try this view first:

    • DEFINE VIEW bp AS

    • SELECT

    • obstype, [email protected],varno,

  • obsvalue, fg_depar, press, [email protected],

  • [email protected], [email protected], [email protected],

  • [email protected], [email protected]

    • FROM hdr, body

    • WHERE (obstype == 1 OR obstype == 5) && statid = ‘12843 ‘

  • Guidance: A prepared viewer par file and sql file can be found in ~wshop01/workdir/OdbViewer as bp.par bp.sql. Copy them to your directory, set the sql path fo your file and type „viewer –p bp.par” to run viewer.

  • If you are brave enough try to write the same view for AIREP (obstype == 2) and explore bit fields of [email protected]!


Exercises

  • Re-run screenig with a 10 km thinning distance for AIREP. To do this you have to use the exe you created yesterday (by modifying arp/obs_preporc/thiair.F90to reduce ISCALE to 1000!) Modify PROC_LAM_ODB and d_RUN in include.inc!

Compare the default and the new airep thinning in the output listings or in the ODB!


ad