Practice of Quality Control

1. Practice of Quality Control Dick Dee Global Modeling and Assimilation Office NASA Goddard Space Flight Center NCAR Summer Colloquium 2003 NCAR Summer Colloquium 2003

2. Outline • Motivation • QC procedures • The background check • The buddy check • An adaptive buddy check algorithm • The Bayesian framework • Variational quality control • Summary NCAR Summer Colloquium 2003

3. QC Example 1: Rotated earth scenario NCAR Summer Colloquium 2003

4. QC Example 2: Strange sat winds NCAR Summer Colloquium 2003

5. QC Example 3: French Christmas Storm No. 2 NCAR Summer Colloquium 2003

6. Quality Control Procedures • At the instrument site: • E.g. radiation correction for rawinsonde temperatures • During the retrieval process: • E.g. cloud-track wind height assignment • As part of preprocessing at the DAS site: • E.g. aircraft wind checks • E.g. hydrostatic checks for rawinsonde temperatures • During the assimilation: Statistical quality control • Background reading: Some of the early papers in numerical weather map analysis: Bergthórsson and Döös 1955; Bedient and Cressman 1957 More recent papers with a good general discussion of QC: Lorenc and Hammon 1988; Collins and Gandin 1990 NCAR Summer Colloquium 2003

7. Statistical Quality Control • Since this takes place late in the data assimilation process, a lot of information is at hand: • Observations from various instruments • A short-term forecast valid at the time of the observations • Some information about expected errors • Basic idea: check if each observed value is reasonable in view of all other available information • Danger: rejecting good data / including bad data This is clearly a problem in probability theory.. NCAR Summer Colloquium 2003

8. Background check • Bergthórsson and Döös 1955; Bedient and Cressman 1957 • Compare each observation against its prediction based on first-guess fields (e.g. interpolated background) • Flag or reject the observation if the difference is large (but what is large?) Example: rawinsonde observed-minus-forecast temperature residuals NCAR Summer Colloquium 2003

9. Definitions: observations background data residuals In terms of errors: errors for ‘good’ data Assumptions: background errors Therefore in the absence of gross errors. For each single residual, the null hypothesis is for some fixed tolerance  Reject the hypothesis if Probability of false rejection: The background check as a hypothesis test NCAR Summer Colloquium 2003

10. Traditional buddy check • Identify a suspect observation (e.g. using a background check) • Define a set of buddies(e.g. based on distance, data type) • Predict the suspect from the buddies (e.g. using local OI) • Reject the suspect observation if it is too far from the predicted value (based on error statistics) • See: Lorenc 1981 NCAR Summer Colloquium 2003

11. Null hypothesis H0: Divide intosuspectsandbuddies: Given H0, the conditional pdf of the suspects given the buddies is where Let for some fixed tolerance  Reject the null hypothesis if The choice of  determines the significance level δ of the test, which bounds the probability of false rejection of the null hypothesis: The buddy check as a hypothesis test NCAR Summer Colloquium 2003

12. Illustration of the buddy check NCAR Summer Colloquium 2003

13. An adaptive buddy check algorithm Loop: identify suspects predict suspects from buddies prediction error covariances adjust the error estimates null hypothesis: End loop NCAR Summer Colloquium 2003

14. Illustration with fixed tolerances suspect observations true range (μ ± 2σ) expected range predicted suspects acceptable discrepancy rejected observations NCAR Summer Colloquium 2003

15. Illustration with adaptive tolerances adjusted range adjusted range NCAR Summer Colloquium 2003

16. Illustration with real data Fixed tolerances Adaptive tolerances NCAR Summer Colloquium 2003

17. Some remarks on the adaptive buddy check Very little dependence on prescribed error statistics in densely observed regions … but reverts to a simple background check for isolated observations Not effective for detecting systematic gross errors (coherent batches of bad data) Cheap and simple to implement, although parallel implementation takes some care Does not incorporate prior information about instrument reliability … but that can be done, following Lorenc and Hammon (1988) Quality control and analysis are treated as separate steps in the assimilation process The analysis is not a smooth function of the observations NCAR Summer Colloquium 2003

18. The Bayesian framework (1) We can formulate the analysis problem in terms of conditional probabilities: For example, our earlier Gaussian error models: can also be written as See:Lorenc 1986, Cohn 1997 NCAR Summer Colloquium 2003

19. Example: Gaussian distributions Lorenc and Hammon (1988) NCAR Summer Colloquium 2003

20. Actually we’d be happy with just the mode of the conditional pdf: This represents the most likely state in view of the available information. For Gaussian distributions, When h(x) is linear, J(x) is quadratic and the solution is with The Bayesian framework (2) The Bayesian framework is not restricted to Gaussian distributions and/or linear operators. NCAR Summer Colloquium 2003

21. Error models that account for bad data Generalize the observation error model to account for possible gross errors: If G is the event that a gross error occurred, then: and This is no longer a Gaussian pdf, and the variational problem becomes non-linear. See: Purser 1984, Lorenc and Hammon 1988. NCAR Summer Colloquium 2003

22. Example: Non-Gaussian observation errors Lorenc and Hammon (1988) NCAR Summer Colloquium 2003

23. Variational Quality Control at ECMWF (1) Minimize cost function Assuming independent Gaussian errors, the contribution of a single observation is (cost) (gradient) After modification of p(y|x) to account for gross errors we have instead where and It turns out that is the a posteriori prob. of gross error NCAR Summer Colloquium 2003

24. Example: Impact of an observation in VarQC Andersson and Järvinen(1999) NCAR Summer Colloquium 2003

25. Some remarks on variational QC Implementation for observations with correlated errors is much more complicated Incorporates prior information about instrument reliability Quality control and analysis are done simultaneously – each can take advantage of iterative improvement during the optimization In principle, the analysis is a smooth function of the observations … but not really (multiple minima) Requires a relatively strict background check to avoid convergence issues Strong dependence on prescribed error statistics Not effective for detecting systematic gross errors (coherent batches of bad data) NCAR Summer Colloquium 2003

26. Summary NCAR Summer Colloquium 2003

27. Literature • Andersson, E., and H. Järvinen, 1999: Variational quality control. Quart. J. Royal Meteor. Soc., 125, 697-722 • Bedient, H. A., and G. P. Cressman, 1957: An experiment in automatic data processing. Mon. Wea. Rev., 85, 333-340. • Bergthórsson, P., and B. R. Döös, 1955: Numerical weather map analysis. Tellus, 7, 329-340 • Collins, W. G., 1998: Complex quality control of significant level rawinsonde temperatures. J. Atmos. Ocean. Tech., 15, 69-79. • Collins, W. G., and L. S. Gandin, 1990: Comprehensive hydrostatic quality control at the National Meteorological Center. Mon. Wea. Rev., 118, 2752-2767 • Dee, D. P., L. Rukhovets, R. Todling, A. M. da Silva, and J. W. Larson, 2001: An adaptive buddy check for observational quality control. Quart. J. Royal Meteor. Soc., 114, 2451-2471. • Dharssi, I., A. C. Lorenc, and N. B. Ingleby, 1992: Treatment of gross errors using maximum probability theory. Quart. J. Royal Meteor. Soc., 118, 1017-1036 • Gandin, L. S., 1988: Complex quality control of meteorological observations. Mon. Wea. Rev., 116, 1137-1156 • Ingleby, N. B., and A. C. Lorenc, 1993: Bayesian quality control using multivariate normal distributions. Quart. J. Royal Meteor. Soc., 119, 1195-1225. • Lorenc, A. C., 1981: A global three-dimensional multivariate statistical interpolation scheme. Mon. Wea. Rev., 109, 701-721. • Lorenc, A. C., and O. Hammon, 1988: Objective quality control of observations using Bayesian methods: Theory, and a practical implementation. Quart. J. Royal Meteor. Soc., 114, 515-543. • Purser, R. J., 1984: A new approach to the optimal assimilation of meteorological data by iterative Bayesian analysis. Proceedings of 10th Conf. On Weather Forecasting and Analysis, American Meteorological Society, Boston, 102-105. NCAR Summer Colloquium 2003