A unifying framework for hybrid data-assimilation schemes

1. A unifying framework for hybrid data-assimilation schemes Peter Jan van Leeuwen Data Assimilation Research Center (DARC) National Centre for Earth Observation (NCEO) University of Reading

2. Data assimilation: general formulation Bayes theorem: The solution is a pdf!

3. Incorporating approximate pdfs:proposal densities 4DVar: the prior is Gaussian EnKF: prior assumed to be Gaussian Both: Observation errors are assumed to be Gaussian We know the actual pdfs will be non-Gaussian. However, we can use any other density in Bayes theorem but will have to compensate: in which q(x) is the proposed density, so the Gaussian, and p(x) the real prior.

4. Variational methods A variational method looks for the most probable state, which is the maximum of the posterior pdf also called the mode. Instead of looking for the maximum one solves for the minimum of a so-called costfunction: The pdf can be rewritten as in which Find min J from variational derivative: J is costfunction or penalty function

5. Issues with 4DVar 1) Prior is assumed Gaussian 2) Previous observations inform starting point, not B matrix 3) No posterior error estimate 4) No model errors 5) Pdf cannot be multimodal! 6) Mode of the wrong pdf!!! Possible pdf at time t=0

6. 4DVar gives mode of wrong pdf Nonlinear model Evolved mode at end of window Mode at t=0 4DVar not a natural method for forecasting!

7. Combine the two:Hybrid Methods EnKF x x 4EDnVKaFr: x x 4DVar ?

8. ETKF-4DVar x x x x x x x x An enormous effort to find a good B for the convective scale…

9. Which problem have we attacked? 1) Prior is assumed Gaussian 2) Previous observations inform mode, not B matrix 3) No posterior error estimate 4) No model errors 5) Posterior pdf cannot be multimodal! 6) 4Dvar gives mode of wrong pdf! 7) Extra linearity by replacing ensemble mean by 4Dvar solution 4DEnVar is computationally more efficient but solves non of the red issues above.

10. Ensemble of 4DVars x x x x x x x x x x x x Perturb observations

11. Which problem have we attacked? 1) Prior is assumed Gaussian 2) Previous observations inform mode, not B matrix 3) No posterior error estimate 4) No model errors 5) Posterior pdf cannot be multimodal! 6) 4Dvar gives mode of wrong pdf! 7) Extra linearity assumption by perturbing observations… Long-window 4DVar does not solve any of the red above.

12. Unifying framework: nonlinear filtering Use ensemble with the weights.

13. What are these weights? • The weight is the normalised value of the pdf of the observations given model state . • For Gaussian distributed variables is is given by: • One can just calculate this value • That is all !!! • Or is it? More is needed for high-dimensional problems…

14. Standard Particle filter

15. Particle filter with proposal transitiondensity

16. How to pull particles to observations? • Ensure statistics is ok, we don’t want to change the problem… • Use e.g. simple relaxation • Use EnKF ? • Use 4Dvar, 4DEnVar, EnKS ? • (Plus Equivalent-weight-like step)

17. Use proposal densities: We use a different prior: Use this in Bayes: Generate ensemble from q(x) (e.g. EnKF, or ensemble of 4DVars):

18. Use proposal densities: Use this in Bayes: or with

19. Example: ‘4Dvar’ as proposal x x x x x x

20. Example: ‘4DVar’ as proposal x x x x x x Model errors are essential !

21. What problem did we attack? 1) Prior is assumed Gaussian 2) Previous observations inform mode, not B matrix 3) No posterior error estimate 4) No model errors 5) Posterior pdf cannot be multimodal! 6) 4Dvar gives mode of wrong pdf! 7) Extra linearity assumption by perturbing observations… We now need Q matrix !!! B matrix not needed !!!

22. Unifying framework: proposal densities for Bayes Theorem

23. Conclusions • EnKF, EnKS, ensembles of 4DVars, 4DEnsVar, etc. can be viewed as proposal density samples in a particle filter: nonlinear hybrids! • Weights will diverge if these are used as is. • However, efficient schemes that control the weights are available • Forces the community to work on model errors !!! Finally… • Can reduce B matrix effort • Opens the road to systematic model improvement