1 / 33

Outline

Ensemble b ased Data Assimilation Massimo Bonavita massimo.bonavita@ecmwf.int Contributions from: Lars Isaksen, Elias Holm, Mike Fisher, Laure Raynaud, A. Clayton, D. Barker. Outline. Variational vs Ensemble methods Hybrid methods The Ensemble of Data Assimilations (EDA) method

donna-tran
Download Presentation

Outline

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ensemble based Data AssimilationMassimo Bonavitamassimo.bonavita@ecmwf.intContributions from:Lars Isaksen, Elias Holm, Mike Fisher, Laure Raynaud, A. Clayton, D. Barker

  2. Outline • Variational vs Ensemble methods • Hybrid methods • The Ensemble of Data Assimilations (EDA) method • Applications of the EDA

  3. Variational vs Ensemble • “DA is the process through which all the available information is used to estimate as accurately as possible the state of the atmospheric or oceanic flow” (Talagrand, 1997) • Information comes from observations and the model • Both observations and model are affected by random and systematic errors • DA is thus best described in probabilistic terms • Bayesian statistics offer a convenient probabilistic framework for combining observations and model information

  4. Variational vs Ensemble • In a Bayesian framework the typical DA problem can be described as follows: • “given the prior distributionp(xt-1|y1:t-1) how does a new batch of observations ytchange our estimate at time t? (Filtering problem)” • Two step procedure, described in probabilistic terms: • Compute the background (prior) distribution at time t(forecast step) : • p(xt|y1:t-1) = ∫ p(xt|xt-1) p(xt-1|y1:t-1)dxt-1 • Compute the analysis (posterior) distribution (analysis step) : • p(xt|y1:t) = p(xt|yt,y1:t-1) p(yt|xt,y1:t-1)p(xt|y1:t-1) = p(yt|xt) p(xt|y1:t-1)

  5. Variational vs Ensemble • In general we are not able to obtain analytical forms for the forecast and analysis distributions. What to do? • One idea is to use Monte Carlo methods to try to sample the posterior pdf • In a sequential framework they are known as Particle Filters. • Shown to work effectively in very low-dim. systems, but still problematic for high-dim. geophysical applications: PF requires ensemble sizes which scale exponentially with the system dimensions (Snyder, 2012; Van Leeuwen, 2012; Fisher, DA lecture notes).

  6. Variational vs Ensemble • In the case of Gaussian error distributions and linear model and observation operators an analytical form for the forecast and analysis distributions can be derived: Kalman Filter (KF) • Under these assumptions the KF is the optimal solution of the filtering problem • Assuming Gaussian errors allow us to treat pdfs in terms of mean and covariances • Variational and ensemble methods currently used in operational DA are both approximations and extensions of the standard KF

  7. Variational vs Ensemble • Kalman Filter • Notation: • xt= State vector at time t • yt= Observation vector at time t • Mt-1,t= Model operator from time t-1 to t • Ht= Observation operator, valid at time t • Pt|t= Error covariance of xt|y1:t • Pt|t-1 = Error covariance of xt|y1:t-1 • Qt-1|t = Model error covariance in the (t-1,t] interval • Kt= Kalman Gain matrix

  8. Variational vs Ensemble • Kalman Filter • xt = Mt-1,txt-1 + ηt-1,tη~ N(0,Qt-1,t)(1) • yt = Htxt + εtεt~ N(0,Rt) (2) • Models (1) and (2) give the prior p(xt|xt-1) and data p(yt|xt) • distributions, so that the forecast distribution xt|y1:t-1~ N(xt|t-1,Pt|t-1): • xt|t-1 = Mt-1,txt-1(3) • Pt|t-1= Mt-1|tPt-1|t-1MTt-1|t+Qt-1|t(4) • The analysis distributionxt|y1:t~ N(xt|t,Pt|t)has the form: • xt|t = xt|t-1 + Kt(yt- Ht xt|t-1)(5) • Pt|t= (I - KtHt)Pt|t-1(6) • Kt= Pt|t-1HTt(Rt+ HtPt|t-1HTt)-1 (7)

  9. Variational vs Ensemble • The KF is the optimal solution of the filtering problem for linear, Gaussian systems. • But it is virtually impossible for large-dim. systems: a current NWP system has Nstate~108. In the KF we have to compute and evolve in time error covariances of NxN dimension • Two possible types of practical solutions: • 4D Variational methods • Reduced-rank Kalman Filters

  10. Variational vs Ensemble • 4D Variational methods • If we neglect model error (perfect model assumption) the • problem of finding the model trajectory that best fits the • observations over an assimilation interval (t=0,1,…,T) given a • background state xband its error covariancePbcan be solved by • finding the minimum of the cost function: • This is equivalent, for the samexb, Pb,to the Kalman filter • solution at the end of the assimilation window (t=T) (Fisher et al., • 2005).

  11. Variational vs Ensemble • 4D Variational methods • The 4D-Var solution implicitly evolves background error • covariances over the assimilation window (Thepaut et al.,1996) • with the tangent linear dynamics: • Pb(t) ≈ MPbMT

  12. Variational vs Ensemble MSLP and 500 hPaZ (shaded) background fcst Temperature analysis increments for a single temperature observation at the start of the assimilation window: xa(t)-xb(t) ≈ MPbMTHT(y-Hx)/(σb2 + σo2) t=+0h t=+3h t=+9h

  13. Variational vs Ensemble • 4D Variational methods • The 4D-Var solution implicitly evolves background error covariances over the assimilation window (Thepaut et al.,1996) with the tangent linear dynamics: • Pb(t) ≈ MPbMT • But it does not propagate error information from one assimilation cycle to the next: Pbis not evolved according to KF equations ( i.e., Pb= MPaMT+Q) but is reset to a climatological, stationary estimate at the beginning of each assimilation window. • Only information about the state (xb) is propagated from one cycle to the next.

  14. Variational vs Ensemble • What if we pushed back the start of the assimilation window ‘enough’ so that the filter solution at the end of the window would no longer depend on the specified initial Pb? • How long is enough? 3-5 days in the troposphere for current NWP models, longer in the stratosphere

  15. Variational vs Ensemble • 4D Variational methods • For assimilation windows > 12h it is not accurate to assume the • model to be perfect over the assimilation window. For long • windows we have to add a model error term to our cost function • (Weak-constraint 4D-Var): • Two caveats: • Problem is shifted from estimation of Pb to estimation of Q: this is not easier! • It is difficult in the 4D-Var framework to produce good estimates of Pa: this is important for ensemble prediction!

  16. Variational vs Ensemble • Reduced-rank Kalman Filters • Alternatively we might try to approximate the Kalman Filter update with a low-rank approximation ofPband Pa, i.e.: • Ptb=Xb(Xb)TwhereXb is Nxmandm<<N • It then follows: • K= PtbHT[HPbHT+R]-1 = Xb(HXb)T[(HXb)(HXb)T + R]-1 • Pta=Xb[Imxm + (HXb)TR(HXb)] (Xb)T • Pt+1b= Mt->t+1PtaMTt->t+1+Qt->t+1= • Mt->t+1Xb [Amxm] (Mt->t+1Xb)T+Qt->t+1 • i.e., matrices NxN have been eliminated from the KF • equations! • However…

  17. Variational vs Ensemble Reduced-rank Kalman Filters However… xa-xb = K(y – H(xb)) = Xb(HXb)T [(HXb)(HXb)T + R]-1 (y – H(xb)) It then follows that the analysis increments are a linear combination of the columns of Xb: they are confined to the subspace spanned by Xb, which has rankm<<N

  18. Variational vs Ensemble Reduced-rank Kalman Filters • Reduced-rank KF has become popular also for large-dim. systems with the introduction of the Ensemble Kalman Filter (EnKF, Evensen, 1994; Burgers et al., 1998) • EnKF is a low-rank, Monte Carlo approx. of the KF which crucially does not require the Tangent Linear and Adjoint of MandH. • It is a linear method. It is optimal (as Nens->) for linear, Gaussian systems, but does not assume Gaussianity (Snyder, 2012) • But the subspace spanned by Pbens= 1/(Nens-1) Xb’(Xb’)T , (Xb’are the ensemble perturbations w.r.to the ensemble mean) has still dimensionNens–1 << N (see also M. Fisher lecture slides)

  19. Variational vs Ensemble Reduced-rank Kalman Filters Pbens= 1/(Nens-1) Xb’(Xb’)T , Nens–1 << N There are ways to artificially increase the effective ensemble size (Shur product covariance localization, Hamill and Whitaker, 2001; Local analysis, Evensen, 2003; Ott et al., 2004; adaptive localization, Anderson 2007, Bishop and Hodyss, 2007,2009), but they (too!) come at a price: • Dynamical balance may be degraded; • Asymptotic optimality of the EnKF lost; • More difficult for non-local observations(i.e., observations that have non-local weighting functions; Campbell et al., 2010)

  20. Variational vs Ensemble Results with the ECMWF EnKF Surface Pressure observations only N.Hem. 500 hPa AC

  21. Variational vs Ensemble Results with the ECMWF EnKF Conventional observations only N.Hem. 500 hPa AC

  22. Variational vs Ensemble Results with the ECMWF EnKF All observations N.Hem. 500 hPa AC

  23. Variational vs Ensemble Results with the ECMWF EnKF All observations S.Hem. 500 hPa AC

  24. Variational vs Ensemble Quick recap: • Kalman Filter is computationally unfeasible for operational NWP; • Variational (4D-Var) do not cycle state error estimates: work well for short assimilation windows (6-12h). Longer windows, where Q is required, have proved more difficult; • Reduced rank KF (EnKF) cycle reduced-rank estimates of state error covariances: need for spatial localization to combat rank deficiency, degrades dynamical balance, problematic for non-local observations (radiances); …. Hybrid approach: Use cycled, flow-dependent state error estimates (from an EnKF/Ensemble DA system) in a 3/4D-Var analysis algorithm

  25. Hybrids Hybrid approach: Use cycled, flow-dependent state error estimates (from an EnKF/EDA system) in a 3/4D-Var analysis algorithm This solution would: • Integrate flow-dependent state error covariance information into a variational analysis • Keep the full rank representation of Pb and its implicit evolution inside the assimilation window • More robust than pure EnKF for limited ensemble sizes and large model errors • Allow consistent localization of ensemble perturbations to be performed in state space (advantageous for radiances); • Allow for flow-dependent QC of observations

  26. Hybrids In operational use (or under test), there are currently three main approaches to doing hybrid DA in a VAR context: • Alpha control variable method (Met Office, NCEP/GMAO) • 4D-Ens-Var • Ensemble of Data Assimilations method (ECMWF, Meteo France)

  27. Hybrids: α control variable Conceptually add a flow-dependent term to the model of Pb (B): Bc is the static, climatological covariance Pe○ Clocis the localised ensemble covariance In practice this is done through augmentation of the control variable: and introducing an additional term in the cost function: • Alpha control variable method (Barker, 1999; Lorenc, 2003) from: A.Clayton

  28. Hybrids: α control variable • Alpha control variable method • The increment is now a weightedsum of the static B component and the flow-dependent, ensemble based B • The flow-dependent increment is a linear combination of ensemble perturbations X’, modulated by the α fields • If the αfields were homogeneous δxens could only span Nens-1 degrees of freedom; α fields are then smoothly varying fields, which effectively increases the degrees of freedom • Cloc is a covariance (localization) model for the flow-dependent increments: it controls the spatial variation of α

  29. Pure ensemble 3D-Var 50/50 hybrid 3D-Var Hybrids: α control variable u response to a single u observation at centre of window from: A.Clayton

  30. Hybrids: 4D-Ens-Var • 4D-Ensemble-Var method (Liu et al., 2008) • In the alpha control variable method one uses the ensemble perturbations to estimate Pb only at the start of the 4DVar assimilation window: the evolution of Pb inside the window is due to the tangent linear dynamics (Pb(t) ≈ MPbMT) • In 4D-Ens-VarPbis sampled from ensemble trajectories throughout the assimilation window: from: D. Barker

  31. Hybrids: 4D-Ens-Var • 4D-Ensemble-Var method (Liu et al., 2008) • The 4D-Ens-Var analysis is thus a localised linear combination of ensemble trajectories perturbations: conceptually very close to a pure EnKF • While traditional 4DVar requires repeated, sequentialruns of M, MT, ensemble trajectories from the previous assimilation time can be pre-computed in parallel • Developing and maintaining the TL and Adjoint models requires substantial resources and it is technically demanding: 4D-Ens-Var does not need them

  32. Hybrids: 4D-Ens-Var • However 4D-Ens-Var requires all ensemble trajectories to be stored in memory: increasingly difficult for larger ensemble sizes/resolutions • It is typically more accurate to evolve an initial estimate of Pbby the model TL dynamics than sampling it from an ensemble of trajectories M0,t(xa)-M0,t(xb) M0,t(xa-xb) TL integration Non-linear finite difference from: M. Janiskova

  33. Hybrids: EDA method • Ensemble of Data Assimilations method • To be continued…

More Related