1 / 39

Open Source Ensemble Kalman Filtering: the Data Assimilation Research Testbed - DART

Open Source Ensemble Kalman Filtering: the Data Assimilation Research Testbed - DART. Tim Hoar, Jeffrey Anderson, Nancy Collins, Kevin Raeder, Hui Liu, Glen Romine NCAR Institute for Mathematics Applied to Geophysics. 10 years in 12 hours. Sunday Afternoon (4 hours):

ellery
Download Presentation

Open Source Ensemble Kalman Filtering: the Data Assimilation Research Testbed - DART

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Open Source Ensemble Kalman Filtering: the Data Assimilation Research Testbed - DART Tim Hoar, Jeffrey Anderson, Nancy Collins, Kevin Raeder, Hui Liu, Glen Romine NCAR Institute for Mathematics Applied to Geophysics

  2. 10 years in 12 hours • Sunday Afternoon (4 hours): • Introductions / making teams • Configure environment • DART WWW-site • Download DART • DART_LAB – Matlab-based exercises to learn DA • Monday Morning (4 hours): • Recap of yesterday / questions • Select chapters of the DART tutorial • Moving from toy models to large models • Monday Afternoon (4 hours): • Diagnostics • Testing Strategies … 1 observation, please. • Real Observations • CLM CAHMDA-V July 2012 pg 2

  3. Sunday Afternoon • Introductions: I need to know where to start and what to cover. As you introduce yourself, please let me know how much experience you have with each of the following: • Unix/Linux command line • Shell programming (i.e. csh, ksh, sh) • vi / emacs / kedit / … • Matlab / NCL / IDL / NCO / R • Ensemble Data Assimilation Theory • I cannot give individual attention to this many people during these exercises. We will need to make teams of 3 that will need to work together during this tutorial. We need to make teams that have some experience in each of the skills listed above. When I tell you to go stand in a corner, do not take it personally! I mean no disrespect! CAHMDA-V July 2012 pg 3

  4. Configuring your *nix environment on “x50” • Your $HOME directory … • Customizations … ‘dotfiles’ • Compilers, Libraries, inconsistencies … • Your shell … csh, sh, bash, ksh, tcsh … • Your commands … $PATH, aliases … • Your favorite $EDITOR • Remote logins, X forwarding … Windows • Batch jobs • Being nice – lots of people on one machine – “top” • Being graceful – orphaned processes CAHMDA-V July 2012 pg 4

  5. DART “home page” http://www.image.ucar.edu/DAReS/DART • The most useful (to me) pull-down menus: • Getting Started • Documentation • Diagnostics • Miscellany: Platform-specific Notes CAHMDA-V July 2012 pg 5

  6. Download DART • Register for the DART code – really. • Actually download the code • we will cheat • “svn” - making modifications with NO FEAR • DART file tree / schematic • DART documentation • DART tutorial • Review DART interface requirements … • DART build mechanism – mkmf • DART_LAB CAHMDA-V July 2012 pg 6

  7. What is Data Assimilation? Observations combined with a Model forecast… • … to produce an analysis. + = Overview article of DART: Anderson, Jeffrey, T. Hoar, K. Raeder, H. Liu, N. Collins, R. Torn, A. Arellano, 2009: The Data Assimilation Research Testbed: A Community Facility. Bull. Amer. Meteor. Soc., 90, 1283–1296. doi:10.1175/2009BAMS2618.1 CAHMDA-V July 2012 pg 7

  8. Ensemble Filter for Large Geophysical Models 1. Use model to advance ensemble (3 members here) to time at which next observation becomes available. Ensemble state estimate after using previous observation (analysis) Ensemble state at time of next observation (prior) CAHMDA-V July 2012 pg 8

  9. Ensemble Filter for Large Geophysical Models 2. Get prior ensemble sample of observation, y = h(x), by applying forward operator h to each ensemble member. Theory: observations from instruments with uncorrelated errors can be done sequentially. CAHMDA-V July 2012 pg 9

  10. Ensemble Filter for Large Geophysical Models 3. Get observed value and observational error distribution from observing system. CAHMDA-V July 2012 pg 10

  11. Ensemble Filter for Large Geophysical Models 4. Find the increments for the prior observation ensemble (this is a scalar problem for uncorrelated observation errors). Note: Difference between various ensemble filters is primarily in observation increment calculation. CAHMDA-V July 2012 pg 11

  12. Ensemble Filter for Large Geophysical Models 5. Use ensemble samples of y and each state variable to linearly regress observation increments onto state variable increments. Theory: impact of observation increments on each state variable can be handled independently! CAHMDA-V July 2012 pg 12

  13. Ensemble Filter for Large Geophysical Models 6. When all ensemble members for each state variable are updated, there is a new analysis. Integrate to time of next observation … CAHMDA-V July 2012 pg 13

  14. DART_LAB: Matlab-based tutorial • DART/DART_LAB/presentations • Section 1 … the 1D perspective • Section 2 … impacting an unobserved state variable • Section 3 … sampling error and localization • Section 4 … perturbed observations (EnKF) • DART/DART_LAB/matlab • Section 1 … gaussian_product, oned_ensemble, oned_model • Section 2 … twod_ensemble, run_lorenz_63, run_lorenz_96 • Section 3 … run_lorenz_96 • Section 4 … oned_ensemble, twod_ensemble, oned_model, run_lorenz_63 and run_lorenz_96 all allow selection of EnKF. I’m going to focus on CAHMDA-V July 2012 pg 14

  15. Ensemble Filter for Large Geophysical Models A generic ensemble filter system like DART just needs: 1. A way to make model forecasts; 2. A way to compute forward operators, h. CAHMDA-V July 2012 pg 15

  16. That’s all for today …. CAHMDA-V July 2012 pg 16

  17. Monday morning • Introductions and lies … which of these is true about me? • “I have seen all 7 continents.” • “I am a competitive square dancer.” CAHMDA-V July 2012 pg 17

  18. Monday morning • Questions from yesterday • netCDF : ncdump, ncview • Matlab customizations for DART … • DART tutorial (in the interest of time, we’re skipping a lot – at your leisure go back and be complete) • Sections 1,2,3 in their entirety (but quickly) • Section 4: skip to pg 29 • Section 5: only pg 15 • Section 7: introduces lorenz_96 (L96) pg 10 • Section 8: sampling error (L96) pgs 10,13 • Section 9: inflation (L96) after pg 15 • Section 11: building DART • Section 14: observation quality control • Section 18: not knowing the truth • Scripting for standalone executables – large models. CAHMDA-V July 2012 pg 18

  19. Monday Afternoon • Diagnostics • State-space (useful if you know the truth) • Observation-space (useful in general) • link_obs example with dev/POP • obs_diag example wth dev/WRF • Testing Strategies • Lorenz_96 – perfect model experiment • Observation sequence file creation • Ameriflux data • CLM/DART • CESM multi-instance facility • Modifying the run script • CLM variable specification – picking a state vector • Adding new DART kinds/types • Observation operators CAHMDA-V July 2012 pg 19

  20. Reasons to NOT reinvent the wheel • DART has proven methods to address the most common (and some not-so-common) issues affecting the performance of ensemble filters: • Inflation Anderson, J. L., 2009 Spatially and temporally varying adaptive covariance inflation for ensemble filters. Tellus A,61, 72-83 doi:10.1111/j.1600-0870.2008.00361.x Anderson, J. L., 2007. An adaptive covariance inflation error correction algorithm for ensemble filters. Tellus A, 59, 210-224. doi:10.1111/j.1600-0870.2006.00216.x • Novel algorithms Anderson, J.L., 2010 A Non-Gaussian Ensemble Filter Update for Data Assimilation. Monthly Weather Review, 138, 4186-4198, doi:10.1175/2010MWR3253.1 Anderson, J. L., 2007 Exploring the need for localization in ensemble data assimilation using a hierarchical ensemble filter. Physica D, 230, 99-111.  doi:10.1016/j.physd.2006.02.011 • Parallelization Anderson, J., Collins, N., 2007. Scalable Implementations of Ensemble Filter Algorithms for Data Assimilation. Journal of Atmospheric and Oceanic Technology, 24, 1452-1463. doi:10.1175/JTECH2049.1 CAHMDA-V July 2012 pg 20

  21. More reasons to NOT reinvent the wheel • Diagnostics • Many routines/methods are provided to explore the performance of the assimilation. • Native ability to explore ‘value’ of different types of observations • Immediate ability to perform ‘perfect model’ experiments • Documentation • Each code module has a companion HTML document to describe its use and purpose. • http://www.image.ucar.edu/DAReS/DART • All documentation/code available online • Workshop materials • Self-paced tutorials included in the download • Portable, tested on many platforms • Free, open source. • Too many platforms/compilers to bother listing. • Distributed and maintained with subversion. • Can exploit, but does not need MPI. • Humans! • Tim, Nancy, Jeff, Kevin, Hui, Glen …all reached at dart@ucar.edu CAHMDA-V July 2012 pg 21

  22. Creating the initial ensemble of CLM. Replicate what we have N times. Use a unique (and different!) realistic DATM for each. Run them forward for “a long time”. model time “spun up” Getting a proper initial ensemble is an area of active research. “a long time” We don’t know how much spread we NEED to capture the uncertainty in the system. CAHMDA-V July 2012 pg 22

  23. The ensemble advantage. You can represent uncertainty. In a free run, the ensemble spread frequently grows. With a good assimilation: ensemble spread ultimately remains stable and small enough to be informative observation times CAHMDA-V July 2012 pg 23

  24. Atmospheric Reanalysis Assimilation uses 80 members of 2o FV CAM forced by a single ocean (Hadley+ NCEP-OI2) and produces a very competitive reanalysis. O(1 million) atmospheric obs are assimilated every day. 1998-2010 4x daily is free and available. Contact dart@ucar.edu 500 hPa GPH Feb 17 2003 Contours 5200m:5700m by 100 CAHMDA-V July 2012 pg 24

  25. Code to implement all of the algorithms discussed is freely available from: http://www.image.ucar.edu/DAReS/DART dart@ucar.edu CAHMDA-V July 2012 pg 25

  26. DART is used at: 43 UCAR member universities More than 100 other sites • Public domain software for ensemble Data Assimilation • Well-tested, portable, scalable, extensible, free! • Models • Toy to HUGE • Observations • Real, synthetic, novel • An extensive Tutorial • With examples, exercises, explanations • People: The DAReS Team

  27. Moving towards coupled assimilationfor earth system models. Tim Hoar, Nancy Collins, Kevin Raeder, Jeffrey Anderson, NCAR Institute for Math Applied to Geophysics Data Assimilation Research Section Steve Yeager, Mariana Vertenstein, Gokhan Danabasoglu, Alicia Karspeck, and Joe Tribbia NCAR/NESL/CGD/Oceanography

  28. Hypothesis: Need Ensemble of Atmospheres to Force Ensemble Assimilation for Ocean 500 hPa GPH Feb 17 2003 • Case 1: 23 POP members forced by a single atmosphere. • Case 2: 48 POP members forced by 48 CAM/DART analyses. • Case 2 Generates additional ocean spread, improved analyses.

  29. DATM 2D forcing from CAM assimilation DART Obs 3D state 3D restart POP 2D forcing Current POP Assimilation from within the Climate Earth System Model - CESM Coupler

  30. World Ocean Database T,S observation counts These counts are for 1998 & 1999 and are representative. FLOAT_SALINITY   68200               FLOAT_TEMPERATURE 395032             DRIFTER_TEMPERATURE 33963               MOORING_SALINITY   27476              MOORING_TEMPERATURE  623967                 BOTTLE_SALINITY   79855             BOTTLE_TEMPERATURE   81488                     CTD_SALINITY  328812                   CTD_TEMPERATURE  368715                     STD_SALINITY     674                  STD_TEMPERATURE     677                   XCTD_SALINITY    3328                 XCTD_TEMPERATURE    5790                 MBT_TEMPERATURE   58206                  XBT_TEMPERATURE 1093330                 APB_TEMPERATURE  580111 • temperature observation error standard deviation == 0.5 K. • salinity observation error standard deviation == 0.5 msu.

  31. Ensemble Spread for Pacific 100m XBT Spread of the “climatological” ensemble Twice as much! Small spread!

  32. 100m Mooring Temperature RMSE – Pacific POP/CAM as good or better RMSE

  33. Physical Space: 1998/1999 SST Anomaly from HadOI-SST POP forced by observed atmosphere (hindcast) Coupled Free Run 48 POP 48 CAM 23 POP 1 DATM

  34. Fully coupled assimilation will need data from all models at the same time DART Obs CAM CICE POP Coupler CLM This is a very CESM-centric view of fully coupled data assimilation.

  35. DART works with many geophysical models Global Atmosphere models: CAM Community Atmosphere Model NCAR CAM/CHEM CAM with Chemistry NCAR WACCM Whole Atmosphere Community NCAR Climate Model AM2 Atmosphere Model 2 NOAA/GFDL NOGAPS Navy Operational Global US Navy Atmospheric Prediction System ECHAM European Centre Hamburg Model Hamburg Planet WRF Global version of WRF JPL MPAS Model for Prediction Across NCAR/DOE Scales (under development)

  36. DART works with many geophysical models Regional Atmosphere models: WRF/ARW Weather Research and NCAR Forecast Model WRF/CHEM WRF with Chemistry NCAR NCOMMAS Collaborative Model for NOAA/NSSL Multiscale Atmospheric Simulation COAMPS Coupled Ocean/Atmosphere US Navy Mesoscale Prediction System CMAQ Community Multi-scale Air Quality EPA COSMO Consortium for Small-Scale DWD Modeling

  37. DART works with many geophysical models Ocean models: POP Parallel Ocean Program DOE/NCAR MIT OGCM Ocean General Circulation MIT Model ROMS Regional Ocean Modeling Rutgers System (under development) MPAS Model for Prediction Across DOE/LANL Scales (under development) Land Surface models: CLM Community Land Model NCAR (under development)

  38. DART works with many geophysical models Upper Atmosphere/Space Weather models: ROSE NCAR TieGCM Thermosphere Ionosphere NCAR/HAO Electrodynamic GCM GITM Global Ionosphere Thermosphere Model Michigan

More Related