Reconstruction of missing data in satellite and in situ data sets with DINEOF

1. Reconstruction of missing data in satellite and in situ data sets with DINEOF A. Alvera-Azcárate, A. Barth, J.-M. Beckers GHER – University de Liège FNRS – National Fund for the Scientific Research, Belgium

2. Characteristics of DINEOF DINEOF: Data Interpolating Empirical Orthogonal Functions • Reconstruction method for gappy data based on an EOF decomposition • Parameter-free, no need of a priori information • resolves time and space • much faster than objective analysis • Determines optimal number of EOFs by cross-validation. Error estimation • Lanczos method for EOF decomposition (Toumazou and Crétaux, 2001): large matrices

3. 2nd: EOF decomposition with N=1 EOF Calculate missing values: Improved guess for missing values EOF decomposition with N=2 EOFs Calculate missing values Improved guess for missing values N EOFs How it works 1st:Demeaned matrix: missing data flagged and set to zero Some data are set aside for cross-validation Convergence: best value for missing data with 1 EOF cross validation: error ● ● error ● ● ● ● ● Then we repeat with N= 3 EOFs and so on…

4. Some Examples SST Wind-induced filaments in Adriatic Sea Alvera-Azcárate et al, 2005 Chlorophyll Mississippi river plume advected by Loop Current

5. Multivariate DINEOF: examples SST + HF-Radar currents ˚C m/s Radial currents: positive values: current towards the antenna negative values: current away from the antenna

6. Time series reconstruction Hourly air temperature at Izas catchment (Pyrenees)‏ ˚C hours hours hours months months

7. Time series reconstruction:original vs. reconstruction Original data, ˚C Reconstruction, ˚C

8. Error maps and outlier detection Error maps are calculated for SST reconstructions using: i) the EOF basis from DINEOF as background covariance ii) the location of valid data Outliers (pixels with value larger than the statistically expected misfit calculated during the analysis) will be objectively identified and removed from initial data

9. Spike removal in temporal EOFs When too few data are present: temporal EOFs poorly constrained: unrealistic discontinuities Sharp transition

10. Spike removal in temporal EOFs We apply a filter to the temporal covariance matrix days days days days • F is a Laplacian filter • Filter on C instead of X: C is much smaller and less sensitive to missing data • Filter applied iteratively: more iterations, further reach of the filter Alvera-Azcárate et al, 2009

11. Spike removal in temporal EOFs

12. Conclusions • DINEOF is a reliable, easy to use technique to reconstruct missing data Widely tested and results validated with in situ data • A variety of data and platforms can be used SST, chlorophyll, TSM, HF-Radar…. • Univariate and Multivariate approaches • Satellite data and time series • Error maps and outliers Generation of error maps to assess the confidence of the reconstruction Outliers (cloud edges, haze...) can be removed from original data • Time filter to remove spikes • Code is freely available; installation instructions; mailing list http://modb.oce.ulg.ac.be/mediawiki/index.php/DINEOF

13. References (chronologic)‏ GHER • J.-M. Beckers and M Rixen. EOF calculations and data filling from incomplete oceanographic data sets. Journal of Atmospheric and Oceanic Technology, 20(12):1839-1856, 2003. • A. Alvera-Azcárate, A. Barth, M. Rixen, and J. M. Beckers. Reconstruction of incomplete oceanographic data sets using Empirical Orthogonal Functions. Application to the Adriatic Sea. Ocean Modelling, 9:325-346, 2005. • J.-M. Beckers, A. Barth, and A. Alvera-Azcárate. DINEOF reconstruction of clouded images including error maps. Application to the Sea Surface Temperature around Corsican Island. Ocean Science, 2(2):183-199, 2006. • A. Alvera-Azcarate, A. Barth, J. M. Beckers, and R. H. Weisberg. Multivariate reconstruction of missing data in sea surface temperature, chlorophyll and wind satellite fields. Journal of Geophysical Research, 112:C03008, 2007. • A. Alvera-Azcárate, A. Barth, D. Sirjacobs, J.-M. Beckers. Enhancing temporal correlations in EOF expansions for the reconstruction of missing data using DINEOF. Ocean Science, 5, 475-485, 2009. • A. Alvera-Azcárate, A. Barth, D. Sirjacobs, F.Lenartz, J.-M. Beckers. Data Interpolating Empirical Orthogonal Functions (DINEOF): a tool for geophysical data analyses. Submitted to Mediterranean Marine Science. 2009. • U. Ganzedo, A. Alvera-Azcarate, G.Esnaola, A. Ezcurra and J. Saenz. 2009. Reconstruction of sea surface temperature by means of DINEOF. A case study during the fishing season in the Bay of Byscay. Accepted in International Journal of Remote Sensing. • D. Sirjacobs, A. Alvera-Azcárate, A. Barth, G. Lacroix, Y. Park, B. Nechad, K. Ruddick, J.-M. Beckers. Cloud filling of ocean color and sea surface temperature remote sensing products over the Southern North Sea by the Data Interpolating Empirical Orthogonal Functions methodology. Accepted in Journal of Sea Research, 2010.

14. References … and other applications: • Y.Z. Ding, Z.H. Wei, Z.H. Mao, X.F. Wang, D.L. Pan. Reconstruction of incomplete satellite SST data sets based on EOF method. Acta Oceanologia Sinica, 28 (2): 36-44 2009. • E. Mauri, P.M. Poulain, Južnič-Zonta, Z. MODIS chlorophyll variability in the northern Adriatic Sea and relationship with forcing parameters. Journal of Geophysical Research, vol 112, C03S11, 2007. • E. Mauri, P.M. Poulain, G. Notarstefano. Spatial and temporal variability of the sea surface temperature in the Gulf of Trieste between January 2000 and December 2006. Journal of Geophysical Research. Vol. 113, No. C10, C10012. 2008. • T.N. Miles, R. He, M. Li. Characterizing the South Atlantic Bight seasonal variability and cold-water event in 2003 using a daily cloud-free SST and chlorophyll analysis. Geophysical Research Letters, 36: Art. No. L02604 JAN 22 2009. • N.P. Nezlin, K. Kamer, J. Hyde, E.D. Stein, ED. Dissolved oxygen dynamics in a eutrophic estuary, Upper Newport Bay, California. Estuarine, Coastal and Shelf Science, 82 (1): 139-151 MAR 20 2009. • R. F. Sánchez, P. Relvas, M. Delgado. Coupled ocean wind and sea surface temperature patterns off the western Iberian Peninsula. Journal of Marine Systems, 68, 103-127. 2007.