WTF-CEOP Status and Plan “Data Mining” and “Data Integration” September 17, 2003

1. WTF-CEOP Status and Plan “Data Mining” and “Data Integration” September 17, 2003 Osamu Ochiai/NASDA Ben Burford/RESTEC Toshio Koike/University of Tokyo

2. Contents CEOP Data WTF-CEOP (“Data Mining”) The science CEOP Data Integration CEOP Interoperability Way Forward

3. - CEOP - Consolidated Enhanced Observation Period. • In-situ data (36 Reference Sites globally) • Satellite Data (200 Tbytes, subset and full scene) • Model Output Data (NWP from 10 Agencies) EOP-1: 1 July 2001 thru 30 September 2001 EOP-2: 1 October 2001 thru 30 September 2002 EOP-3: 1 October 2002 thru 30 September 2003 EOP-4: 1 October 2003 thru 31 December 2004

4. CEOP In-situ Data

5. TRMM MODIS CERES Satellite Data • Subset scenes over Reference Sites • Full scenes over research areas • 200 TBytes • ASTER • MISR • ENVISAT • AQUA • ADEOS-II • ALOS

6. Major National and Multi-National Centers BoM, CPTEC, ECMWF, ECPC, JMA, DAO, GLDAS, NCEP, NCMRWF, UKMO NWP – Numerical Weather Prediction data Max Planck Institute for Meteorology (MPIM) at Hamburg, Germany Model Output Sources

7. Resolution of the data (JMA)(example : Lindenburg) 320*160 (4 grids average ) 3D - 40 levels (heating rates) 288*145 (1.25 x 1.25 degree) 3D - 23pressure levels (fundamental variables) 640*320 (Native Gaussian, T213) 2D (TOA, Surface processes )

8. NWP Assimilation and Forecast Day N Day N+1 00 03 06 09 12 15 18 21 00 03 06 09 12 15 18 21 00 A A A A A A A A A F F F F F F F F F F F F F F F F NCEP Global Model Forecast (12-36 hour) 03 06 09 12 15 18 21 24 27 30 33 36 F F F F F F F F F AssimilationForecast Repeat 12-36 hour forecast 421.6 MBytes/day 153.9 GBytes/year Forecast

9. Model Output Data (1/2) CEOP has requested 82 variables from the providers of Model data u-component at 10 m v_component at 10 m potential temperature at 10 m specific humidity at 10 m snow water equivalent Snow depth vegetation water planetary boundary layer height shortwave downward flux shortwave upward flux longwave downward flux longwave upward flux sensible heating latent heating Snow and frozen ground conversion to soil water meridional wind stress zonal wind stress shortwave downward flux shortwave upward flux longwave upward flux temperature pressure humidity zonal wind meridional wind pressure velocity horizontal kinetic energy (KE) geopotential (gZ) cloud water convective latent heating rate stable latent heating rate convective moistening rate stable moistening rate turbulent moistening rate turbulent heating rate short-wave heating rate long-wave heating rate water vapor zonal flux water vapor meridional flux water vapor vertical flux water vapor flux divergence energy (CpT+gZ+KE) zonal flux energy (CpT+gZ+KE) meridional flux energy (CpT+gZ+KE) vertical flux energy flux divergence local time tendency of temperature local time tendency of KE local time tendency of moisture Local time tendency of cloudwater surface pressure skin temperature 2-meter temperature 2-meter specific humidity

10. Model Output Data (2/2) 10 m turbulent kinetic energy in a layer precipitation (total) precipitation (snow) surface runoff baseflow local surface pressure tendency local skin temperature tendency local snow water equivalent tendency soil moisture soil temperature infiltration rate local soil moisture tendency local temperature tendency subsurface temperature ground Heat Flux base flow runoff Precipitation type 1rain or 2snow elevation Surface albedo station land/sea/ice mask 0(land)or1(sea)or(2)ice Cloud cover aerosol concentration surface exchange coefficient roughness length Vegetation cover Water table (wells) streamflow Stream discharge Reservoir storage • CEOP is mapping these 82 variables to: • the standard GRIB list of variables • the variable names of model data • the variable names of in-situ data • the variable names of MOLTS • the variable names of satellite data This mapping among variable names will be a key component of “Data Integration” for CEOP data!

11. Data Integration “Data Integration” • Users can access all types of data in all locations • Models can be run on data from all areas. • Model output can be compared. • Common formats • Common data types • Consistent data variable names What else can we do to expand “Data Integration”?

12. Validate Soil Moisture Algorithm Input data - AMSR data (6.9 GHz, 18.7 GHz, 36.5 GHz) Validation data - Soil Moisture in-situ data First Get the scenes of AMSR data over a CEOP Reference Site (Mongolia) Data for July, August and September, 2001 CEOP “Data Mining”(WTF-CEOP)

13. MongoliaReference Site AMSR Data (6.9 GHz, 18.7 GHz, 36.5 GHz) Soil Moisture In-situ Data AWS - Automated Weather Station (6) ASSH – Automatic Station for Soil Hydrology (12)

14. Two years of validation Mongolia Reference Site/AMSR Data:2 scenes/day x 365 = 730 scenes of AMSR data (times 3 frequencies = 2190 scenes)2 scenes/day x 365 x 2 yr = 1,460 scenes of AMSR data (extract 17, 520 pixels from 114 GBytes of data)36 Reference Sites: 52,560 scenes (4.1 TBytes of data)

15. WTF-CEOP User Interface CEOP Data Integration Center “Data Analysis System” User Interface Select LocationSelect Time Range Reference Site From: 01 07 2001 From: 30 09 2001 Select Data In-situ Satellite MOLTS Model Output Reference Site Tibet Mongolia Himalayas NSCSSJ etc. Satellite Data TRMM/PR TRMM/TMI ADEOS II /AMSR ADEOS-II/GLI etc. In-situ Data Ps (mb) Ts (degC) RH (%) Td (degC) Mv (%) Ws (m/s) etc. AMSR Data 6.9 GHz 10.7 GHz 18.7 GHz 23.8 GHz 36.5 GHz 89.0 GHz

16. Prepare a “Distributed Data Mining Request” Time Range – July 1 to Sep. 30, 2001. AMSR data (6.9 GHz, 18.7 GHz, 36.5 GHz) Soil Moisture in-situ data Locations (12 sets of latitude/longitude values) Format for Data Mining Request (Use OGC Web Coverage Server (WCS) format?) Data Mining Request

17. NASDA User Interface Module 1. User I/F (menus). 2. Send Data Mining Request. NASA NASDA ESA Data Mining Module 3. Do data mining of Satellite Data. Data Mining Module 3. Do data mining of Satellite Data. Data Mining Module 3. Do data mining of Satellite Data. NASDA Final Module 4. Do data mining of CEOP in-situ data. 5. Send satellite and in-situ data results set to user.

18. 1. Catalog Search for all AMSR scenes over Mongolia (bounding box search) for July 1 to Sep. 30, 2001 Use agency catalog search client (e.g. EDG) Prepare a file with scene times and file names of AMSR data over “Mongolia” 2. Processing - for each scene of AMSR data Open file, geolocate pixels, extract 12 pixels over the Mongolia lat/lon locations. 3. Send pixels, with the times of the satellite scenes, to the Final Module at NASDA. NASDA, NASA, ESASatellite “Data Mining Module”

19. 1. Extract in-situ data Find time of satellite scene (e.g. July 1, 2001 at 01:05). Find closest time of in-situ data (e.g. July 1, 2001 at 01:00). Extract soil moisture values, at 12 ASSH locations, from Mongolia in-situ data. 2. Put satellite scene time, satellite pixel values, in-situ data time and in-situ soil moisture values into Results File (in a convenient format). 3. Send Results to the user. Final Module

20. The Science Picture Develop Algorithms Validate algorithms (Produce product – GRID?) “Virtuous Cycle” of algorithm development “Expand the Picture”

21. Hydrology Research Soil Moisture Snow Land Surface Scheme Snow Physics Model Precipitation CloudPhysicsModel Surface Emissivity & Temp. Microwave Radiometer

23. Precipitation Algorithm

24. Water Cycle – Energy Budget Dry Air-mass Energy Water Vapor Energy convergence/divergence Radio Sonde Profilers Radiation Sensors Eddy Corr. Bowen Ratio Model Output Rt Top of atmosphere Air column AIRS AMSU HSB SSM/T2 HIRS GVAP MODIS GLI CERES SRB ISCCP ERB AMSR AMSRE TMI MODIS (ASAR) (ETM) +4DDA convH+convLQ Hs LEs Rn land surface

25. Water Cycle – Water Budget convergence/divergence Rain Gauge Radar Eddy Corr. Bowen Ratio Model Output Radio Sonde Profilers Tropopause AIRS AMSU HSB SSM/T2 HIRS GVAP AMSR AMSRE TMI SSM/I PR GPCP MODIS (ASAR) +4DDA Air column Pre-GPM divQ divQ P Es Land surface

26. Geolocation and Temporal Integration 60 59 58 57 …………………………. …………………………. …………………………. …………………………. …………………………. …………………………. …………………………. …………………………. …………………………. 100 101 102 103

27. CEOP Data Integration • Common formats • Consistent data types • Consistent variable names With CEOP “Data Mining” • Single user interface (menus) • Geolocated Integration (in-situ, satellite, model) • Temporal Integration (in-situ, satellite, model) • Delivery of exact pixels required (validation)

28. WGCV Land products for validation? Core Site WTF Satellite or land products? NOMADS, Earth System Grid, NERC Data Grid Add Interoperability

29. Problems? Yes!! Example – cloud cover. Put cloud cover selection in menus (“No scenes with cloud cover greater than xx%”)? What cloud metadata do we have? Cloud masks? “Bad Data” values (e.g. 999.99 is bad data) What is acceptable to scientists? What can we do with the metadata? Are we finished yet?

30. Algorithm development Algorithm validation Climate Research model output validation (!!) “Data Integration”/”Data Mining” + GRID – Produce product, other? + WGCV – Data mining of land cover products? + Future missions - GPM Don’t Need Everything! WTF-CEOP is enough! (For a great Plenary Demo)

31. Do we agree to support this process? Agencies determine resources. NASA, ESA, NASDA, Other – WTF-CEOP server! Develop a CEOP science scenario – CEOP scientists + CEOS team (choose sat. data to be supported) Science Scenario [balance] CEOS – Types of data (MODIS is difficult, promote AIRS?) Schedule (CEOS Plenary in 2004?) Write a Project Plan Telecons/Detailed Schedule/Develop in stages?/ . . . Way Forward