IHOP_2002 DATA MANAGEMENT UPDATE Steve Williams UCAR/Joint Office for Science Support (JOSS)

1. IHOP_2002 DATA MANAGEMENT UPDATE Steve Williams UCAR/Joint Office for Science Support (JOSS) Boulder, Colorado 2nd International IHOP_2002 Science Workshop Toulouse, France 14-17 June 2004

2. IHOP_2002 Field Catalog Statistics http://www.joss.ucar.edu/ihop/catalog/ • Status Reports and Operations Summaries • (600+ documents) • Mission Related Reports and Products • (2700+ documents) • Near Real-Time Operational Image Products • Satellite, Surface, Upper Air, Forecast, and Model • Nearly 200,000 files/images using 20 gigabytes • Report Authoring and Submission Tools • Product Ingest Tools: upload forms, email and ftp • submissions, automated web downloading • Mirror Site in Boulder, CO, with near real-time synchronization • Links to other project related real-time data sources

3. Composites: What and Why A composite dataset is a collection (over some time period and region) of similar data (e.g. surface met) from a variety of sources, put into a common format, and passed through a uniform quality control. Why does UCAR/JOSS develop composites? - Provides data in a uniform format with QC. - Allows determination of network/site problems. - Useful for model applications. - Prevents duplication of effort.

4. Composite Data Sets Developed by UCAR/JOSS for IHOP-2002 • Surface Meteorological - 1 min, 5 min, and Hourly • Precipitation - 15 min, Hourly, and Daily • Soundings - High Resolution and 5 hPa • Complete Station List for all Composites

5. 1-min Surface Meteorological Data Composite (65 stations) ASOS (41) ARM SMOS (14) ABLE AWS (5) NCAR Supp. (5)

6. 5-min Surface Meteorological Data Composite (224 stations) Oklahoma Mesonet (115) NCAR ISS/ISFF (10) West Texas Mesonet (34) 1-min sites (65)

7. Hourly Surface Meteorological Data Composite (743 stations) CoAgMet (*) (23) AWOS (*) (110) RAWS (+) (10) SCAN (o) (6) GWMD (*) (18) Ameriflux (+) (3) HPCN (o) (72) Texas ET (o) (19) KVII-TV (+) (4) MADIS (o) (246) NMSU (*) (7) 5-min sites (o) (225)

8. 15-minute Precipitation Composite NCDC Coop (429) 5-min surface (224)

9. Hourly Precipitation Composite NCDC Coop (509) NCEP hourly (1155) Hourly Met (743)

10. Daily Precipitation Composite NCDC Coop (1581) Hourly Met (743) NCEP Daily (1611)

11. IHOP_2002 Hourly Surface Composite Development KVII CVT Visual ARM CVT HRLY Visual LTER CVT Visual AWOS CVT HRLY Visual Gross Limit Checks ASOS CVT HRLY Visual ABLE CVT HRLY Visual HPCN CVT Visual Horizontal QC MERGE SCAN CVT Visual NMSU CVT Visual Examine Statistics GWMD CVT Visual MADIS CVT Visual Ameriflux CVT Visual Visual NCAR (4) CVT HRLY Visual OKMESO CVT HRLY Visual IHOP_2002 Hourly Surface Composite WTXMESO CVT HRLY Visual Others (7) CVT Visual

12. Gross Limit Checking Procedures • Verify file format. • Data values reasonable for the Earth. • Negative wind speed or precipitation values. • Wind direction between 0o and 360o. • Dew point less than or equal to air temperature. • Precipitation values gross limit checked based on temporal resolution of data (i.e. allow higher precipitation rates at higher temporal resolutions).

13. Horizontal QC Statistical Summary • Overall statistics for all data included in a composite. • Summary statistics for entire networks. • Summaries of the 10 worst performing stations for each network and each parameter. • Summary statistics for each individual station. • Summary statistics for each day. • Summary statistics for each hour. • Summary plots of variance values by parameter. • All occurrences of zero variance noted.

14. What Can Lead to Surface Data Being Flagged? • Instrument failure, bias, drift, random errors • Incorrect latitude, longitude, or elevation • Differences in siting or height of instruments • Misidentified units, dates, or stations • Terrain impacts • Data problems at surrounding stations • Unique meteorological phenomena

15. Example of Applied HQC Data Flags

16. Severe Problems Detected

17. Major Biases Detected

18. Subtle Problems Detected

19. Embedded Accumulative Precipitation

20. High Resolution and 5 hPa Radiosonde Composites Fixed Sites: Mobile Sites: NWS (14) ARM (5) ISS (1) Falcon Drops (86) Learjet (330)NCAR GLASS (146) NSSL CLASS (80)

21. UCAR/JOSS Atmospheric Sounding Processing Procedures Observing System Output e.g. NWS Micro-ART Convert to JOSS Quality-control Calculate Derived Format (ASCII) Parameters e.g., u, v, dz/dt Gross Limit Checks Automated Checks QC Flags Error Files Vertical Consistency Checks Visual Examination QC Flags High-resolution and/or Interpolated Vertical Composite e.g., 5 hPa Inventory, Archive, and Develop Metadata Data and Metadata Available Online

23. IHOP DATA WORK IN PROGRESS….. • Transfer model output files from NOAA/FSL (e.g. new LAPS and WRF model runs) • Transfer LEANDRE Lidar data and imagery files • Transfer new Boundary Layer data/products • Obtain OK Mesonet OASIS data and documentation • Link to or transfer research radar data and imagery (PSU/NSSL) • Complete IHOP Instrument Intercomparison Tables • Transfer additional PI data/products IHOP DATA MANAGEMENT –http://www.joss.ucar.edu/ihop/dm/ Questions – contact Steve Williams (sfw@ucar.edu)

24. FUTURE IHOP DATA WORK? • Produce CD ROM(s)? • Development of IHOP GIS datasets? • Design, Develop, and publish IHOP 4-D Water vapor data set? • Other? IHOP DATA MANAGEMENT –http://www.joss.ucar.edu/ihop/dm/ Questions – contact Steve Williams (sfw@ucar.edu)