INTERGRATING IN SITU NETWORKS STANDARDIZING DATA QUALITY ASSURANCE

1. INTERGRATING IN SITU NETWORKS & STANDARDIZING DATA QUALITY ASSURANCE INTERGTATED SURFACE DATA PROCESSING SYSTEM Stephen Del Greco National Climatic Data Center

3. HISTORY Current In-Situ Networks are processed with Quality Assurance/Control, (QA/QC�d) based on network, not element For example; The NOAA Automated Surface Observing System (ASOS), Network Volunteer Cooperative Observers Network (COOP) & Climate Reference Network (CRN) are processed using systems unique to the individual network This is also the case for the Global Climate Observing System Network (WMO) as well as other various Meso networks owned and operated by local and state governments or the military The global Integrated Surface Hourly dataset, integrated from several sources has a unique QA/QC system. Often current final QA/QC is performed when all data for the month are received

4. OBJECTIVE Today�s climate related issues dictate the need for high quality homogenous data sets, often in near real time Towards that end, NCDC is designing new QA/QC processing systems to replace existing systems The goal is to move towards fully automated QA/QC validation for hourly/daily/monthly weather data, keeping in mind that interactive QC will take place when situations warrant The new system processes data on a daily or hourly basis instead of the current end of month time scale Any QA/QC standardization needs collaboration among NOAA line offices, State and Regional Climate Centers (HPRCC, linear regression, FSL mesonets)

6. OBJECTIVE While similar, QA/QC rules and algorithms for like parameters from different observing networks are not standard Current technology provides a means for integrating like data into one standard format and processing these data through standardized QA/QC algorithms & procedures NCDC is designing a new QA/QC system - Integrated Surface Data Processing System (ISDPS) ISDPS is an end to end system for processing in-situ data where QA/QC is network independent and based on reporting frequency (hourly, daily, etc..) ISDPS integrates assessment techniques and links algorithms into one unified system (ISD input/output)

9. BENEFITS OF INTEGRATION Reduction of subjectivity and inconsistencies among data sets that span multiple observing networks and platforms Standardize QA/QC based on reporting time (QC methodology for hourly temp data independent of network) Standardized products are more easily developed Collective experience and expertise leads to a better product Software is modular for ease of modification Conformance of data to documentation (reference manuals, FMH, etc.)

10. NCDC CONSISTENCY CHECKS Mathematical and meteorological Maxima > minima with other values between maximum and minimum Physically plausible ranges Physically plausible combinations of data Extremes (Wakeby probability model, empirical curve fitting, statewide extreme) Source-specific rules Original vs. original, replacement vs. original, replacement vs. replacement Spatial checks (TempVal, PrecipVal)

12. ASOS Hourly QC Algorithms (Examples)

13. ASOS Hourly QC Algorithms (Examples)

14. Cooperative Observers Network

15. Cooperative Observers Network

16. Cooperative Observers Network

17. The Quality Control of the Integrated Surface Hourly (ISH) Global Database 2 QC phases thus far ISH is being renamed ISD � Integrated Surface Data, to reflect its usage in the Integrated Surface Data Processing System, and inclusion of hourly, daily, etc data ISH Quality Control Phase 1 Data comparisons as part of merge process to ensure records are for same station-time Inventory system to ensure no data loss from beginning to end of process Test data to verify all software is working properly Random checks of �Version 1� output data Reprocessing as necessary

18. ISH Quality Control Phase 2 57 quality control algorithms (fully automated) Validity checks for all data fields Extreme value checks to eliminate gross errors Temporal continuity checks (2-sided) for elements such as temperature, dew point, wind, & pressure

19. ISH Quality Control Phase 2 Internal consistency checks between elements Checks for several known systematic problems Test data verification Random checks of �Version 2� output data ISH Lessons Learned Thorough test data are critical, and allow for the establishment of baselines for future comparison Peer review is important�don�t be an �island� Use phased approach�don�t insist on the �big house� initially; start with the foundation Have a long-term plan, but be flexible Expect to reprocess, but limit its frequency by following good design and testing principles

20. FULLFILLING USER NEEDS Unedited Local Climatological Data publication replaced with on-line final Climatological Data publication dynamically updated daily Today�s end user need data that is processed daily instead of the current end of month time scale High quality data that in the past were available 45 to 90 days after the end of the month will be available within days after receipt Products based on preliminary or unedited data to be replaced with daily QC�d data on line (automated QC) ISDPS will provide capability for dissemination of surface data into legacy data formats (ASOS, HPD, COOP, Global daily, hourly)

21. FULLFILLING USER NEEDS Complete a distributed network for both data and software Integrate real-time and historical data so that any data can be provided to users within servicing time constraints