1 / 23

RAINEX Data Management

RAINEX Planning Meeting in Miami, FL focusing on data management considerations, services, and tools. Collaboration with PIs, consistent implementation of data strategy, and easy access to datasets for the broader community.

jsykes
Download Presentation

RAINEX Data Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. RAINEX Planning Meeting, Miami, FL RAINEX Data Management UCAR Joint Office for Science Support José Meitín Jim Moore Dick Dirks

  2. RAINEX Planning Meeting, Miami, FL JOSS Data Management • Early involvement in project planning • Involvement with PIs to develop data management strategy • Consistent implementation of strategy for lifetime of project and beyond • Reliable and efficient archive and distribution system • Easy and efficient access to datasets by broader community UCAR Office Of ProgramsJoint Office for Science Support

  3. RAINEX Planning Meeting, Miami, FL Data Management Considerations • Develop a consistent Data Policy • Consider Data Types Formats and Documentation (metadata) • Data Collection strategies (operations support, supplementary data) • Real-time Requirements ( data calibration, intercomparison, and needs for operational decision making) • Data Quality Control and Processing responsibilities • Data Archive structure, functions and responsibilities • Data Distribution protocols (user interface, metadata inventories, browse products) • Collaboration with other programs and data centers (interoperability)

  4. RAINEX Planning Meeting, Miami, FL

  5. RAINEX Planning Meeting, Miami, FL JOSS Data Services • Data Questionnaire • Data Management Plan • Project Web Pages • Real-time Operations and Data Ingest • Field Data Catalog • Quality control and dataset development • Interactive Data Archive and Distribution • Special media products (e.g. CD-Roms) UCAR Office Of ProgramsJoint Office for Science Support

  6. RAINEX Planning Meeting, Miami, FL JOSS Data Services Data Questionnaire UCAR Office Of ProgramsJoint Office for Science Support

  7. RAINEX Planning Meeting, Miami, FL JOSS Data Services • Project Web Pages

  8. RAINEX Planning Meeting, Miami, FL JOSS Data Services • Real-time Operations Support and Data Ingest • Satellite Ground Station • Forecast Products • Numerical Model Analyses • Operational Data Streams UCAR Office Of ProgramsJoint Office for Science Support

  9. RAINEX Planning Meeting, Miami, FL GTS METAR Observation Locations 6-10 November 2004

  10. RAINEX Planning Meeting, Miami, FL GTS Ship and Buoy Observation Locations 6-10 November 2004

  11. RAINEX Planning Meeting, Miami, FL Coastal and Marine Observation Locations * NDBC Buoy o TCOON (TX) * CMAN * TABS (TX) o NWLON + LUMCON (LA) o PORTSo WAVCIS (LA) + COMPS (FL)

  12. RAINEX Planning Meeting, Miami, FL Land Observation Locations * ASOS o LAIS (LA) * AWOS + TX ET (TX) * AWOS modem o FAWN (FL) o GAEMN (GA) + TNRCC (TX) + GFC (GA)o ALMESO (AL)

  13. RAINEX Planning Meeting, Miami, FL • JOSS Data Services • Field Data Catalog (WWW based) • Daily Mission Reports • Operations Summary • Status Reports • Data Analysis Products • Authoring Tools UCAR Office Of ProgramsJoint Office for Science Support

  14. RAINEX Planning Meeting, Miami, FL • JOSS Data Services • Quality control and dataset development • Processing of research-quality, high resolution data • Processing of standard operational data streams • Compatible formats • Composite datasets and data integration UCAR Office Of ProgramsJoint Office for Science Support

  15. RAINEX Planning Meeting, Miami, FL Composite Data Sets at UCAR/JOSS A composite dataset is a collection (over some time period and region) of similar data (e.g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control. Why does UCAR/JOSS develop composites? - Provides data in a uniform format with QC. - Allows determination of network/site problems. - Useful for model applications. - Prevents duplication of effort.

  16. RAINEX Planning Meeting, Miami, FL KVII CVT Visual IHOP_2002 Hourly Surface Composite Development ARM CVT HRLY Visual LTER CVT Visual AWOS CVT HRLY Visual Gross Limit Checks ASOS CVT HRLY Visual ABLE CVT HRLY Visual HPCN CVT Visual Horizontal QC MERGE SCAN CVT Visual NMSU CVT Visual Examine Statistics GWMD CVT Visual MADIS CVT Visual Ameriflux CVT Visual Visual NCAR (4) CVT HRLY Visual OKMESO CVT HRLY Visual Hourly Surface Composite WTXMESO CVT HRLY Visual Others (7) CVT Visual

  17. RAINEX Planning Meeting, Miami, FL Aircraft Flight Level Datasets NCAR / RAF NetCDF NOAA / AOC “standard” format JOSS RAINEX Archive JOSS convert “type 5” to NetCDF Integrated RAINEX NetCDF

  18. RAINEX Planning Meeting, Miami, FL JOSS DATA MANAGEMENT TOOLS • Field Catalog • In-field tool to ingest and display operational and preliminary research data and project documentation for making real-time decisions and evaluating project progress • Features: • Daily Mission Reports • Operations Summary • Facility Status Reports • Data Analysis Products • Authoring Tools • Web-based access • Interactive Data Archive/Access • System • Primary means for all project scientists and researchers to browse and retrieve data from any JOSS-supported projects • Features: • Long-term field project data archival and distribution • Interactive data browsing, subsetting, and format conversion • Remote Archive Access • Value-added datasets • Data documentation

  19. RAINEX Planning Meeting, Miami, FL • JOSS Data Services • http://www.joss.ucar.edu/codiac/ • Interactive Data Archive and Distribution • Ease of Data Access (web based) • Data available via FTP, magnetic tape, CD-Rom • Metadata descriptions • Data Browse, Subsetting, Format Translation Capabilities • Quality Control • Distributed Data Archives • Customized data ordering and reformat UCAR Office Of ProgramsJoint Office for Science Support

  20. RAINEX Planning Meeting, Miami, FL • RAINEX Data Management Issues: • Data Policy • Data Management Plan • Real Time Data Needs (model, satellite) • Distributed Archive Responsibilities • Special Data Processing (surface, upper-air, airborne radar aircraft flight-level)

  21. RAINEX Planning Meeting, Miami, FL http://www.joss.ucar.edu/ meitin@ucar.edu jmoore@ucar.edu dirks2@ucar.edu UCAR Office Of ProgramsJoint Office for Science Support

  22. Acronyms PORTS – Physical Oceanographic Real-Time System NWLON – National Water Level Observation Network TCOON – Texas Coastal Ocean Observation Network (TAMU – Corpus) TABS – Texas Automated Buoy System (TAMU) LUMCON – Louisiana Universities Marine Consortium WAVCIS – Wave Current Surge Information System (LSU) COMPS – Coastal Ocean Monitoring and Prediction System (USF) GAEMN – Georgia Automated Environmental Monitoring Network (U of Georgia) GFC – Georgia Forestry Commission ALMESO – Alabama Mesonet (Auburn U) LAIS – Louisiana Agriclimatic Information System (LSU) TX ET – Texas Evapotranspiration Network (TAMU) FAWN – Florida Automated Weather Network (U of Florida) TNRCC – Texas Natural Resources Conservation Commission

More Related