1 / 32

GSICS Data Management Working Group

GSICS Data Management Working Group. 15 th Session of Executive Panel Guangzhou, China. Manik Bali(Chair), Aleksandar Jelenak , Rob Robelling , Peter Miu , Masaya Takahashi, Byung -II NOAA Satellite and Information Service. 16 May 2014. Outline. Introduction

tausiq
Download Presentation

GSICS Data Management Working Group

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. GSICS Data Management Working Group 15th Session of Executive Panel Guangzhou, China Manik Bali(Chair), AleksandarJelenak, Rob Robelling, Peter Miu, Masaya Takahashi, Byung-II NOAA Satellite and Information Service 16 May 2014

  2. Outline • Introduction • GDWG Membership News. • GDWG –Progress in Action Items . • GSICS Communications • GSICS user messaging service. • GSICS Dev server. • Review of activities since the last meeting. • NOAA. JMA, EUMETSAT • CMA, KMA • Roadmap-2014 • Summary/Conclusions. Progress Past 1 Yr Future 15th Session of Executive Panel Guangzhou, China 16 May 2014

  3. Introduction • GDWG Membership News • Min Min • ZhigouRong • Yuan Li • Peter Miu • Simon Elliott • R. K Giri • Masaya Takahashi • Younghooi Hwang • Sergey Uspensky • Kazuo Umezawa • Jerome Lafeuille • Masaya Takahashi has indicated that he would become the chair when he gets back to JMA. • Peter Miu has indicated that he would be the vice chair. 15th Session of Executive Panel Guangzhou, China 16 May 2014

  4. Review of activities since the last meeting • 26 action items from the last Annual Meeting. • 24 completed. • 2 overdue (some reflect change of circumstances; some will likely be reformulated at this meeting) Action Items Progress 15th Session of Executive Panel Guangzhou, China 16 May 2014

  5. Communication • GSICS User Messaging Service • 259 Subscribers. • Since last annual meeting 41 new subscribers added. Registration in GUMS is steadily increasing GDWG maintains the GUMS 15th Session of Executive Panel Guangzhou, China 16 May 2014

  6. Communication • GSICS Dev Server GDWG maintains the Dev Server

  7. Review of activities since the last EP meeting • GEO and LEO Data receiving stations have been setup. • Real time visualization and quality monitoring for NPP, METOP-A/B , MODI NOAA available • Infrastructure for cross calibration being setup. IMD GDWG progress 15th Session of Executive Panel Guangzhou, China 16 May 2014

  8. NOAA GDWG progress Review of activities since the last meeting • Support to GCC for the transition of MW FCDR product demo and then to pre-operational phase. • SRF of GEO instruments in NetCDF format [support to DCC] • Developed Product Taxonomy This is supposed to be the basis for redoing the THREDDS configuration to group GSICS correction products differently. Also the taxonomy would come handy in planning how to utilize the file naming convention and any other data management task. • Developed a data model for satellite instrument event logs and translated it into an extension to the ISO 19115-2 metadata standard's XML Schema. 15th Session of Executive Panel Guangzhou, China 16 May 2014

  9. NOAA GDWG progress Review of activities since the last meeting • Initiated process to get DOI number for all GSICS products and documents. DOI for MW already assigned by NOAA ( outside of GSICS ) GDWG to have a web meeting in June 2014 to decide on doi Vs oid. • Other GSICS products need review to progress to op. (need clarity from EP on archival & GPPA modification, Chair’s decision) • New website for GDWG ( content requested from members). • New GPPA ( proposed by Peter Miu, Manik Bali). 15th Session of Executive Panel Guangzhou, China 16 May 2014

  10. JMA GDWG progress Review of activities since the last meeting • Satellite event log • Provided the information on the JMA website in Mar/June 2013 • Change of the GEO-LEO IR RAC NetCDF contents • Met the requirements of EUMETSAT bias plotting tool in July 2013 • Provision of official SRF data to GRWG VIS/NIR sub-group • Provided data to GDWG in Apr 2013 • Collaboration with WIS • Reported feasibility to the GDWG chair in May 2013 • Calibration change alerts • Reported feasibility to gsics-dev in June 2013 15th Session of Executive Panel Guangzhou, China 16 May 2014

  11. EUMETSAT GDWG progress Review of activities since the last meeting • Proposed New GPPA. • Support to all GDWG activities. • Visualization tool. 15th Session of Executive Panel Guangzhou, China 16 May 2014

  12. EUMETSAT GDWG progress On Event Logging Data Model….. CGMS is tasking GSICS to support activities relevant to their actions (e.g. Event logging). If progress is expected in this or any GSICS action, the GSICS EP needs to urgently and decisively take a lead in managing this long standing issue EUMETSAT (Miu) 15th Session of Executive Panel Guangzhou, China 16 May 2014

  13. EUMETSAT GDWG progress EUMETSAT on Bias Monitoring tool…. • The tool is under development. 15th Session of Executive Panel Guangzhou, China 16 May 2014

  14. Review of activities since the last meeting CMA progress • IASI acquirement transferred from CLASS to EUMETSAT pushing (March, 2013) • Update GSICS data collection system for FY new sensors (FY-3C) and new reference instrument (NPP-CrIS, NPP-VIIRS) • Automatically PICS data extraction for FY-3/MERSI/VIRR • Provide the instrument telemetry/Engineering data (OBC files) from FY operational processing system • CMA GPRC new website is developing and will finish shortly 15th Session of Executive Panel Guangzhou, China 16 May 2014

  15. Review of activities since the last meeting KMA progress • Developing KMA’s GSICS website • General Introduction of COMS inter-calibration • Methods and Bias Monitoring for IR and VIS • Links to Other GSICS Website • Corrected Data filename and format 15th Session of Executive Panel Guangzhou, China 16 May 2014

  16. KMA progress KMA GSICS NMSC Homepage Data Display Four steps • 1) Data Collection • COMS, IASI, AIRS, MODIS, OMI • 2) Data Processing • Daily, Monthly Process and archiving • 3) Data Conversion • NetCDF format conversion • 4) Display • GSICS in NMSC Website • Visible and Infrared Statistics Module Image Module Data Conversion NetCDF Module Storage Storage Data Process Cloud VISCC Desert VISCD Ocean VISCO IR AIRS/IASI Internal Data Collection External Data Collection Data Collection COMS Data MODIS, OMI, AIRS, IASI KMA, Byung NASA NOAA 16 May 2014

  17. KMA progress KMA GSICS Old Filename • MT1R_20130531_0600_AIRS_060.nc • MT1R_20130531_1100_IASI_1008449763.IASI_xxx_1C_M02_20130531112 • 958Z_20130531113254Z_N_O_20130531125921Z__20130531130029.nc New Filename : GSICS standard pflag_productidentifier_oflag_originator_yyyyMMddhhmmss[_freeformat].type[.compression] • AIRS : W_KR-KMA-NMSC,SATCAL+NRTC+GEOLEOIR,COMS+MI- • Aqua+AIRS_C_RKSL_201305310600.nc • IASI : W_KR-KMA-NMSC,SATCAL+NRTC+GEOLEOIR,COMS+MI_ • MetopA+IASI_C_RKSL_201305311100.nc KMA, Byung  From April 1, 2011 to present 15th Session of Executive Panel Guangzhou, China 16 May 2014

  18. FROM RESEARCH TO PRODUCTIONBASIC PATHWAYS/TOOLS HAVE BEEN MET 15th Session of Executive Panel Guangzhou, China 16 May 2014

  19. Road Map for 2014 • Maintenance of current services in 24x7 mode • THREDDS Server • Product Catalog and Products archival. • Wiki Page/ Web Page • GDWG Support to GPPA to enable acceptance of Newer Data into GSICS fold. • Centralized Bias monitoring system. 15th Session of Executive Panel Guangzhou, China 16 May 2014

  20. Road Map for 2014 • Satellite Instrument Event Logging specification, design and implementation. • Identify a process for archiving GSICS products- need a policy document • Implementing new capabilities in the GSICS product plotting tool. • GSICS document management plan definition and associated document templates. • Doi /Oid numbers for Documents and Products. • Address resource crunch 15th Session of Executive Panel Guangzhou, China 16 May 2014

  21. Road Map for 2014 • Perform analytics when instruments are compared with stable references • Detect instrument degradation trends • Detect impacts of diurnal/seasonal instrument temp variations. Asses New tools like Big Data that can 15th Session of Executive Panel Guangzhou, China 16 May 2014

  22. Summary • NOAA GDWG has been able to ensure running of key services like Product Catalog, GSICS Dev and GUMS • File naming and DOI number for products. NOAA has the resources to assign DOI number . Outside of GSICS MW product already got the doi number. 15th Session of Executive Panel Guangzhou, China 16 May 2014

  23. Summary • Progress Reports NOAA, CMA, JMA , KMA. Excellent progress demonstrated by members towards achieving goals of GSICS. • Calibration Change Alerts ( Peter Miu, Rob Robelling) Automation, Connection , Harmonization and online depiction discussed. • Big Data (LauriRokke) Performance of algorithms, analytics • GSICS Archiving Process ( Peter Miu) Procedure for archiving, where to archive and until when. 15th Session of Executive Panel Guangzhou, China 16 May 2014

  24. Summary • Event Log ( Rob). WMO to include GPRC event logging links to the OSCAR website. • GPPA improvements ( Peter Miu, Manik Bali ) To make GPPA submission online and reduce time of review. • GEO-LEO Product promotion ( Manik Bali ). GDWG to extend file naming and meta data standards to other channels. • GDWG to support to User workshop 2014 . 15th Session of Executive Panel Guangzhou, China 16 May 2014

  25. Summary • Harmonize working hours of GDWG members. Successfully collaboration requires all GSICS GPRCs to: Nominate a representative to the GDWG. Each GDWG representative needs to commit time to work on GSICS activities (see details in the 2014 GDWG minutes). CGMS is tasking GSICS to support activities relevant to their actions (e.g. Event logging). If progress is expected in this or any GSICS action, the GSICS EP needs to urgently and decisively take a lead in managing this long standing issue.

  26. THANKYOU 15th Session of Executive Panel Guangzhou, China 16 May 2014

  27. GDWG status of members. The following GPRCs are represented in the GDWG: • EUMETSAT – 2 members • JMA – 1 member • KMA – 1 member • NOAA – 1 member Successfully collaboration requires all GSICS GPRCs to: • Nominate a representative to the GDWG. • Each GDWG representative needs to commit time to work on GSICS activities (see details in the 2014 GDWG minutes). .

  28. Introduction Advise the Executive Panel in the planning and implementation of data management activities.

  29. Introduction • Milestones since 2007 Implementation Concept 2008 • Concept of Catalogue.  • Data format.  • Event log. • Data Management Services 2010 • The essential requirements of the server as a data exchange platform were satisfied. The server was ready for operations. 2007 • Archiving requirements.  • Access methods.  • Data sets.  • Meta Data  v  Documentation  v  Security Requirements. 2011 • Bias Monitoring. • NetCDF file naming conventions 2012 • GSICS User Messaging Service • NetCDF file naming conventions • THREDDS Config. finalization 2009 • GPPA  (Product Acceptance ) • Docs on WIKI  • NetCDF file definitions 2013 • Product Catalogue. • Web-based bias monitoring tool 2014

  30. Key Discussions and Action items • Resources needed to maintain the running of servers, websites ,catalogs • AK Sharma from IMD indicated that IMD would identify a person to work on GDWG. • Accounts on Wiki would be provided to Masaya and Peter. • Masaya (VC), Peter ( Nominated himself VC). • Harmonize working hours of GDWG members. • Aleksandar’s ideas on Taxonomy for products was presented. • It was felt that currently we don’t have resources to implement it at server level. However GCC website would give an interface that would more or less follow the Taxonomy.

  31. Adoption of the proposed product taxonomy. GCC applying Taxonomy directly on the Catalog. New organizing of the products on the THREDDS catalog based on proposed taxonomy Not necessary and need resources. NetCDF file content organization of new products GDWG would look into it and asses its feasibility. Formats and procedure for storing and exchanging satellite instrument event logs. Revisit GSICS Product Acceptance. Is Big Data an answer to our questions.

  32. 5-8 March 2012 GSICS Joint Working Groups Meeting, Beijing, China

More Related