1 / 24

Status of Meta-Files, ADMT 13 Hyderabad, India

Status of Meta-Files, ADMT 13 Hyderabad, India. Esmee van Wijk , Nov 2012 CSIRO Marine and Atmospheric Research, Australia. Photo from Katy Hill. Metadata format version 2.4. Number of defined missions, not number of profiles.

dwight
Download Presentation

Status of Meta-Files, ADMT 13 Hyderabad, India

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Status of Meta-Files, ADMT 13 Hyderabad, India Esmee van Wijk, Nov 2012 CSIRO Marine and Atmospheric Research, Australia Photo from Katy Hill

  2. Metadata format version 2.4 Number of defined missions, not number of profiles

  3. Metadata variables – 2.4.3 Float characteristics • IMEI numbers • Security risk • Remove from meta file? • Or restrict numbers even further? • TC: YES to remove • (not needed in metafiles). • But AIC needs it for tracking • (authorization, private use) • New variables • PLATFORM_FAMILY (SRT) • PLATFORM_TYPE (SRT) • Standard reference tables developed by TC (SRT)

  4. Metadata variables – 2.4.3 Float characteristics STR PLATFORM_FAMILY PLATFORM_TYPE PLATFORM_MAKERPLATFORM_TYPE_KEY STANDARD_FORMAT_ID and also applied to GTS tables (TESAC/BUFR). 1) STR Needs agreement 2) STANDARD_FORMAT_ID detailed as far as possible (on progress on-line) 3) link to each WMO: Suggestion to fill up a new table for global update for 9000 floats (AIC, DACs, GDACs) including key new metadata (new tab, blank on-line) http://tinyurl.com/cweskt6

  5. Metadata variables – 2.4.3. Float characteristics • New variables • FLOAT_SERIAL_NO replaces INST_REFERENCE variable • should only be filled with the serial number of the float i.e. 1679 NOT sbe1679 or APEX1679 etc. • Attention PROVOR serial is “complex” (typos) • FIRMWARE_VERSION • MANUAL_VERSION • STANDARD_FORMAT_ID • DAC_FORMAT_ID • important for identifying float format and enabling cross-reference to online manuals • important information required for trajectory analyses

  6. Metadata variables – 2.4.3 Float characteristics • New variables • BATTERY_TYPE (SRT) • BATTERY_PACKS (SRT) • CONTROLLER_BOARD_TYPE_PRIMARY (SRT) • CONTROLLER_BOARD_TYPE_SECONDARY (SRT) • CONTROLLER_BOARD_SERIAL_NO • _PRIMARY • CONTROLLER_BOARD_SERIAL_NO • _SECONDARY • Standard reference tables developed by TC where possible

  7. Metadata variables – 2.4.3 Float characteristics • New variables • Standard reference tables developed by TC where possible (SRT) • SPECIAL_FEATURES • SAMPLING_MODE (SRT) • REDEPLOYED • FLOAT_OWNER • OPERATING_INSTITUTION • CUSTOMISATION • ARGO_GROUP (SRT)

  8. Redefine descriptions – i.e. core Argo currently too specific, e.g. 2000 profile, 1500 park? • - equivalent Argo? – Argo floats that are not considered part of the core Argo mission, e.g. navy, special funding, no delayed mode activity, accelerated profiling etc. • Do we want additional categories..? Currently, we have Argo equivalent, Bio Argo, Core Argo categories that are well known within the Argo community.. • Do we want others..? • coastal.. (issues - floats may be deployed in a coastal region but drift out..) • marginal seas.. • boundary current.. • under ice.. (only floats with under ice algorithm or hardware..? Or those floats deployed above 60˚N and below 60˚S ?

  9. Mandatory Meta-data Parameters Have we missed anything?

  10. Mandatory Meta-data Parameters con.

  11. 2.4.5. Configuration Parameters • For each configuration parameter the name and value of the parameter are recorded • Users can identify which mission belongs to which cycle by looking at the: CONFIGURATION_MISSION_NUMBER (N_CYCLE) variable in the trajectory file • Do we also want CONFIGURATION_MISSION_NUMBER (N_PROF) in the profile file?

  12. 2.4.5. Configuration Parameters • Historically, input was not standardised, not all information required was reported and fields were not populated correctly. • Clearly there is a need to standardise configuration parameters. • Scheme adopted is similar to the technical files. • All configuration parameters are identified by a CONFIG_* prefix. • Configuration parameters are float settings, not measurements reported by the float. • DACS were asked to check their float types and provide feedback before this meeting.

  13. Configuration Parameter Names • Reference table 18 in Argo User’s Manual (version 2.4 March 30th 2012) • Available from http://www.argodatamgt.org/Documentation

  14. Configuration Parameters • New configuration variables will need to be approved and added to the table before they are accepted, to ensure names remain standard – similar to the technical file approach. • Requests for new names should be sent to: argo-dm-chairman@jcommops.org • Configuration parameter values are stored as numerals • Any parameters with logical or string input will require an equivalent numeric code to be added to the “Explanation” section of the parameter names table. • Currently, only 3 parameters are affected: CONFIG_Direction_LOGICAL (Ascending = 1, Descending = 2) CONFIG_MeasureBattery_LOGICAL (Yes = 1, No = 0) CONFIG_TriProfileOption_LOGICAL (Yes = 1, No = 0) • Units are irrelevant i.e. CONFIG_MissionPreludeTime can have units of _HOURS or _MINUTES or _HHMMSS, do not need a new parameter in each case

  15. Mission Parameters • Typically, an Argo float configuration is valid for the life of the float (one configuration) CONFIG_MISSION_NUMBER = 0 (launch or pre-deployment info) and 1 (basic mission) • For floats with multiple configurations, the configuration from the first cycle is set to 1. Each subsequent configuration change has a new mission number, i.e. from 1 to N. • Argo best practice is a minimum of configuration missions. • i.e. if there is a change to configuration parameters that does not mirror a previous change then use a new mission number. • If the configuration parameters change but mirror a previous mission, then that mission number should be re-used (in complex cases where mission changes are unclear – then a new mission number can be used. CONFIG_MISSION_NUMBER • Each time a new mission number is added, the metafile will need to be rewritten.

  16. Floats with Multiple Missions * N_CONF_PARAM = int value * N_MISSIONS = <unlimited> *1=ascending, 2=descending, ^0=no, 1=yes • All variables from Mission 1 must be repeated in subsequent missions even if they do not change. • If new variables are added in later missions, previous missions will need to be rewritten.

  17. Data Formats • We need a way to describe the float data format.. • A way to link the data manual and the ANDRO decoding information to a standard format ID for each float. • This is important information for the trajectory processing and for decoding the raw data (delays!) • Input needs standardisation. • For some of the APEX APF9 floats there is no data format number on the manual – CSIRO has made up their own numbers, we need to standardise these. • PROVOR float manuals also have no numbers and have two different kinds of format, i.e. layer or chords. • need a copy of each version of the data manual for all float types, scanned and made available on the AIC or/and ADMT websites.

  18. Float Data Formats CSIRO: http://www.cmar.csiro.au/argo/dmqc/html/Australian_float_manuals.html MANUAL_VERSION DAC_FORMAT_ID FIRMWARE_VERSION

  19. Float Data Formats JMA: http://argo.kishou.go.jp/document/float_type.html SIO: http://sio-argo.ucsd.edu/manuals.html DAC_FORMAT_ID ? • Some floats are returning version numbers e.g. SOLOII i.e. FIRMWARE_ • VERSION MANUAL_VERSION ? DAC_ FORMAT_ID ?

  20. Float Data Formats • Progress with Online Formats… • INCOIS: Done – publically available? • Other DACS…??? estimated finish date, impediments? • Require feedback from DACS as to what they are populating in their DAC_FORMAT_ID, FIRMWARE_VERSION and MANUAL_VERSION fields ? • Working with the ANDRO team and Argo TC on a table linking the online float documentation, e.g. the DAC_FORMAT_ID field to the ANDRO decoder format ID and a proposed new STANDARD_FORMAT_ID field. • Issues: ANDRO float decoders only exist for Argos floats where: • There is a description of the format (i.e. in a user’s manual) • There is original Argos data • ANDRO has implemented a decoder • Float data was transmitted before Jan 1st 2010 • The float made a contribution to ANDRO (POPS are excluded) • Iridium Apex floats have not been decoded, only Iridium Provors

  21. Float Data Formats • Proposed table linking DAC_FORMAT _ID to a STANDARD_FORMAT_ID • Table hosted at AIC by Argo TC • STANDARD_FORMAT_ID – should we use ANDRO decoder ID or do we need a new universal number ? • i.e. "PPPNNN“, where PPP = a unique platform id for APEX, PROVOR etc and NNN = a made-up standard data id # • manufacturers/operators to declare each new format at AIC and circulate to DACs (FAR BEFORE DEPLOYMENT) • Operators could then register this information at float deployment notification step (via AIC web form), andinclude it in metafiles • ANDRO Decoder’s do not exist for float data post Jan 1st 2010 – how do we fill this gap?

  22. Float Data FormatsSOLO Example Work to be refined gradually ... ALL DACS, PIS, manufactuers to participate !!!!

  23. Standard Reference Tables We need to agree on a set of new metadata reference tables.. • PLATFORM_FAMILY (in manual) • PLATFORM_TYPE (in manual) • PLATFORM_MAKER (in manual) • SENSOR_MAKER (in manual) • SENSOR_MODEL (in manual) • SENSOR_UNITS (in manual) • SAMPLING_MODE (in manual) • CONTROLLER_BOARD_TYPE (in manual) • BATTERY_PACK (in manual) • BATTERY_PACKS (define approved abbreviations for battery types, in manual) • STANDARD_FORMAT_ID/DAC_FORMAT_ID/ANDRO_DECODER_ID – table linking all 3 will be hosted at AIC. • ARGO_GROUP (already exists, needs refinement) Actions: On-line tables for drafting (see e.g. http://tinyurl.com/cweskt6 ) TC will call community for contributions

  24. Action Items Action Item 36 – Configuration Parameter Names Action Item 37 – Manual Updates completed (few small recent additional changes) Action Item 38 – new format scheme devised after ADMT12 in Korea, DACS should now be able to generate metafiles correctly Action Item 39 – decoder format to online documentation Action Item 44 – Resubmit oxygen data in agreed format

More Related