1 / 63

CyberInfrastructure Status

This status update provides an overview of the current progress and plans for the CyberInfrastructure project. It includes details on the completion of Build 3, integration efforts, UI evaluation plan, and metadata model updates.

roxiej
Download Presentation

CyberInfrastructure Status

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CyberInfrastructure Status Presented March 2, 2015 Status as of February 27, 2015

  2. Agenda • Top level Schedule/plan • Status with a line for each major task. • Status of Individual Builds • Build Composition (describes goal of each build) • Build Detail Status (describes tasking of each build) • Highlights of Weekly Status Charts • Financial / Performance Charts • Back-up • Weekly status charts from each organization: • What is planned overall and when will it be done (actual and plan) • What was planned for this week and what was done • Finances (actual vs. planned spending) • What is planned for next week

  3. Delivery Build 3 Status Overview • Build 3 completed on 2/26 • 12 additional sensors integrated • With associated L0, L1, and L2 data products (complete) • Data/Asset Management Framework • Functional capabilities delivered (complete) • Data schema changes now in review pending implementation in later deliveries • Web Framework Services • Query/Retrieval/Application services (complete) • Sends commands to cabled assets (complete) • Status Reports • Support for status, reporting, planning (in analysis) • UI Capabilities • Manage User Accounts - Annotate Data • Update Metadata -Blocked - Trouble Ticket Integration • Operator Turnover - Health/Status - Blocked

  4. Delivery Build 3 Integration • Build 3 Pre-integration (ASA/Rutgers/Omaha) • Build 3 release delivered 2/20 • Pre-integration evaluation and bug fixes 2/23 through 2/27 • Cassandra enhancement completed 2/26 • uFrame & GUI integrated on Rutgers servers • Build 3 Integration (Rutgers) • Baseline delivered to Integration Team 2/27 • Install on Rutgers server 3/2 • Ingest data sets 3/2 • Exercise User Interface with uFrame CI Framework • Verify functionality in integrated build 3/2 through 3/6 • Begin verification 3/9

  5. UI Evaluation Plan • Build 3 UI screens to be run from Rutgers • Limited distribution to representatives of: • Users: • Operators: • Scientists: • Redmine used to collect user feedback

  6. Metadata Model Path Forward • Assess uFrame Metadata Model for completeness • Metadata Review with MIOs scheduled for 3/2 • Complete Data Schema (3/6) • OL/NSF Review/Approval (~ 3/11) • uFrame Metadata Model Updates • Implementation meeting held on 3/2 • Omaha to incorporate updates into Asset Management • Updated Asset Management DB available in Delivery 4 • UI Screens updated to support Asset Management access • UI screens to support Asset Management available in Delivery 4 • Validate Asset Management Data • MIOs use UI screens to validate Asset Management

  7. CyberInfrastructure Deliveries ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ *All dates are plan dates

  8. Glider Status Details

  9. Build 1B Composition • Components: • Drivers (OL: Dan Mergens) • 6 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • L0 & L1 Data Products for the 6 Instrument Agent Drives • L1 Data Products • Web Services: Proof of Concept • UI Capabilities (ASA: Dan Maher) • None • Install/Deployment (Rutgers: Ivan Rodero) • Website • ERDDAP Database

  10. Build 1B Status Details

  11. Build 2A Composition • Components: • Drivers (OL: Dan Mergens) • 23 Dataset Agent Drivers (each has a “telemetered” and “recovered” driver) • 3 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • L0, L1, L2 Data Products • Asset Management: • Record and manage mooring/platform data, expose UI interface • Web Services: • Test harness UI for plotting/charting, provide UI with access to data in Asset Management • UI Capabilities (ASA: Dan Maher) • Search & download science data • Graph science time-series data • Landing page (OOI Banner, navigational links)

  12. Build 2A Status Details

  13. Build 2B Composition • Components: • Drivers (OL: Dan Mergens) • 14 Dataset Agent Drivers (each has a “telemetered” and “recovered” driver) • 0 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Asset Management: • Extended support for mooring/platform data that have not yet been deployed • Web Services: • Expose UI interface for query and data retrieval • UI Capabilites (ASA: Dan Maher) • Download support data • Query telemetered data from instruments • Landing page for Pioneer Array - Science UI

  14. Build 2B Status Details

  15. Build 3 Composition • Components: • Drivers (OL: Dan Mergens) • 6 Dataset Agent Drivers • 4 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Data/Asset Management: • Framework complete, including provenance tracking calibration and QC • Asset Management complete with full lifecycle management • Web Services: • Web framework complete: query/retrieval/application services • Integrate ASA user account management services • UI Capabilities (ASA: Dan Maher) • Manage User Accounts - Annotate Data • Update Metadata - Trouble Ticket Integration • Operator Turnover - Health/Status

  16. Build 3 Status Details

  17. Build 4 Composition • Components: • Drivers (OL: Dan Mergens) • 14 Dataset Agent Drivers • 15 Instrument Agent Drivers • 6 Platform Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Updates/Fixes • All functionality delivered, bug fixes will be worked by priority • UI Capabilities (ASA: Dan Maher) • Initial Command/Control functionality • QA/QC functionality; Alerts & Alarms • Help/Glossary/How-to • Initial Asset Management

  18. Build 4 Status Details

  19. Build 5 Composition • Components: • Drivers (OL: Dan Mergens) • 29 Dataset Agent Drivers • Includes telemetered and recovered variants • 15 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • Updates/Fixes • All functionality delivered, bug fixes will be worked by priority • UI Capabilities (ASA: Dan Maher) • Final User Management • CI Status Metrics

  20. Build 5 Status Details

  21. Build 5 Status Details (cont.)

  22. uFrame Performance Status • Ingestion Robustness/Efficiency • Problem: Source data files must be carefully fed to uFrame to avoid overload of queues • Status: Correction is prototyped, planned for 2/26 build update • Redmine ticket opened to perform analysis after update • Impacts: • Only an issue with large incremental data insertion (recovered and backlog data); • Will be resolved inDelivery 3+ • Notes: Preliminary tests show that new ingestion engine (Cassandra) can easily handle higher bandwidth instruments: • Successfully ran a 48 hour ingestion test for PARAD at 1 hz sample rate, running uFrame in a Virtual Machine on a laptop • Successfully ingested FLORD instrument data at 10 hz sample rate, running uFrame in a Virtual Machine on a laptop –

  23. Combined CyberInfrastructure • Status from last week • Plan for incorporating Health & Status, Alarms/Alerts • Held mtg w/ Raytheon Omaha and ASA 3/2 (complete) Plan for supporting metrics data Assess Performance/Evaluation metrics (complete) Update Asset Management schema to support metrics (Outlook 2/9) Review w/ OL (Outlook 2/10) • Present path forward for High Definition Camera and Video (in work) • Refine schedule for supplying EPE interface (in work) • Update requirements to support EPE (Outlook 2/10) • Plan for OMC Data Management • Define requirements (complete) • Document path forward (in work) • CCB Data Management plan (tbd) • Plans for this week • Continue “in work” items • Complete Delivery 3 baseline to Rutgers/Integration team • Begin Work on Delivery 4 baseline • Actions • N/A

  24. User Interface Details (Build 3)

  25. User Interface Details (Build 3 - cont)

  26. User Interface Details (Build 4)

  27. User Interface Details (Build 4 - cont)

  28. uFrame CI Plan vs. Actuals • Updated plot now includes “prototype” • Actuals have been correlated to 2/13 financial report

  29. Cabled Plug-ins Plan vs. Actuals –

  30. Uncabled Plug-ins Plan vs. Actuals –

  31. UI Plan vs. Actuals Will be updated at month-end when ASA actuals are available –

  32. Back-up –

  33. OL CI Tasking • Status from last week • CCB remaining Use Cases (Set 2, 3, 4) (outlook 3/2) • Brought to CCB, due to NSF concerns will reboard next week • Resolve issues with Use Case 3 - Asset Management • Draft plan for OMC Data Management (in work) • Document low-level capabilities for releases (in work) • Plans for this week • Metadata schema comments incorporation • Metadata review (2/23) • Follow-up tentatively scheduled for 2/26 or 2/27 • Actions • N/A

  34. Cabled Driver Plug-in Status • Status from last week • Working CAMDS Code Review changes • Working ZPLSC Group 5 dataset parser development • Plans for this week • Continue working Omaha uFrame integration tasking • Support development and execution of test plan for live instruments • Complete CAMDS integration • CAMDS Plugin for image archive still to be done • CAMHD path forward presentation (John P. to schedule)

  35. RSN Commissioning Plan

  36. RSN Commissioning Plan (cont.)

  37. Instrument Driver Overall Status

  38. Instrument Driver Remaining Work

  39. Uncabled Driver Plug-in Status • Status from last week • Resolve instrument configuration details with Hydroid Inc. -Complete • Hydoroid provide general configuration, answered existing detailed questions • Complete review of fdchp_a (recovered) IDD -Complete • waiting for MIO to complete review • Begin coding fdchp_a (recovered) parser - -Complete • coding and testing complete, driver delivered • Deliver code auv common parser and adcpa_n_auv parser/driver -Complete • Code and test remaining auv science data parsers -In Progres • ctdav_n_auv, dosta_ln_auv, parad_n_auv completed and delivered • flort_kn_auv, nutnr_n_auv require updates due to additional information from Hydroid • Submit auv_eng_dcl IDD for external review -Complete • Plans • Complete coding/testing, deliver flort_kn_auv, nutnr_n_auv • Complete review of auv_eng_dcl IDD • Submit adcpa_n (recovered version of adcpa_n_auv) IDD for review • Issues/Risks • None

  40. Dataset Parser Plug-in Progress Source: Unified Status Tracking - DAD Status

  41. Parser Plug-in Development Plan • Remaining development (to Omaha) expected to complete late April Source: Unified Status Tracking - DAD Summary

  42. Dataset Parser Plug-in Issues

  43. Unaddressed Dataset Parsers

  44. PAD Development Status • Status from last week •Testing pulling OMS data from live system in Shore Station • Power Feed • PN LVPS • Updated Error Handling • Plans for this week • Ready to support higher level of integration (group 4). • Continue testing against live system. –

  45. CI Framework Status • Status through 2/20 (Delivery 1-3) – Sensor Integration Group 1-3: 141 tickets total, 117 resolved – 2 resolved last week – Data Management framework: 177 tickets total, 98 resolved – 2 resolved last week – Asset Management: 66 tickets total, 60 resolved – 2 resolved last week – Web Services: 18 tickets total, 14 resolved – 0 resolved last week • Plans for this week – Short Term Plan for RSN Data Transfer to Rutgers – Revise Data Storage Requirements (based on Cassandra ingest) – Plan for ingesting RSN archived data – Continue Delivery 4 task execution • Actions – Need mooring and calibration information (see Blocker tasks)

  46. CI Integration Blockers

  47. Data Product Algorithm Status • Status from last week/ongoing • Support algorithm or test dataset blockers as requested or discovered (triage): • Discovered missing algorithms for VEL3D series A,L (RSN and Papa). Under investigation – anticipate completion by . Redmine 2424 filed. • Supporting work on other Redmine tickets. • Time-stamp issue in coastal glider RDI – analysis underway. Redmine 2427 filed Helped resolve Redmine 2378, 2396 – VEL3D-K – now for Delivery 3 • Investigating Redmine 2386 – Fault during call to TEOS-10 gsw C-libraries during uFrame ingestion. Sent back to uFrame for details. • Awaiting meeting to discuss path forward for Redmine 1404: (METBK - L1/L2 data processing (still) results in products with a different time-domain and a different number of products than the input L0/L1s)

  48. User Interface Status • Status as of 2/27 • Testing and Bug Fixing • Travelled to Omaha • Integrated with most recent uFrame • Integrated onto Rutgers’ hardware • This Week • Asset Management • Create APIs to populate Asset Management screens • Create dynamic filtering capabilities • Additional Array Landing Pages • Backend work on Command and Control • Risks • Asset Management Schema

  49. User Interface Schedule

  50. Rutgers Status • Status from last week • Scott and Oscar visited COL regarding mgmt plans • Several job interviews for Data Analysts. We believe we have an assistant data manager and 2 data evaluators identified. • Review of ASA GUIs, and follow on discussions of EPE web services needs. • Creating single document linking reference designators all the way through uFrame directories (Kerfoot) • Observation and Sampling Approach doc almost complete • Plans for this week • Data reviews of all 2B data • Continued job interviews • Metadata document assistance (Vardaro lead) • Assistant Data Manager begins job (Friday?)

More Related