CyberInfrastructure (CI) Build

1. CyberInfrastructure (CI) Build

2. Review of the Development Process • Concept of Operations (CONOPS) • Describes how system should function at a high level • Use Cases • Flows from CONOPS, describes how users interact with system • Requirements Development • Derived from Use Cases and CONOPS; list of “Shalls” that describe the functionality of the system • Software Development • Implementation of requirements into software • Software Testing • Verification that the software performs as specified by requirements • Validation • Validation that the system performs a desired by the user, and provides valid output data products

3. CI Requirements Development • Requirements Development Path • Capability list from the uFrame architecture • Based on other uFrame instantiations (such as Tsunami, AWIPS II, etc.) • “Must have” list • List derived from use cases and CONOPS • Missing “must haves” added to CI Capability list • Vet the updated CI Capability list through the CCB to gain concurrence from all stakeholders • Follow-on Tasks • Omaha team developing functionality based on capability list • Rutgers team developing test procedures based on capability list

4. Software Development Teams • Instrument Drivers, Dataset Agent Drivers, Algorithms • Software interface to sensors, data processing • Drivers: Raytheon IDS & UW software engineers • Algorithms: OSU • uFrame (CI) Functionality • Core Infrastructure: data ingestion, storage, web services • Raytheon IIS (AWIPS Team) • User Interface Functionality • User Interface screens, user functionality • ASA Science

5. System Summary UI Drivers/Alg.

6. Prior Build Deliveries • Build 1 Components: • Drivers (OL: Dan Mergens) • 6 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • L0 & L1 Data Products for the 6 Instrument Agent Drives • L1 Data Products • Web Services: Proof of Concept • UI Capabilities (ASA: Dan Maher) • None • Install/Deployment (Rutgers: Ivan Rodero) • Website • ERDDAP Database • Build 2A Components: • Drivers (OL: Dan Mergens) • 23 Dataset Agent Drivers (each has a “telemetered” and “recovered” driver) • 3 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • L0, L1, L2 Data Products • Asset Management: • Record and manage mooring/platform data, expose UI interface • Web Services: • Test harness UI for plotting/charting, provide UI with access to data in Asset Management • UI Capabilities (ASA: Dan Maher) • Search & download science data • Graph science time-series data • Landing page (OOI Banner, navigational links) • Build 2B Components: • Drivers (OL: Dan Mergens) • 14 Dataset Agent Drivers (each has a “telemetered” and “recovered” driver) • 0 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Asset Management: • Extended support for mooring/platform data that have not yet been deployed • Web Services: • Expose UI interface for query and data retrieval • UI Capabilites (ASA: Dan Maher) • Download support data • Query telemetered data from instruments • Landing page for Pioneer Array - Science UI • Build 3 Components: • Drivers (OL: Dan Mergens) • 6 Dataset Agent Drivers • 4 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Data/Asset Management: • Framework complete, including provenance tracking calibration and QC • Asset Management complete with full lifecycle management • Web Services: • Web framework complete: query/retrieval/application services • Integrate ASA user account management services • UI Capabilities (ASA: Dan Maher) • Manage User Accounts - Annotate Data • Update Metadata - Trouble Ticket Integration • Operator Turnover - Health/Status • Build 4 Components: • Drivers (OL: Dan Mergens) • 6 Dataset Agent Drivers • 4 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Data/Asset Management: • Framework complete, including provenance tracking calibration and QC • Asset Management complete with full lifecycle management • Web Services: • Web framework complete: query/retrieval/application services • Integrate ASA user account management services • UI Capabilities (ASA: Dan Maher) • Manage User Accounts - Annotate Data • Update Metadata - Trouble Ticket Integration • Operator Turnover - Health/Status

7. Build 5 Composition • 10 Drivers(OL: Dan Mergens) • 8Dataset Agent Drivers • Includes telemetered and recovered variants • 2Instrument Agent Drivers • uFrame Capabilities(Omaha: Scott Risch) • Mission Execution • Quality Control Algorithms execution/results display • Non-Time Series Plotting • Asset Management Updates • Alarm Notification • UI Capabilities(ASA: Dan Maher) • Final User Management • CI Status Metrics

8. Build 6Composition • Drivers(OL: Dan Mergens) • 21Dataset Agent Drivers • Includes telemetered and recovered variants • 2Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • Data Annotation • Telemetry Status (feeds into Health Status) • DOI Schema support • Further Metadata Access/Retrieval • Updates/Fixes • Bug fixes will be worked by priority • UI Capabilities (ASA: Dan Maher) • Updates/Fixes

9. Driver StatusSummary • Driver Development Stages: • IDD, Code/Unit Test, Software Integration Testing • Summary of Driver Development Progress • Uncabled: 181 Dataset Agent Drivers / 3 remaining • 1 (camds_abc_dcl) in work • 2 (flord_g_dcl, zplsg_a_sio_mule) – no test data available • Cabled: 32 Instrument Agent Drivers / 1 remaining • Driver Integration Stages • uFrame OOI Integration, Verification, Validation

10. uFrame OOI • uFrame CI focused on: • Leveraging existing Raytheon uFrame SOA technology • Improved User Interface • Simplified hardware and network architecture • Collocated Data Management organization • Incremental Build Schedule • Allows for identification and resolution of integration issues • Permits feedback into later builds • Systems Engineering Management Plan (SEMP) • OOINet Build schedule is linked with new SEMP, which defines commissioning plan, specific deliverables, and is represented in schedule

11. uFrame OOI Architecture

12. uFrame OOI Architecture Notes • Plug-in Development • The uFrameteam is developing or extending plug-ins to meet some of OOI CI capabilities. • Other capabilities will be implemented via hardware or network infrastructure and via graphical user interface development by other OOI CI development teams. • Database • Postgress database implemented to hold metadata about ingested data files and to hold the information about the physical assets that collect the data. • Streaming Ingest • Data ingest performed using Cassandra engine. Cassandra provides for better performance than original (PyPies) technology, and includes support for clustering (allowing scalability), and replication.

13. Data Ingestion Processing • Profilers & Moorings • Input through Java Messages to Apache Qpid queue • Streaming Data • Instrument Agent driver translates data stream into a Qpid queue Streaming • Data Streams ingested via Sensor Integration extension • Python parsers used to parse incoming data • L0 Data stored within Cassandra • Meta data stored within Postgress

14. Algorithm Processing • Input Data streams (L0) from Mooring/ Node/ Sensors are transformed to output Product Derived (L1 & L2) via algorithms • Algorithms require that calibration information be stored and managed (as calibration data may change over time Data arriving via streams is converted to usable data products through application of algorithms which apply calibration constants, perform quality control functions, and merge data from co-located sensors to generate data products

15. Data Management/Processing • Asset data is retrieved from OOINetuFrame via REST services hosted by the Web Services plug-in to uFrame. • The REST request from the Asset Management GUI is handled by the Asset Web Service which translates the REST request to a database query. Asset Data can be ingested by bulk upload (for operators) and via Web based GUI GUI will provide for entering individual data records, including editing and viewing existing records

16. User Interface Functionality • UI Capabilities will be available with Build 5 Delivery • Some UI Capabilities dependent upon uFrame capabilities • Integration testing to continue with Build 6 • Expect some changes from user feedback load testing

17. User Interface Functionality • Screens can be coded and unit tested, but not fully tested until full System is functional

18. User Interface Examples: Landing Page

19. User Interface Examples: Infrastructure Info

20. User Interface Examples: Science Data

21. OOI-CI Testing • Driver Development Unit / Integration Test • Verify that drivers function per IDD • Performed by Driver Development Teams (Raytheon, OSU, UW) • Framework Development Unit / Integration Test • Verify that software functions as expected in the development environment • Performed by Raytheon Omaha team • Verification / Acceptance Test • Verify that software supports defined use cases and meets documented requirements • This is done by running through test cases which are created from use cases • Perform by Rutgers Software Test team • Data Validation Test • Verify that data transformation algorithm is correct via output data validation • Perform by Rutgers Data Management team • Science Test • Verify that software meets science needs • Perform by wider science community

22. Test Artifacts Test Procedures Test Reports Test Planning Test Scheduling

23. User Interface Demonstration Video of walkthrough of UI screens

24. Back-up Slides Detail slides provided as back-up data follow this slide

25. Instrument Driver Overall Status Software Development Effort Systems Integration and Test Effort

26. Dataset Parser Plug-in Progress Software Development Effort Source: Unified Status Tracking - DAD Status

27. uFrame Build Capabilities

28. uFrame Build Capabilities

29. uFrame Build Capabilities

30. uFrame Build Capabilities

31. uFrame Build Capabilities

32. Build 1B Composition • Components: • Drivers (OL: Dan Mergens) • 6 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • L0 & L1 Data Products for the 6 Instrument Agent Drives • L1 Data Products • Web Services: Proof of Concept • UI Capabilities (ASA: Dan Maher) • None • Install/Deployment (Rutgers: Ivan Rodero) • Website • ERDDAP Database

33. Build 2A Composition • Components: • Drivers (OL: Dan Mergens) • 23 Dataset Agent Drivers (each has a “telemetered” and “recovered” driver) • 3 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • L0, L1, L2 Data Products • Asset Management: • Record and manage mooring/platform data, expose UI interface • Web Services: • Test harness UI for plotting/charting, provide UI with access to data in Asset Management • UI Capabilities (ASA: Dan Maher) • Search & download science data • Graph science time-series data • Landing page (OOI Banner, navigational links)

34. Build 2B Composition • Components: • Drivers (OL: Dan Mergens) • 14 Dataset Agent Drivers (each has a “telemetered” and “recovered” driver) • 0 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Asset Management: • Extended support for mooring/platform data that have not yet been deployed • Web Services: • Expose UI interface for query and data retrieval • UI Capabilites (ASA: Dan Maher) • Download support data • Query telemetered data from instruments • Landing page for Pioneer Array - Science UI

35. Build 3 Composition • Components: • Drivers (OL: Dan Mergens) • 6 Dataset Agent Drivers • 4 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Data/Asset Management: • Framework complete, including provenance tracking calibration and QC • Asset Management complete with full lifecycle management • Web Services: • Web framework complete: query/retrieval/application services • Integrate ASA user account management services • UI Capabilities (ASA: Dan Maher) • Manage User Accounts - Annotate Data • Update Metadata - Trouble Ticket Integration • Operator Turnover - Health/Status

36. Build 4 Composition • Components: • Drivers (OL: Dan Mergens) • 14 Dataset Agent Drivers • 15 Instrument Agent Drivers • 6 Platform Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Alerts/Alarms/Notifications • User specified Annotations • UI Capabilities (ASA: Dan Maher) • Initial Command/Control functionality • QA/QC functionality; Alerts & Alarms • Help/Glossary/How-to • Initial Asset Management

37. Data Management Science/Engineering Data Data Management Cabled/Platforms Instruments L0 Data Uframe Data Ingestion Parser to data stream association Drivers/Parsers Asset Management: eg. Location, Depth, Calibration Coefficients, startup config, telemetry Creation of L1, L2 Data Products Command and Control QC look up tables Algorithms Mission Execution QC on L1, L2 Data Products Alarm, Alert, Health/Status flags and thresholds Glider Schedules Alerts/Alarms, Health Status monitoring Profiler Mission Plans

38. Asset Management Database Schema for storage of all Metadata related to assets and their associated events

39. Earned Value Definitions • CPI = Cost Performance Index • Work Performed/Cost of work Performed, 1.0 = Spending as planned • SPI = Schedule Performance Index • Work Performed/Work Scheduled, 1.0 = Completing work as planned • Cur. = Current Period Performance • Status of most current 2 weeks of work • Cum. = Cumulative Performance • Status of work since beginning of task • Plot = Historical data • Plot is bi-weekly graph of cumulative Plan Costs/Actual Costs/Performance Costs

40. uFrame CI Plan vs. Actuals

41. UI Plan vs. Actuals Showing data through end of Apr. Plot is updated at the end of the month, per ASA financial report –