Aquarius Instrument Operations

1. Aquarius Instrument Operations Aquarius/SAC-D Mission Operations Review February 8 – 10, 2010

2. Agenda Peer Review Results Key Requirements Processes/Operations Overview Aquarius Command and Control System ArchitectureTesting StatusProcedures/Status/PlansRisks Risk Mitigation Documentation/Status/Plans for completion

3. Aquarius Instrument Operations Peer Review results

4. Recommended Actions • Clearly define the final set of instrument operations procedures • Identify procedures to be written • Identify what procedures must be verified on the Observatory before flight • The project has been addressing these since the Peer Review. • Concern remains that, due to project demands, personnel may not be available to rehearse critical procedures

5. Aquarius Instrument Operations Key requirements

6. Key Requirements • Mechanics of the Ground System Design • Use of the Oceans Biology System • ACCS relies on its facilities, networks, security, and software support • Use of CONAE for Mission Operations • The ACCS Scheduler Client and Telemetry Viewer are CONAE deliverables • The AQ Test-bed also interfaces with SInter, the CONAE commanding tool. • ACCS processes and procedures must align with CONAE’s • Instrument Operations - Maximizing Quality and Quantity of Science Data Collection • Science data loss budget of 8% • Science data accuracy to 0.2 PSU • Perform Aquarius command planning within flight rule constraints • Monitor Aquarius science and telemetry to determine state of health • Perform Aquarius anomaly handling

7. Key Requirements Flow-down Summary • As captured in the documents: • Aquarius Ground System Level-3 Requirements (AQ-326-0152) • L4 AQ Command and Control System & Test Requirements (AQ-426-0197) Instrument Operations Responsibilities DO include: Aquarius command planning Aquarius offline state-of-health analysis Aquarius anomaly handling Instrument Operations Responsibilities do NOT include: Real-time commanding Real-time telemetry monitoring Aquarius safing SAC-D anomaly handling All real-time tasks and all SAC-D tasks are the responsibility of the SAC-D Flight Operations Team

8. Aquarius Operations Processes

9. Aquarius Instrument Operations Processes • The Aquarius Command and Control System (ACCS) supports: • Aquarius command planning • Standard: daily science data downloads; monthly Cold Sky Calibrations • Special: science requests such as Look Up Table changes • Emergency: anomaly response • Aquarius offline state-of-health analysis • Receipt of Aquarius/SAC-D science and housekeeping data from the ADPS. • Receipt of Aquarius/SAC-D pass prediction files from the ADPS. • Processing of Aquarius/SAC-D housekeeping and Aquarius science data for the purpose of monitoring instrument trends and state-of-health. • Publishing of telemetry trends and alarms to Web-based tools to enable off-site health checks. • Aquarius anomaly handling • Aquarius instrument anomaly tracking and resolution.

10. Command Planning Overview • Two standard command planning activities: • Perform science data downloads at least once every 12 hours • Plan 2 weeks of downloads every week • CONAE has the flexibility to send a subset of these every few days • CONAE performs C&DH management • Perform a Cold Sky Calibration • No Aquarius commands; purely a request for the S/P state of “Cold Sky Calibration” • Science may request other command planning activities such as: • Thermal set-point change • Adjust the radiometer receive window

11. Command Planning Steps • Daily: Check for science requests • Include any science requests in the upcoming command plan • Create a stand-alone command plan if urgent • Weekly: Generate the command plan • Two weeks of downloads • Command plan upload to the S/P is base-lined as being two weeks in the future • Command plan execution by the S/P is base-lined as being 2 weeks + 3 days in the future • Any science requests • Enter command plan into the Scheduler Client • A command plan is an “Action Request” to the Scheduler Client. • An “Action Request” is composed of “Basic Action Components”, “Massive Upload Components” and “Extended Condition Components”. • Example Basic Action Component: start downlink/stop downlink command pair • Example Massive Upload: Aquarius Flight Software Patch • Example Extended Condition Component: X-band transmitter on/off command pair • Scheduler Client is a CONAE deliverable tool • Interfaces with the CONAE Spacecraft Operations Planning tool (SOP) • CONAE SOP provides progress reports on Action Request planning via the Scheduler Client: • Action Request Received by the SOP • Action Request Accepted by the SOP • ACCS verifies reports, then verifies execution via telemetry/downlink file reception

12. Standard Command Planning Process at a Glance Aquarius Commands Planned at the ACCS Aquarius Action Requests Submitted to the SOP ACCS Receives Action Report: Action Request Received ACCS Receives Action Report: Action Request Accepted

13. Command Planning - Emergency • The emergency command planning process is defined as follows: •  Utilize emergency channels to notify the CONAE MOC of an Aquarius anomaly: • Phone • Email • And also enter a ticket into the: • Aquarius Reporting and Tracking Tool • Generate the (contingency) Aquarius command plan as per standard procedure, but with accelerated implementation: • CONAE will schedule NEN support • CONAE will process received Aquarius command plans as quickly as possible

14. Daily Monitoring & Trending Process • Alarm and Error Notification Tool: • Verify expected download files have been received • Download file times match commanded download times • File sizes match expected file sizes • No Cyclic Redundancy Check errors have occurred • Note: Data integrity (ICDS CRC and contiguous blocks) checked automatically • Verify No Alarms present • If Alarms present, perform Anomaly Resolution process • Web-based graphs • Examine science and telemetry trends on various scales: • Daily • Monthly, Seasonal • Yearly, Mission • Telemetry Viewer • Verify Alarms in agreement with Analysis Tool • Verify instrument configuration is as expected • Review data for software messages

15. Anomaly Response The SAC-D and Aquarius teams generated a common anomaly response flow. Aquarius Operations has designed an anomaly process aligned with that flow.

16. AS Anomaly Process

17. Aquarius Specific Anomaly Resolution Process I. Discovery a. CONAE FOT or AQ OPS detects telemetry Out of Limits Team has procedures and authority to SAFE Aquarius b. AQ OPS detects abnormal science or telemetry trend c. Science detects abnormal science performance II. Notification a. If detected by CONAE or Science, party notifies AQ OPS via email/phone b. AQ OPS notifies AQ GS Manager, CONAE Manager, AQ PI by submitting a TRAC anomaly ticket III. Gather Investigation Team a. AQ OPS asks AQ GS Manager for additional support b. AQ OPS updates TRAC with criticality, preliminary investigation report; updates TRAC status to “team formation”. (Publishes TRAC ticket and report to AQ/Science/CONAE management) c. AQ GS Manager asks the AQ PI to authorize funds for JPL support d. AQ PI asks JPL science liaison to recruit resolution team e. JPL science liaison recruits team, notifies AQ PI, AQ GS Mgr, AQ OPS of team members f. AQ OPS updates TRAC with team member list; updates TRAC status to “in work”. (Publishes TRAC ticket and report to entire team).

18. Aquarius Specific Anomaly Resolution Process IV. Develop Solution a. Technical team uses TRAC to confer, exchange data, develop solution and verification criteria, exercise with Test-bed. b. When ready, technical team buy-in confirmed by setting TRAC status to “solution proposed”. V. Obtain Approvals a. Upon team acceptance, technical team leader updates TRAC status to “solution accepted, technical”. b. CONAE exercises procedure against Observatory EM; forwards accept/reject to CONAE Mgmt c. If “reject”, return to step V. d. Management team (AQ GS/AQ PI/CONAE) reviews solution; upon acceptance, AQ PI updates TRAC status to “solution accepted, management”. VI. Execute and Verify a. AQ OPS plans commands according to procedure b. CONAE FOT executes c. Full team verifies per step V criteria. VII. Report AQ OPS updates TRAC with Final Report and Resolution status: completed/reissued. (If reissued, go to step V). Aquarius operations documents and procedures updated accordingly.

19. Aquarius Command and Control System Architecture

20. Aquarius Ground System The Aquarius Ground System is contained within the Oceans Biology Data System • The Aquarius Ground System is made up of 2 segments: • The Aquarius Data Processing System (ADPS) • The Aquarius Command and Control System (ACCS) • The Aquarius Data Processing System receives Aquarius science data, SAC-D auxiliary data, and other science ancillary data to generate Aquarius science products. • The Aquarius Command and Control System receives Aquarius science data, SAC-D and Aquarius House Keeping Telemetry, and pass prediction files in order to monitor Aquarius health and status, plan commands, and respond to anomalies.

21. Oceans Biology Data System Support to the Aquarius Command and Control System • Functions/services provided: • Facilities • Infrastructure • Staff • IT network and security • Inherent redundancy (except CPT; hot backup) • Use of automation • Configuration Management • Software maintenance • Data storage • Use of Oceans Biology Data System interface with CONAE

22. ACCS Architecture Aquarius Command and Control tools and interfaces

23. Plans and Status of testing

24. ACCS and Aquarius Instrument Operations Test List • ACCS Ground System Testing • Software Tools • Component • Internal Interfaces • Functional • External Interfaces • Command and Telemetry Testing • Command Database • Telemetry Database • Cross-Tool Verification • Mission Scenarios

25. Testing Status • COMPLETED TESTS • Software Tools • Component – baseline completed January 11, 2010 • Internal Interfaces – baseline completed January 14, 2010 • Functional – baseline completed January 14, 2010 • Command and Telemetry Testing • Command Database – completed June 15, 2009 • Telemetry Database – completed in instrument I&T at JPL • UPCOMING TESTS • Cross-Tool Verification – test design meeting scheduled February 15, 2010 • External Interfaces – will participate in CONAE led tests • Mission Scenarios – under development by the SAC-D and Aquarius Mission System Engineers.

26. ACCS Components Tested Component Testing included the: • Analysis Tool • Alarm and Error Notification Tool • Command Planning Tool • Web-based graphs • Reporting and Tracking Tool • JPL performed the Test-Bed functional test

27. ACCS Interfaces Tested • Interface testing included the: • L0 file delivery • Level 1A reports • Alarm Notifications • HKT file delivery • Text files to Web graphs • Web graph publication • Download time file delivery

28. ACCS Functional Test Description • ACCS Functional Test – encompasses component and internal interface tests • 100% of Level 4 Requirements verified • Data Monitoring Test: • Correlated science and HKT data files from I&T processed through the full ACCS system • Included a DPU memory download • Command Planning Test • All Aquarius commands and known sequences planned • Each singular command • Downlinks • LUT programming • DPU memory download • Flight software patch • Introduced illegal times, sequences, parameters • Anomaly Handling Test • Simulated use of Recording and Tracking tool in anomaly resolution process • Included file exchanges, milestones, notifications • Simulated science and instrument team external access

29. Command and Telemetry Database Tests • Command and Telemetry Database Testing • Command Database • Worked with JPL to verify hex output of CONAE SInter tool matched output of I&T commanding tool • Exercised all legal commands and wide range of legal and illegal command parameters • Telemetry Database – completed in instrument I&T at JPL • The telemetry database is the telemetry dictionary as captured in Telemetry Viewer scripts. • Scripts define how: • Telemetry bytes are extracted from telemetry frames • Hex values are converted to Engineering Units • Alarms are applied • Telemetry Viewer pages display hex, EU values and alarms • Ultimately compared to values from trusted sources: • Power meters • Calibrated sensors • Software tools

30. AQ/SAC-D GS External Interface and Verification Test Lists These external interface and verification tests are CONAE-led and will verify Aquarius Ground System Level 3 Requirements: • SDGS-GSD-I-0042-a Science data delivery • SDGS-GSD-I-0044-a ST TM delivery • SDGS-GSD-I-0024-a AR Reception and Confirmation Integration Test • SDGS-GSD-I-0025-a Action Reports Request and Response Integration Test • SDGS-GSD-I-0028-a Voice Communication Integration Test • SDGS-GSD-I-0026-a Orbit Products delivery • SDGS-GSD-V-0002 X band data publication • SDGS-GSD-V-0004 MOC ancillary data production and delivery • SDGS-GSD-V-0006 Pass plan generation • SDGS-GSD-V-0008 Script validation and compilation • SDGS-GSD-V-0012 Execution report generation • SDGS-GSD-V-0011 Flight Software replacement

31. Status of Procedures

32. Key Aquarius Operations Procedures The key procedures for Aquarius instrument operations are: Aquarius Command Planning Aquarius Monitoring and Trending Aquarius Anomaly Response ACCS Configuration Management And the individual standard and contingency instrument procedures.

33. ACCS Procedures: Status and Plans for Completion • Key ACCS procedures are in preliminary form. • Extracted from the ACCS Functional Test Procedure. • Exception: Configuration Management Procedure • This procedure provides the steps for performing configuration management • It also identifies what items are to be maintained under CM: • Processes, Procedures • Tool software (including CONAE deliverables) • Flight Software/patches • Command and Telemetry scripts (databases) • Tests and test reports • ACCS Procedures must be updated in lockstep with the Functional Test. This is expected with: • Expansion of Functional Test to encompass Joint Tests • Corrections to Functional Test as Joint and Mission Tests progress

34. Aquarius Instrument Procedures, Standard and Contingency • Instrument standard and contingency procedures identified to date (except deployment): • Standard Procedures • AQ Start downlink • AQ Stop downlink • Contingency Procedures • Temperature Stability Failure • ATC to Constant Power • ATC PI parameter change • ATC Reset • ATC Set-point change • ICDS Fails to Accept Commands • ICDS Reset Ground Initiated • No AQ Science Data Down-linked • FIFO errors • Aquarius Power-down • Aquarius Power-up • Recovery from ICDS reset • Patching • Replacement Heaters • Turn Scatterometer Off • Scatterometer Exciter Out of Limits • Scatterometer Beam Mode Select • Scatterometer Receive Only • Scatterometer Receive Window Adjust • Scatterometer reset Chirp Generator • Scatterometer Select Redundant CG • Scatterometer Attenuator Settings Select • Scatterometer Transmit Mode • Scatterometer RFI Flag Threshold Adjust • Scatterometer science anomaly • Radiometer Custom LUT upload • Radiometer Integration Window Adjust • Radiometer Memory Dump • Radiometer Select Operational LUT • Radiometer reset (soft, hard) • Radiometer Science Anomaly

35. Instrument Operations Procedures Status • Individual Aquarius procedures: • Have been drafted by the instrument team as flowcharts (where needed) with command sequences and telemetry responses identified. AND/OR • Exist as ATLO scripts/procedures • Work to be done: • The Aquarius team should come to consensus on which procedures are operations-critical. • The SAC-D Flight Operations Team and the Aquarius Instrument Ops Team should rehearse/observe these critical procedures.

36. Generating Instrument Operations Procedures • Aquarius flows and ATLO scripts will be transferred to operations procedures • The transfer steps from ATLO scripts to operations procedures are: • Correlate specific ATLO/I&T scripts to each operations procedure • Parse the scripts by decision point. ACCS procedures direct the use of the CONAE Aquarius procedures. Thus: • ACCS procedures include the decisions within a given ATLO script • CONAE procedures include the steps between decisions • Write the ACCS procedures and the CONAE procedures (scripts) according to the parsed ATLO/I&T scripts • Rehearse/observe the ACCS and CONAE Aquarius procedures against the Observatory. AND/OR • Rehearse the ACCS and CONAE procedures in mission scenarios.

37. ACCS vs CONAE Procedures • Two types of instrument procedures: • ACCS • CONAE Flight Operations • ACCS procedures are documents in step-by-step format • All decision points occur within the ACCS procedures • CONAE procedures are scripts • CONAE Aquarius scripts are written in the Sinter scripting language • The scripts consist of: • Aquarius commands established in known sequences • Associated telemetry responses • Any other real-time activities required • Examples: • Inserted delays between commands • Special data logging • Information to note before proceeding

38. Risks/concerns

39. Risk Assessment • After an extensive risk/concern discussion during the Aquarius Instrument Peer Review and follow-up discussions with the Aquarius team, two risks/concerns remain. • The two risks/concerns are listed by priority from the instrument operations perspective.

40. Risk RISK: Personnel availability may be too limited to allow for implementing the existing command sequences as procedures and rehearsing them. IMPACT: The Operations Team would then be inadequately trained for Aquarius operations. Mitigation: Project is planning a workshop in Bariloche in the April timeframe to rehearse critical procedures.

41. Concern CONCERN: Awaiting Scheduler Client delivery. IMPACT: Unable to develop adequate Aquarius command planning procedure in time for interface tests and scenarios. Interface testing of the command planning link could be delayed. Mitigation: Scheduler Client User’s Guide and ongoing conversations with developers have provided baseline information for ACCS tool and procedure development.

42. Description and Status of Documentation

43. Documentation/Status

44. ACCS – CONAE Rec/Dels - Status

45. Instrument Team – ACCS Rec/Dels - Status

46. Summary The Aquarius instrument operations team and the ACCS tools are ready to move into the operations testing phase. The testing, process development and procedure development is working in conjunction with SAC-D ground system and Aquarius instrument testing.