LSST Telescope & Site Software Testing: Tools, Process, and Results Reporting

1. LSST Telescope & Site Software TestingRob BovillTelescope & Site Software Test EngineerLSST 2018 Project and Community WorkshopAugust 14, 2018

2. Agenda • Software testing process • Tools • Automated/Manual • Issue tracking/Project management • Results reporting • Status • Future goals and plans • Questions and Discussion Highlight who I am and what I do

3. Agenda • Software testing process • Tools • Automated/Manual • Issue tracking/Project management • Results reporting • Status • Future goals and plans • Questions and Discussion

4. Documentation • LTS-699 – Telescope and Site Software Test Process • Methods • Exploratory, Regression, Functional and Acceptance • Strategies • Manual and Automated • Execution • Management • Roles and Responsibilities • Specific System Test Plans

5. Testing Process • Primary responsibility is to provide data to assess software status • Start with manual walk-through • Exploratory testing • “Happy-path” – determine if software does what it is supposed to do • Translate this work into test scripts • Iterative process • Explore “off-nominal” interactions • Determines if software does NOT do what it is NOT supposed to do • Insufficient arguments, incorrect type, improper configuration, etc. • Run the tests repeatedly • Feedback to/from developers and customers • Bugs, Improvements

6. Types of Testing • Software testing teams DO NOT perform Unit testing • Exploratory • Functional • Verify functional requirements • Regression • Verify application performs the same way after a change • Ensures bugs (regressions) do not return • Acceptance • Verify software functionality in a new environment (delivery) • Integration • Testing software applications when combined into the system

7. Agenda • Software testing process • Tools • Automated/Manual • Issue tracking/Project management • Results reporting • Status • Future goals and plans • Questions and Discussion

8. Automated Testing • Best suited for Regression testing • Quick and repeatable • Validates issues do not return • Does not test new features (until written) • Writing tests takes time • Results stored online • https://mebsuta.lsst.org/~rbovill/ • Tests currently maintained in GitHub • https://github.com/lsst-ts

9. Automated Testing - Tools • Robot-Framework • Open-source • Extendable with test libraries • Well supported • https://github.com/robotframework/robotframework • https://groups.google.com/forum/?hl=en#!forum/robotframework-users

10. Continuous Integration • Jenkins - an open source continuous integration tool • https://ts-ci.lsst.codes/ • Integrates with Robot-Framework • SAL, EFD, XML • Scheduler team is exploring build management with Jenkins • Used by Pathfinder teams to verify readiness

11. Manual Testing • Only performed when automated testing is not feasible • One-off applications (ILC Calibration) • Simple applications • Tests run infrequently • Results stored online • https://atlassian.lsstcorp.org/~rbovill/ • Tests currently maintained locally • DocuShare? Confluence? Other?

12. Agenda • Software testing process • Tools • Automated/Manual • Issue tracking/Project management • Results reporting • Status • Future goals and plans • Questions and Discussion

13. Issue Tracking & Project Management • Atlassian JIRA • Task management • New features/improvements • Bugs/issues • Planning - Agile development process • QA closes ALL (TSS) JIRA issues • Except our own tasks

15. Jira Workflow • Issue tracking process • Anyone can open a bug or improvement ticket • Ticket is automatically assigned to Component owner • Triage • Validity • Duplicate • Correct attributes • Process through the workflow • External vs Internal facing Jira projects • Tickets from External project duplicated into Internal project

16. Agenda • Software testing process • Tools • Automated/Manual • Issue tracking/Project management • Requirements traceability • Results reporting • Status • Future goals and plans • Questions and Discussion

17. Testing Results • Results stored online • https://mebsuta.lsst.org/~rbovill/ • https://ts-ci.lsst.codes/ • Email sent to all interested parties when results are posted • TSSW team • CAMs • T&S management • T&S Systems Engineer • T&S Chief Engineer

18. Results Deep Dive

19. Robot-Framework Results • Pass -> <salgen_results_green.tar.gz> • Fail -> <salgen_results_red.tar.gz>

20. Jenkins Continuous Integration • https://ts-ci.lsst.codes/

21. Agenda • Software testing process • Tools • Automated/Manual • Issue tracking/Project management • Results reporting • Status • Future goals and plans • Questions and Discussion

22. Testing Status • Complete • ILC (Inner Loop Controller) calibration software • FAMT (Force Actuator Model Tester) • ILC Firmware • Current • XML validation • SAL (Service Abstraction Layer) • M1M3 Support System • EFD schema validation

23. Agenda • Software testing process • Tools • Automated/Manual • Issue tracking/Project management • Results reporting • Status • Future goals and plans • Questions and Discussion

24. Testing Next Steps • Next • AuxTel • AOS SW (OFC and WEP) • Scheduler • Future • M1M3 Thermal Control • Summit Facility

25. Testing Goals • Software Integration Testing • Dave’s simulation cluster? • Mock control room? • Dedicated software testing environment? • Automate the Pathfinder Activities • Deployment validation • Lab (Patrick Ingraham and Tiago Ribero) • Puppet deployments (Andrés Villalobos) • EFD performance

26. What I Need • Feedback • Technical Support • Jenkins • Simulators • Proper Unit testing • Stable code • XML, SAL

27. Agenda • Software testing process • Tools • Automated/Manual • Issue tracking/Project management • Results reporting • Status • Future goals and plans • Questions and Discussion

28. Questions and Discussion