Effective Testing of Healthcare Simulation Software

1. Effective Testing of Healthcare Simulation Software Presenter: ShantMandossian

2. Simulation Software • Modeled after real world systems. • Allows users to observe an operation without actually performing it. • Used in initial system designs in order to optimize component selection.

3. Simulation Software Two Different Types • Discrete Event Simulation • Continuous Simulation

4. Simulation Software ContinuousSimulation

5. Simulation Software Continuous Simulation

6. Simulation Software Discrete Event simulation

7. Simulation Software Discrete Event simulation

8. Simulation Software Discrete Event simulation • Input • Agent Resources • Non-Agent Resources • Artifacts Significant Components • Oracle • Resource Manager • Output

9. Simulation Software TestingCommon methods • Research in simulation software focuses on validation • Insufficient investigation into testing the software itself.

10. Simulation Software TestingWhy is it Difficult? • Simulation software exists to find the answers in the first place. • No test oracle exists because exact simulation results are simply not known. • Subtle errors in implementation can greatly affect the accuracy of the simulation.

11. Simulation Software TestingWhy is it Difficult? • Simulations consist of many interacting components • Each step or event can take a random amount of time to complete.

12. Simulation Software TestingN-Version Programming • Early approach to testing software without a test oracle • Independent programming teams develop an application from the same specifications. • Identical sets of input are fed into each version of the program. • Results are compared in an effort to detect defects.

13. Simulation Software TestingMetamorphic Testing • Use of Metamorphic Testing instead of N-Version Programming • Say that produces • After identifying the metamorphic properties of , we alter them in some way to make a transformation function . • Now we produce • We should be able to predict the results of based on the alterations we made to . • If the results are not as expected, there must be a defect somewhere.

14. Simulation Software TestingMetamorphic Testing • The Bad • Like most tests, this method only shows us if defects exist. • Not that the software is defect-free. • The Good • Provides a way of revealing potential defects without the help of a test oracle

15. Simulation SoftwareHealthcare • Healthcare professionals rely on simulation software

16. Simulation SoftwareHealthcare • Software is used to simulate elements of the human body for purposes of training and decision making.

17. Simulation SoftwareHealthcare • Simulations of a specific organ like the heart. • Simulations of an entire physiological system.

18. Simulation SoftwareHealthcare • Emergency Department

19. Simulation Software TestingEmergency Department Simulator Process • Patient arrives • Patient is seen by a triage-nurse • Patient registers with the clerk • Clerk collects information puts in the patient record • If a bed is available, patient goes to treatment area • If all beds are occupied, patients waits in the waiting room until a bed is cleared • Patient is taken to a free bed • Patient is seen by a nurse • Doctor takes assessment of the patient • Assessment may result in tests • Once tests are done, doctor admits or discharges the patient • Nurse fills out paperwork.

20. Simulation Software TestingEmergency Department Simulator • Agent Resources • Clerk • Triage Nurse • Nurse • Doctor • Non-Agent Resources • Beds

21. Simulation Software TestingEmergency Department Simulator Finding Metamorphic Properties • All resources used in simulation • Doctors, Nurses, beds, etc… • Input given to the simulator • Simulation instructors • Program source code

22. Simulation Software TestingEmergency Department Simulator Potential Bottleneck • Patients wait until a bed is available. • Run multiple simulations with varying numbers of beds, but keep all other resources constant. • Plot average Length of Stay (LOS) for each run-through of the simulator to see if results are as expected.

23. Simulation Software TestingEmergency Department Simulator Results show that LOS improves with more beds, but the amount of improvement decays.

24. Simulation Software TestingEmergency Department Simulator Plotting the utilization of doctors over the number of beds shows that on average, utilization increases but also with a decay

25. Simulation Software TestingEmergency Department Simulator • Both Length of Stay (LOS) of patients vs. number of beds and Doctor utilization vs number of beds, displayed expected results. • New test case with unexpected results. • Five patients arrive, 6 minutes apart. • Resources Available • One nurse, doctor, clerk, and triage nurse. • Three beds. • Results • Average LOS for each patient = 217.4

26. Simulation Software TestingEmergency Department Simulator

27. Simulation Software TestingEmergency Department Simulator

28. Simulation Software TestingEmergency Department Simulator

29. Simulation Software TestingEmergency Department Simulator • With One Nurse • Patient 4 was ready to be discharged by the doctor before patient 5 was ready to be seen by the same doctor. • With Two Nurses • Patient 5 was ready to be seen by the doctor before patient 4 was ready to be discharged. This is unexpected behavior.

30. Simulation Software TestingTesting the Test Method • Testing the effectiveness of Metamorphic Testing • Mutation testing is used. • Defects purposely injected into the source code. • 25 out of 104 generated mutants were used in the test • Many mutants were excluded for various reasons. • Caused obvious errors and crashes • Were caught by exception handling.

31. Simulation Software TestingTesting the Test Method Metamorphic testing successfully caught all mutation defects.

32. Simulation Software TestingOther Methods • Other testing methods (instead of metamorphic) • Assertion checking.

33. Simulation Software TestingEmergency Department Simulator • Assertion checking • Wasn’t used here in comparison • Expected to perform less successfully. • Example • Off by 1 error for range [a, b-1] instead of [a, b] • Will never produce an out-of-range result, causing the assertion test to miss the defect. • Metamorphic testing can take on the statistical mean and variance over many run-throughs and find an inconsistency.

34. Simulation Software TestingConclusion and Summary • Simulation Software • Discrete Event Simulation • Testing Simulation Software • Relation to healthcare software • Difficult to test • Metamorphic Testing • Mutation testing on Metamorphic testing • Assertion Checking