Measuring Healthcare Outcomes using Serious Games, Gamification, and Virtual Reality

1. Measuring healthcare outcomes using serious games, gamification, and virtual reality Todd P Chang, MD MAcM serious play conference 2018.07.11 (Manassas, VA)

2. Disclosures / Conflicts of Interest American Heart Association National Board of Medical Examiners Oculus from FaceBook • • • Independent Contractor with SBC Med Sim, LLC • 2

3. WHY MAKE HEALTHCARE GAMES AT ALL? 3

4. We do it for the patients 4

5. Objectives 1 2 3 Map out measurable Outcomes for both Mediating & Moderating games Define the 4 Outcome levels for measuring Effectiveness of a Game Discuss three case studies measuring effectiveness of Games on patient outcomes

6. HOW CAN GAMES INFLUENCE PATIENT OUTCOMES?

7. 7

8. Landers et al. 2014 Sim & Gaming Mediating Moderating • Knowledge • Skill • Attitude 8

9. MEDIATING Going through the game improves KSA 9

10. 10

11. 1000 CUT JOURNEY Virtually experience life as a black child

12. Landers et al. 2014 Sim & Gaming Mediating Moderating • Knowledge • Skill • Attitude 12

13. The game or gamification elements improve engagement of an otherwise effective tool MODERATING

14. 14

15. Landers et al. 2014 Sim & Gaming Mediating Moderating Going through the game improves KSA The game improves engagement of an effective tool 15

16. Outcome 00 Mood prior to VR Outcome 01 Use Frequency Outcome 02 Change in Mood 16

17. Outcome 04 Health Metric (Result) Outcome 03 Knowledge Application (Behavior) Outcome 00 Prior Outcomes Outcome 01 Use Scores Outcome 02 Knowledge Gain 17

18. Outcome 03 Health Metric (Result) Outcome 01 Use (Behavior) Outcome 02 Attitude 18

19. What if the game user is the healthcare provider, not the patient? 19

20. 20

21. OUTCOME LEVELS Objective 2: Kirkpatrick’s 4 Levels of Evaluation

22. Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Sullivan GM. J Med Educ. 2011;3(2):121-4.

23. Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Reaction Sullivan GM. J Med Educ. 2011;3(2):121-4.

25. Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Learning Reaction Sullivan GM. J Med Educ. 2011;3(2):121-4.

26. Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Behavior Learning Reaction Sullivan GM. J Med Educ. 2011;3(2):121-4.

27. Kirkpatrick’s Levels of Evaluation Evaluation and Feedback Results Behavior Learning Reaction Sullivan GM. J Med Educ. 2011;3(2):121-4.

28. Barsuk et al. Crit Care Med 2009;37(10):2697-701. Barsuk et al. Acad Med 2010;85(10 Suppl):S9-12. Barsuk et al. MJ Qual Saf 2014;23(9):749-56. • • •

29. Outcome C Prior Knowledge Outcome A Use Scores Outcome B Knowledge Gain

30. Outcome C Prior Knowledge Outcome A Use Scores Outcome B Knowledge Gain Outcome 04 Health Metric (Result) Outcome 03 Prior Patient State (Result) Outcome 01 Provider Action (Behavior) Outcome 02 Patient State (Result)

31. CAN MY GAME PREDICT HOW MUCH IMPROVEMENT WILL OCCUR? A question of Validity

34. Discriminant Validity Testing Expected Low Performers Expected High Performers Uncovers any bias due to: • Differences in Video Game Experience • “Game Cheating” behavior • Lack of Functional Fidelity in a Game • A different target audience

35. Gerard et al. Validity Evidence for a Serious Game to Assess Performance on Critical Pediatric Emergency Medicine Scenarios. Simul Healthc 2018;13(3):168-80.

36. 3 CASE STUDIE S

37. HOW TO MEASURE ‘MULTI-PATIENT CARE’? A discriminant validity study

38. Background • With higher patient volumes but fewer providers, multi-patient care is now common • Poor multi-patient care leads to increased errors • Multi-patient care is not routinely taught nor universally assessed

39. Purpose To validate a serious games version of a pediatric ED that measures multi-patient care based on experience: 1. Undergraduate students (jrs, srs) 2. Medical students (MS-II) 3. Resident physicians (PGY-1 through 3) 4. Attending / Fellow physicians (PGY-4+)

40. Acknowledgments to the Stemmler Fund & BreakAway Games, Ltd

43. 43

45. Outcomes & Analyses Predictors Experience Multi-tasking ability Video Game frequency Game Outcomes % Sentinel Orders Time to 1stSentinel Order Time to 2ndSentinel Order Time to Discharge % Patients Seen # correct Diagnoses # Differential Diagnoses • • • • • • • • • • 45

46. Results Undergrads 11 MSIIs 22 Residents 15 Attgs 23 n 46

47. Results # Patients Seen 100 75 50 25 0 Undergrad MSII Residents Attgs 47

48. Results Time-to-Critical Action (game seconds) 750 500 250 0 Time to SO1 Time to SO2 Time to d/c Undergrad MSII Resident Attg 48

49. Results • MANCOVA analyses revealed a statistically significant difference in game performance after controlling for video game experience & multi- tasking ability. • The two outcomes that the game was able to distinguish expertise in were: – # sentinel orders – % correct diagnoses

50. The one-way MANCOVA showed that there was statistically significant difference (p=0.03) between the skill groups on the combined dependent variables after controlling for game play frequency, F(21, 153) = 0.700, p = .828, Wilks' Λ = .768, partial η2 = .084, and for MTAT placement score, F(21, 153) = 1.365, p = .144, Wilks' Λ = .610, partial η2 = .152. Follow up univariate one-way ANCOVAs were performed. A Bonferroni adjustment was made such that statistical significance was accepted at p < .008. There were statistically significant differences in adjusted means for total percentage of sentinel orders completed (F(3, 59) = 31.702, p < .0005, partial η2 = .617), and number of correct diagnoses made per patient seen(F(3, 59) = 21.441, p < .0005, partial η2 = .522). 50