What do we Know about the Effectiveness of Instructional Strategies in Computer Games

1. What do we Know about the Effectiveness of Instructional Strategies in Computer Games Harold F. O’Neil and Joan (Yuan-Chuang) Lang University of Southern California/CRESST AERA v.4 New York March 2008

2. CRESST Model of Learning Content Understanding Collaboration/Teamwork Learning Problem Solving Communication Self-Regulation

3. Instructional Strategy • Definition • Prescribed sequences and methods of instruction to achieve a learning objective • Source Spector, J.M., Merrill, M.D., van Merrienboer, J., & Driscoll, M.P. (2008). Handbook of Research on Educational Communications and Technology. New York: Taylor & Francis Group. • Macro vs Micro Instructional Strategies - Do games work? (macro) - What strategies within games work? (micro)

4. Integrate into Syllabus Instructor training accuracy re standards Limited role for fantasy Integrate into recreation/family time Increased requirement for effective micro instructional strategies (feedback, worked examples) Student “training” need (e.g., self-monitoring) Flexible Fantasy role Macro Instructional Strategies Issues If in class If out of class

5. Motivation provided by instructor Game can be less fun Good application Initial acquisition training Available time in hours/days/weeks Motivation provided by student/game Game needs to be fun Good application Refresher training; prevent skill decay Available time in minutes/hours Macro Instructional Strategies If in class If out of class

6. Research Questions • Will adding effective instructional strategies to commercial off the shelf games improve problem solving? • Trade-off between development and selection

7. Do Games Teach?: Check Validity of Micro Instructional Strategy • Embedded in game • Usually inductive discovery approach • Usually doesn’t result in learning (Kirschner, P. A., Sweller, J., & Clark, R. E. 2006. Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based learning. Educational Psychologist, 41, 75-86.) • What Works in Distance Learning (O’Neil, 2005) • Good instructional practices that can be applied to games

8. Selection of Game for Research • Off-the shelf games lacking learning objectives and assessment of learning • Use wrap around instructional & assessment strategies as no access to source code

9. Common Methodology • Participants: • Young adults selected to have no experience of playing SafeCracker but game players • Measures: • Knowledge mapper • Retention and transfer questions analogous to Mayers’ • Trait self-regulation questionnaire

10. Domain Specific Problem-Solving Strategy Measures • Retention Question List how you solved the puzzles in the rooms. • Transfer Question List some ways to improve the fun or challenge of the game. • Modifications of previous researchers (Mayer, 2001; Mayer & Moreno, 1998; Moreno & Mayer, 2004)

11. Knowledge Mapper

12. Measurement of Self-Regulation • Trait self-regulation questionnaire (O’Neil & Herl, 1998). • planning • self-checking • self-efficacy • effort

13. Study I, II, &III • Study I– Without effective instructional strategies. • Study II– With worked examples. • Study III– With just-in time worked examples O'Neil, H. F. and Perez, R. S. (Eds.) (2008). Computer Games and Team and Individual Learning. Elsevier

14. Study I:Discussion/Implications • There was an increase in problem-solving. But it was small. • Existing instructional strategies (discovery learning) in the game were not effective. • More research on a game designed with effective research-based instructional strategies • Worked examples (Danny Shen) • Pictorial aids (Richard Wainess) • Just-in-Time Worked Examples (Joan Lang) • After-Action Review

15. Study II and Study III: Worked Examples • Worked examples are procedures that focus attention on problem states and associated operators (i.e., solution steps), enabling students to induce generalized solutions or schemas (Sweller, 1998). • Many researchers investigated the efficacy of using worked examples in classroom and computer-based instruction and provided evidence of the effectiveness of worked examples instruction (Cooper & Sweller, 1987; Mayer & Mautone, 2002; Ward & Sweller, 1990). • No research used worked examples in a game-based environment.

16. Study Worked Example: Discussion/Implications • The worked example group significantly improved more than the control group in content understanding and problem-solving strategies. However, the improvement was small. • This study provided evidence that using worked examples could be one of the good instructional methods to facilitate adults’ problem solving with a commercial off-the-shelf computer game.

17. Just in Time Worked examples What worked examples can do? • Facilitate learning of problem solving in the computer game SafeCracker® • Contribute to superior content understanding in problem solving (when presented just-in-time) • May contribute to superior retention skill (partial evidence) but not transfer skill

18. What Are Continuing R&D Issues? • Can we leverage game technology for training? • Embedded instructional and assessment strategies • Wrap-around instructional and assessment strategies

19. Walk Issues • Analytically, would you predict that commercial off-the-shelf games should teach? • No • What support and guidance would help training game developers to do a better job? • Alignment with What Works in Distance Learning • Instructional strategies that could work • Wrap-around or embedded instructional and assessment strategies • Worked examples • feedback

20. CRESST Web Site http://www.cresst.org or any search engine: type CRESST honeil@usc.edu

21. What Is a Game? • A computer game consists of four key components • Settings that are real or imaginary • Roles or agendas for the participants • Rules (real life vs. imaginative) • Scoring, recording, monitoring, or other kinds of systematic measurement • Motivation comes from challenge, complexity, fantasy

22. What Works in Distance Learning* • Develop a core set of research-based DL guidelines and lessons • Instructional Design (Clark)* • Multimedia (Mayer)* • Learning Strategies (Dembo et al.)* • Assessment (Baker et al.)* • Management Strategies (Kazlauskas)* • Self-Regulation and Motivation (O’Neil et al.)* • Team Training and Assessment (Cannon Bowers) • Team Motivation (Clark) *O’Neil, H. F. (Ed.). (2005). What works in distance learning: Guidelines. Greenwich, CT: Information Age Publishing Inc.

23. Sample of a Worked Example Search for the code related to the Liberty Safe Find a Note from switchsafe in the toolbox Click the Note from switchsafe The Note from switchsafe will show on the screen Notice that some codes are missing Click the Note to close it

24. The Specification of What We Are Teaching Is Essential • From goal/objective of teaching leadership, situational awareness, decision making, tactical problem solving • The instructional strategies follow • Nature of feedback, timing of feedback, take-home packages, instructor training, homework assignments, etc. • The type of assessment follows • Different assessment measures, after-action reviews

25. SafeCracker • Puzzle-solving game • Example of problem solving • No special background knowledge, motor skills, or extraordinary visual-spatial ability required • Adult-oriented • Single-player game • Pacing controlled by players • Not popular

26. Results: Content UnderstandingBold = as hypothesized • The JIT worked example (M = 7.97) group demonstrated significantly better (p = .01) content understanding than the worked example group (M = 6.63) and the control group (M = 6.51) • The worked example group did not demonstrate significantly better content understanding than the control group

27. Results: Problem Solving Strategies Retention skill: • The worked example group (M = 2.72, SD = .72) demonstrated significantly greater retention skill than the control group (M = 2.03, SD = .88) by opening significantly more worked-example-illustrated safes (p = .01) Transfer skill: • No difference among groups