Mining for Problem-solving Styles in a Virtual World

1. Mining for Problem-solving Styles in a Virtual World Brian M. Slator, Dept. of Computer Science Donald P. Schwert and Bernhardt Saini-Eidukat, Dept. of Geosciences; North Dakota State University, Fargo, ND

2. NDSU WWWIC World Wide Web Instructional Committee Jeff Clark Paul Juell Donald Schwert Philip McClean Brian Slator Bernhardt Saini-Eidukat Alan White WWWIC faculty supported by large teams of undergraduate and graduate students WWWIC’s virtual worlds research supported by NSF grants DUE-9981094 and EAI-0086142

3. The Geology Explorer Project Educational Game designed to provide authentic learn-by-doing experience •Exploration of a spatially oriented virtual world •Practical, field oriented, expedition planning and decision making •Scientific problem solving (i.e., a “hands on” approach to the scientific method

4. Balancing Pedagogy with Play Games have the capacity to engage! • Powerful mechanisms for instruction • Illustrate real-world content and structure • Promote strategic maturity (“learning not the law, but learning to think like a lawyer”)

5. Advantages of Virtual Worlds • Collapse virtual time and distance • Allow physical or practical impossibilities • Participate from anywhere • Interact with other users, virtual artifacts, and software agents • Multi-user collaborations and competitive play

6. Technical Approach • Networked, internet based, client-server simulation • UNIX-based MOO (Multi-User Dungeon, Object Oriented) • Java-based clients (text version - telnet based; graphical version in development)

7. The Game •Planet Oit - similar to Earth, but opposite the Sun •Students “land” on Oit to undertake exploration •Authentic Geoscience goals - e.g., to locate, identify, and report valuable minerals

8. The Simulation ~50 places: desert, cutbank, cave, etc. ~100 different rocks and minerals ~15 field instruments: rock pick, acid bottle, magnet, etc. ~Software Tutors: agents for equipment, exploration, and deduction

9. Real World of Planet Earth

10. The Geology Explorer: Planet OitGame Scenario • You are a geologist. • Explore this new planet. • Authentic geologic goals. - Locate and report valuable minerals. • Must learn geologic content.

11. The Geology Explorer • 50 Places • 90 Different Rocks and Minerals • 15 Field Instruments • 25 Laboratory Instruments • Software Tutors

12. Maps of Planet Oit

13. The Geology Explorer

14. Virtual Field Instruments

15. Subjective Assessment Rejects the notion of standardized multiple choice tests Pre-game narrative-based survey • short problem-solving stories • students record their impressions and questions Similar post-game survey with different but analogous scenarios Surveys analyzed for improvement in problem-solving

16. Assessment • Not “multiple choice” recall • Content specific: • Problem solving, Hypothesis formation, Diagnostic reasoning

17. Assessment by Scenarios • Assess computer literacy • PreTest: Present scenario, students propose course of action or solution • Engage in learning experience Control vs Virtual • PostTest: Present similar scenario, student response • Analysis of assessment data

18. The Geology Explorer: Assessment Protocol, Fall, 1998 Pre-course Assessment: 400+ students Computer Literacy Assessment: (244 volunteers) Divide by Computer Literacy and Geology Lab Experience Geomagnetic (Alternative) Group: (122 students) Non-Participant Control Group: (150 students, approx.) Geology Explorer Treatment Group: (122 students) Completed (78 students) Non-completed (44 students) Completed (95 students) Non-completed (27 students) Post-course Assessment: 368 students

19. Intelligent Software Tutoring Agents Diagnostic Tutors 1. Equipment tutor 2. Exploration tutor 3. Science tutor Detects when a student makes a wrong guess and why (i.e. what evidence they are lacking); or when a student makes a correct guess with insufficient evidence (i.e. a lucky guess)

20. Tutors are NeededIn Virtual Environments: • Students can join from any remote location • They can log in at any time of day or night • Human tutors cannot be available at all times to help • Students can foul things up and not know why

21. Tutors are NeededIn Virtual Environments: • Information is readily available • The simulation can track actions • The simulation can generate warnings and explanations • Tutor “visits” are triggered by user action

22. Tutors are NeededIn Virtual Environments: • Student interact with the intelligent tutoring agent • Students can ignore advise and carry on at their own risk

23. Learning Style Complete history record for gmercer@9 (#11347) as of Mon Mar 5 21:03:06 2001 Central Standard Time Sep14/09:39 assigned original goal: Sphalerite Goal Sep14/09:39 connected TO MOO Sep14/09:44 entered YOUNG MOUNTAINS (#132) Sep14/09:44 Equipment Tutor says needed Streak Plate to find Sphalerite Goal [...] Sep14/09:49 purchased Streak Plate (#12191) for $0.5 [...] Sep14/09:50 entered YOUNG MOUNTAINS (#132) Sep14/09:51 entered CAVE (#341) Sep14/09:51 entered CAVE (#275) Sep14/09:51 Exploration Tutor says overlooked goal in Cave of Sphalerite Goal Sep14/09:52 entered CAVE (#341) [...] Sep14/09:53 entered ROCK MUSEUM (#594) Sep14/09:54 entered THE MINERAL COLLECTION (#1796) [...] Sep14/10:04 entered CAVE (#275) Sep14/10:04 Exploration Tutor says overlooked goal in Cave of Sphalerite Goal [...] Sep14/10:11 streak yellowish brown resinous vein (#1998) (#1998) with Streak Plate (#12191) (#12191) results: "yellow" Sep14/10:18 reported yellowish vein (#1998) as Native Gold (#657) scoring 0 points {968944698, "Geology Tutor", #1840, #341, "Said, 'Native Gold has a yellow metallic appearance. But the yellowish brown resinous vein (#1998) does not.'"} Sep14/10:19 reported goal yellowish vein (#1998) as Sphalerite (#560) Sep14/10:19 Previously assigned goal solved -- new goal assigned Sep14/10:19 assigned Native Copper Goal scoring 100 points [...]

24. Learning Style • Patterns we noticed: • analytical approach: frequent reference to on-line help, conducting sequences of experiments, deliberative: many experiments • pattern-matching approach: exploring far and wide in search of their goals: many movements • “brute force” approach: simply visiting location after location and identifying everything: many reports

25. Reports Moves Experiments average 42.6 139.2 73.8 st. dev 38.2 83.1 63.2 min 5 19 0 max 238 518 301 Learning Style

26. rme 10 -ME 5 -Me 4 r-e 8 --E 4 rM- 2 r-- 5 R-E 4 r-E 2 -m- 5 R-- 3 RmE 2 -me 4 RME 3 --- 2 --e 4 RM- 3 Rm- 1 rm- 4 -M- 2 -mE 1 rmE 1 R-e 1 Rme 1 Total 40 (49.4%) 24 (29.6%) 17 (21.0%) Learning Style Consistently normal or below normal activity Consistently normal or above normal activity Mixed problem-solving activity Note: R = many reports; r = few reports; M = many moves; m = few moves; E = many experiments; e = few experiments. Example: “-Me” means normal reporting, many moves, below normal experiments (where normal is within one-half standard deviation from the mean).

27. Learning Style • A wide range of approaches are supported • Questions: • Are some of the “pattern matchers” really “curious explorers? • Is there such a thing as TOO much experimentation? • Will software tutors effect what we’re seeing? • How can the game encourage a more analytical approach? • Are students “gaming” the system?

28. http://oit.cs.ndsu.nodak.edu World Wide Web Instructional Committee (WWWIC) North Dakota State University Fargo ND