Valerie Shute, Florida State University

1. A D A P T I V E S Y S T E M S • Valerie Shute, Florida State University ARI Workshop on Adaptive Training Technologies, Charleston, SC (March 3-5, 2009)

2. Shute, V. J. & Zapata-Rivera, D. (2008). Adaptive technologies.In J. M. Spector, D. Merrill, J. van Merriënboer, & M. Driscoll (Eds.), Handbook of Research on Educational Communications and Technology (3rd Edition) (pp. 277-294). New York, NY: Lawrence Erlbaum Associates, Taylor & Francis Group.

3. Simple Logic Adaptive Content Diagnosis Assessment Evidence

4. My Plan • Definitions (foundation) • Rationale (motivation) • Four-process adaptive cycle (frame) • Current technologies (hard & soft) • Concrete example (via my bag) • Future visions (briefly)

6. Definitions

7. 1. Adaptivity • Refers to a natural or artificial system’s ability to alter its behavior (etc.) according to the environment. • Adaptive technologies (hard/soft) allow an instructional system to alter its behavior according to learner needs (etc.). • Typically linked with a learner model (see next slide).

8. 2. Learner Model • Representation of a learner, maintained by an adaptive system. • Models can be used to give personalized assistance to individuals based on cognitive and noncognitive aspects of their profile. • Learner models have been used in many areas, especially advanced educational and training systems.

9. 3. Hard Technologies Eye tracking device • Devices used in adaptive systems to capture learner info or present content. • Used to detect performance data or affective states (e.g., boredom, excitement, confusion, etc.) or present stuff in a more accessible manner. • Best when coupled with soft technologies (next slide). Talking tactile tablet

10. 4. Soft Technologies • Usually algorithms, programs, or envir’s that broaden the types of interaction between learners and computer. • For example, an adaptive algorithm can be used in a program to: (a) select a task that provides the most info about a learner, or (b) suggest additional resources tailored to the learner’s needs.

11. 5. Adaptive Systems Actual room temperature Desired temperature Thermostat heating/AC system Heating/ cooling Temperature difference • AC systems monitor and adjust room temperature, and cruise-control systems monitor and adjust your car speed. • Similarly, adaptive educational/training systems monitor important learner characteristics and make (or suggest) appropriate adjustments to support and enhance relevant competencies.

12. 6. Goal of Adaptive Systems • … to create a sound and flexible environment that supports learning for persons with a wide range of abilities, disabilities, interests, backgrounds, traits, states, etc. • The challenge rests mainly on accurately identifying and estimating these learner variables then leveraging the info to improve learning/skill.

13. Rationale

14. Adapt? Why Why • People differ across countless dimensions. • Different dimensions are more/less suited for different types of instruction/training. • Adaptive systems can enhance learning/skill via extra practice opportunities, alternative multimedia options (especially useful to those with disabilities), tailored instruction/training, etc.

15. Adaptive systems are helpful/relevant in the world of business and education … And they are (and will be) of growing importance in terms of supporting U.S. Army’s evolving training needs.

17. Adaptive Cycle

18. 4-Process Adaptive Cycle Shute & Zapata-Rivera, 2008 • Adaptive technology is intended to support learning (effectiveness, efficiency, and/or engagement). • This requires accurate diagnoses. • Learner info used as basis for content selection. • Our 4-process cycle combines & extends: (a) a simpler 2-process adaptive model (Dx/Rx), and (b) a process model to support assessment (Mislevy, Steinberg, & Almond, 2003).

19. 4-Process Adaptive Cycle Learner Model Select Analyze Present Capture Learner

20. Alternative Cycles

21. Diagnosis Over Time Learner Information

22. Communication: Agents/Learners • Each agent maintains a personal view of the learner. • LM info and content can be distributed in different places. • Agents can communicate with each other directly or through an LM server to share information that can be used to help the learner achieve learning goals.

23. Overview of Technologies Analyze Select Capture Present Quantitative Techniques Qualitative Techniques • Performance data • Eye-gaze tracker • Speech capture • Gesture/posture • Haptic devices • . . . . • Personalized content • Multiple representations • Accommodations • Meaning equivalencies • . . . . Cognitive Variables Noncognitive Variables • Bayesian nets • Machine learning • Stereotypes • Plan recognition • . . . .

24. Experts’ Views What to adapt? • What variables should be taken into account when implementing an adaptive system? • What are the best technologies and methods that you use or recommend? How to adapt? Cristina Conati Jim Greer Tanja Mitrovic Julita Vassileva Beverly Woolf

25. What to Adapt?

26. How to Adapt?

27. Challenges The main barriers to moving ahead in the area of adaptive technologies include the following: Obtaining useful and accurate learning info on which to base adaptive decisions. Maximizing benefits to learners while minimizing costs associated with adaptive technologies. Addressing issues relating to learner control (of environment and LM) and privacy. Figuring out the bandwidth problem (re: scope of learner data). Valid LM Increase ROI Control/Privacy Grain size

28. Example

29. Diagnosis This is the part of the cycle on which I now focus. Sine qua non

30. Flow & Grow • Shute, V. J., Ventura, M., Bauer, M. I., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning: Flow and grow. In U. Ritterfeld, M. J. Cody, & P. Vorderer (Eds.), The Social Science of Serious Games: Theories and Applications. Philadelphia, PA: Routledge/LEA.

31. Games to Learning Games Flow Engagement Learning but… • Games lack assessment infrastructure • Assessments determine what’s been learned • Typical assessments disrupt flow • Thus we need stealth assessments in games • Evidence-centered design can accomplish this

32. Stealth Assessment • New advances in measurement let us to administer formative assessment (FA) during learning to • Extract ongoing, multifaceted info from a learner • Make accurate inferences of competencies • React in immediate and helpful ways. • When FA is so seamlessly woven into the fabric of the learning environment that it’s invisible, this is stealth assessment.

33. Introducing E C D

34. Assessment Design Competency Model What do you want to say about the person? Evidence Model What observations would provide best evidence for what you want to say? Task/Action Model Model What kinds of tasks let you make the necessary observations?

35. Design & Diagnosis Task Competency Evidence Stat Model Evidence Rules Capture Analyze Assessment Models & Metrics Monitor & Diagnose Success Competency Model: Organization of competencies & claims to be made about students, and current mastery estimates. Evidence Model: Criteria or rubrics for evidence of claim (i.e., specific student performance data; observables). Task Model: A range of templates and parameters for task development to elicit evidence needed for the evidence model.

36. Elder Scrolls IV Oblivion

37. Elder Scrolls IV: Oblivion • First person 3D RPG set in a medieval world • Can be one of many characters (e.g., knight, mage, elf), each who has (or can obtain) various weapons, spells, and tools • Primary goal—gain rank & complete quests (like America’s Army) • Quests may include locating a person to obtain info, figuring out a clue for future quests, etc. • Multiple mini quests along the way, and a major quest that results in winning the game (100s of hr of game play) • Players have the freedom to complete quests in any order

38. Quests: Problem Solving • In Oblivion (like AA), problem solving plays a key role in quests since the player has to figure out what to do and how to do it. • Problem solving often viewed as the most important cognitive activity in everyday & professional contexts, but it’s seldom explicitly assessed or rewarded in formal instructional/training settings. • Assessment and support of problem solving skills are very important to improve long-term learning potential.

39. Quests: Persistence • There are many character skills to improve in Oblivion which are frequency based (i.e., number of actions relative to a skill). • Learning to play the game and developing skills require many hours of game play, and many hours of game play implies persistence—in the face of success and failure. • Persistence has been shown to significantly predict achievement—in academic, business, and military worlds.

40. Quests: Attention • In many games (and combat games in particular), attention plays a key role in success. • In Oblivion, you need to attend to factors such as: health, fatigue, enemy maneuvering, escape plan, etc. • The central role of attention in learning has been demonstrated for decades.

41. Oblivion Competency Model Success in Oblivion Cognitive Noncognitive Creative Problem Solving Attention Domain Knowledge Problem Solving Creativity Persistence Efficiency Novelty Working Memory Reading Comp Listening Comp Speaking Skill Reflection Exploratory Behavior

42. Example ECD Models Competency Model Creative Problem Solving Problem Solving Creativity Efficiency Novelty Evidence Model Scene 1 Scene 2 Action Model Scene 1 Action Indicators Scene 2 Unobservables The Glue Observables

43. Action Model with Indicators Problem: Cross river filled with dangerous fish to get to the cave on the other side. * Relevant refers to any action included in a successful solution.

44. Indicators Per Action • Novelty: 1 – frequency • Efficiency: Inverse fn (resources, time) • Action: Find a bridge over the river • Indicators: Novelty = 1 - 0.34 = 0.66 • Efficiency = 1 / [(3 × 0.4) + (5 × 0.6)] = 0.24 • Resources Used = Weapon (1, fight monster with sword) + Health (1, damage from monster) + Object (1, magic potion) = 3 resources (weight = 0.4) • Time expended = 5 minutes (weight = 0.6)

45. Bayes Model—Case 1 CreativeProblemSolving Low 0.60 High 0.40 ProblemSolving Creativity Low 0.64 Low 0.11 High 0.36 High 0.89 Efficiency Novelty Low 0.86 Low 0.02 High 0.14 High 0.98 ObservedEfficiency ObservedNovelty 0 to 0.25 1 0 to 0.25 0 0.25 to 0.5 0 0.25 to 0.5 0 0.5 to 0.75 0 0.5 to 0.75 0 0.75 to 1 0 0.75 to 1 1 0.20 ± 0.07 0.78 ± 0.07 Dig a tunnel under the river: e = 0.20; n = 0.78

46. Bayes Model—Case 2 CreativeProblemSolving Low 0.18 High 0.82 ProblemSolving Creativity Low 0.12 Low 0.03 High 0.88 High 0.97 Efficiency Novelty Low 0.02 Low 0.01 High 0.98 High 0.99 ObservedEfficiency ObservedNovelty 0 to 0.25 0 0 to 0.25 0 0.25 to 0.5 0 0.25 to 0.5 0 0.5 to 0.75 0 0.5 to 0.75 0 0.75 to 1 1 0.75 to 1 1 0.76 ± 0.07 0.80 ± 0.07 Freeze water and slide across: e = 0.76; n = 0.80

47. Supporting “Grow” • Bayes nets can be used in various ways to improve learning and performance. • They continuously observe & integrate evidence of performance for accurate, real-time estimates of competencies. • Info on competencies may be used by (a) trainers (to adjust instruction), (b) the system (to select new gaming experiences), and/or (c) trainees (to reflect on how they’re doing).

48. Supporting “Grow” (cont.) • Re: learning, current estimates of competencies can be integrated into the game and displayed as progress indicators. • This elevates valued competencies to the same level as health & weapons!

49. Example Summary • To address military training challenges and harness the potential of immersive games, I presented an ECD-inspired idea which involved the following: • Specify valuable competencies to be acquired from the game • Define evidence models that link game behaviors to competencies • Update the learner model at certain intervals • Next step—adapt content in the game to fit the current needs of player/learner.

50. Future Visions