Motivated Reinforcement Learning for NPC in Game Worlds

1. Motivated Reinforcement Learning for Non-Player Characters in Persistent Computer Game Worlds Advisor : Dr. Hsu Presenter : Chia-Hao Yang Author : Kathryn Merrick, Mary Lou Maher SIGCHI  06

2. Outline • Motivation • Objective • Introduction • Method • Experiments • Discussion • Conclusions • Habituation SOM • Q-learning

3. Motivation • Many NPC possess a fixed set of pre-programmed behaviors and lack the ability to adapt and evolve in time with their surroundings.

4. Objective • To create NPC that can both evolve and adapt with their environmental.

5. Introduction • Current technologies for NPCs • Reflexive agents • Only recognized states will produce a response • State machines & rule-based approaches • EX : Baldur Gate & Dungeon Siege • Learning agents • It can modify their internal structure to respect to some task. • Black and White • Reinforcement learning agents • The agent records the reward signal. • Then chooses an action which attempts to maximize the long-run sum of the values of the reward signal. • Tao Feng

6. S(t-1) – S(t-2) S(t) – S(t-1) Method • Motivated reinforcement learning agents • It use a motivation function to directs learning. • Skill development is dependent on the agent’s environment & these skills are developed progressively over time. Q-learning

7. Experiments • In order to experiment with MRL agent, we implemented a village scenario in Second Life. • Support character • Trades people • Location, object, inventory sensor • Move to object, pick up object, use object effector • Ex : the pick, when used on the mine, will produce iron which can converted to weapons when used near the forge

8. Experiments • Partner character • Vendor character • Location, object sensor • Move to object effector • Ex : In Ultima Online players can set up vendor characters to sell the goods they have crafted.

9. Conclusions • This paper has presented MRL agents as a means of creating non-player characters which can both evolve and adapt. • MRL agents explore their environment and learn new behaviors in response to interesting experiences, allowing them to display progressively evolving behavioral patterns.

10. Habituation SOM • An HSOM consists of a standard Self-Organizing Map with an additional habituating neuron connected to every clustering neuron of the SOM.

11. Q-Learning • It’s a part of reinforcement learning algorithm which has been widely used for many applications such as robotics, multi agent system, game, and etc. • It allows an agent to learn through training without teacher in unknown environment. • Modeling the Environment • putting similar matrix name Q in the brain of our agent reference

12. Q-Learning • algorithm • example …… reference