Real-Time Strategy Artificial Intelligence Research

1. Real-Time Strategy Artificial Intelligence Research Current Situation and Future Plans Abdelrahman Al-Ogail & Omar Enayet October - 2010

2. Agenda • What do we do. • What did we achieve till now. • The Engine. • The Paper. • The Future. • Short Term Goals. • Long Term Goals.

3. What do we do

4. Project Research Area AI Planning Knowledge Sharing Plan then re-plan according to new givens. Let everyone know instantly what you knew through experience. Make the machine learn. AI Learning

5. Problem Definition Experience Loss Static Scripts Computer AI relies on static scripting techniques. The Absence of sharing experience costs a lot

6. Objectives Mobile Experience Making the Computer Opponent adapt to changes like human do. Import/Export your experience ! Adaptive A.I.

7. Project Domain Real-Time Strategy Games. Severe Time Constraints – Real-Time AI – Many Objects – Imperfect Information – Micro-Actions RTS Games

8. Motivations War Simulation Experimental Relevance Robotics For the corporation of robots. They constitute well-defined environments to conduct experiments. For interest for military which uses battle simulations in training programs.

9. Artificial Intelligence used Reinforcement Learning Case-based Reasoning

10. What did we achieve till now

11. What did we achieve till now • Developed the engine with its basic stuff. • Graduation Project, Excellent Degree. • Participated with a booth in ICT’10 with the GP. • Project was funded by ITIDA. • Another graduation project & more than a master thesis are being done to extend our work in Ain-Shams University. • Published a paper in ISDA’10 • Had Contact with researchers on this field all over the world.

12. What did we achieve till now (Cont’d) • Maintaining the project bloghttp://rtsairesearch.wordpress.com/ • Maintaining the project repository:Our research stuff repository: http://rtsairesearch.googlecode.com/ • I-Strategizer (Engine’s code): http://istrategizer.googlecode.com/ • Maintaining our own blogs:OmarsBrain.wordpress.com (Omar Enayet) AbdelrahmanOgail.wordpress.com (Abdelrahman Al-Ogail)

13. The Future

14. The Future – Long-term Goals • Adding new theory in the area of “Simulation of Human Behavior”. • Developing a commercial AI Engine for RTS Games specifically and games in general. We already started and we have quite experience in game development. • Participate in related contests around the world for AI Engines in RTS Games (As Robocup, AAAI Starcraft Competition, ORTS Competition). • Initializing a major research group in Egypt in this field and become pioneers in it world wide.

15. The Future – Short-term Goals • Enhancing current engine which will efficiently be able to plan and learn when playing against static AI • Use it as a test-bed to publish a number of papers: • 1- Introducing the whole Agent model and theory in AI related conference.2- Introducing the whole AI Game Engine from a game industry point of view in a game-industry conference.3- More Details & Testing concerning the hybridization of Online Case based Planning and Reinforcement Learning ( the topic of our last paper)4- Knowledge representation for plans and experience in RTS games.5- Enhancing agent’s situation assessment algorithm. • Publishing a paper concerning : Comparing Case-Based Reasoning to Reinforcement Learning.

16. The Future – Long-term Papers’ Topics • Include different planning algorithms/systems and let agent use them and make an intensive comparison between these panning systems. • Include different learning algorithms/systems and let agent use them and make an intensive comparison between these learning systems. • Multi-Agent AI : machine collaboration with other machines, or machine collaboration with human players. • Knowledge (Gaming Experience) Sharing. • Opponent Modeling.

17. The Engine

18. The Engine • Named “I-Strategizer”. • Based on an open-source RTS game engine called “Stratagus”. • Currently tailored to serve an open source game Wargus .(clone of an old popular game called Warcraft 2 ) • Still needs a lot of research and development for simulating human behavior and making it generic for strategy games. • Coded in C++ and LUA scripts.

19. Engine Architecture

20. The Paper

21. The Paper -Abstract • Abstract—Research in learning and planning in real-time strategy (RTS) games is very interesting in several industries such as military industry, robotics, and most importantly game industry. A recent published work on online case-based planning in RTS Games does not include the capability of online learning from experience, so the knowledge certainty remains constant, which leads to inefficient decisions. In this paper, an intelligent agent model based on both online case- based planning (OLCBP) and reinforcement learning (RL) techniques is proposed. In addition, the proposed model has been evaluated using empirical simulation on Wargus (an open-source clone of the well known RTS game Warcraft 2). This evaluation shows that the proposed model increases the certainty of the case base by learning from experience, and hence the process of decision making for selecting more efficient, effective and successful plans. • Keywords- Case-based Reasoning; Reinforcement Learning; Online Case-based Planning; Real-Time Strategy Games; Sarsa (λ) Learning; Eligibility Traces; Intelligent Agent.

22. The Paper -Introduction • The Problem : Learning from human then no learning from experience. Our Solution : Learning from experience is maintained through Reinforcement Learning (RL) . • Old Approach : Online Case-Based Planning (OLCBP) Our Approach : Hybridizing OLCBP with RL .

23. The Paper -Background • OLCBP ? • Other approaches done to hybrid CBR with RL.

24. The Paper –Intelligent OLCBP Model

25. The Paper –The Hybridization • We used an RL Temporal-difference learning approach: SARSA(λ) Learning • According to certain rules, SARSA(λ) Learning and interactions from the environment, the certainty value of cases in the case-base change -> Thus Learning from experience occurs.

26. The Paper –The Test-Case

27. The Paper –The Results

28. The Paper –The Results (Cont’d) • Agent has learnt that building a smaller heavy army in that specific situation (the existence of a towers defense) is more preferable than building a larger light army. Similarly, the agent can evaluate the entire case base and learn the right choices.

29. The Paper –Conclusion • Online case-based planning was hybridized with reinforcement learning in order to introduce an intelligent agent capable of planning and learning online using temporal difference with eligibility traces: Sarsa (λ) algorithm. The empirical evaluation has shown that the proposed model –unlike Darmok System - increases the certainty of the case base by learning from experience, and hence the process of decision making for selecting more efficient, effective and successful plans.

30. The Paper –Future Work • Implementing a prototype based on the proposed model. • Developing a strategy/case base visualization tool capable of visualizing agent’s preferred playing strategy according to its learning history. This will help in tracking the learning curve of the agent. • Finally, designing and developing a multi-agent system where agents are able to share their experiences together.

31. Thank you !