temporal difference learning with expectimax search for the threes bot n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Temporal Difference Learning with Expectimax Search for the Threes-bot PowerPoint Presentation
Download Presentation
Temporal Difference Learning with Expectimax Search for the Threes-bot

Loading in 2 Seconds...

  share
play fullscreen
1 / 16
maggie-thomas

Temporal Difference Learning with Expectimax Search for the Threes-bot - PowerPoint PPT Presentation

223 Views
Download Presentation
Temporal Difference Learning with Expectimax Search for the Threes-bot
An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Temporal Difference Learning with Expectimax Search for the Threes-bot National Chiao Tung University Department of Computer Science Computer Games and Intelligence (CGI) Lab Advisor: I-Chen Wu Author: Han Chiang

  2. Reference “Threes!”, http://asherv.com/threes/ “Taiwan 2048 Bot “, http://2048-botcontest.twbbs.org/ CGI-2048 http://2048.aigames.nctu.edu.tw/replay.php “Threesus!”, http://blog.waltdestler.com/2014/04/threesus.html Albert L. Zobrist. A New Hashing Method With Application For Game Playing. Technical Report #88, April 1970/ Bruce W. Ballard, “The ,-Minimax Search Procedure for Trees Containing Chance Nodes “ MarcinSzubert, WojciehJaskowaski, Institute of Computing Science, Poznan University of Technology, Poznan, Poland, “Temporal Difference Learning of N-tuple Networks for the Game 2048”, CIG2014 J. Baxter, A. Tridgell, and L. Weaver, “Learning to Play Chess Using Temporal Differences,” Machine Learning, vol. 40, no. 3, pp. 243–263, 2000. Temporal-Difference Learning, Section II-6, “An Introduction to Reinforcement Learning”

  3. Outline • Background knowledge • Expectimax • TD-Learning • Formula • Tuple network • Our algorithm • Features • Apply to expectimax • Result

  4. Outline • Background knowledge • Expectimax • TD-Learning • Formula • Tuple network • Our algorithm • Features • Apply to expectimax • Result

  5. Expectimax

  6. Outline • Background knowledge • Expectimax • TD-Learning • Formula • Tuple network • Our algorithm • Features • Apply to expectimax • Result

  7. TD-Learning in Game Threes • TD-learning can be successfully applied to game 2048.[Szubert & Jakowski 2014] • We designed our Threes program, • Different definitions to game board. (Threes! vs. 2048) • Use our own features. • Use expectimax search.

  8. TD-Learning in Game Threes • Use TD(0) learning method: • : the expected cumulative reward for a board, implemented using N-tuple networks • : the learning rate • other variables are defined at the next page • Minimize the difference between the current prediction of cumulative future reward and one-step-ahead prediction.

  9. TD-Learning Add a new random tile Move right s s' s'' Learning the expected cumulative result for the board

  10. Tuple Networks Implement the function mentioned before. is the function shown below:

  11. Outline • Background knowledge • Expectimax • TD-Learning • Formula • Tuple network • Our algorithm • Features • Apply to expectimax • Result

  12. Feature • Feature: • Max tile value and position • Possible new tile • 3 different parts of board with rotate and symmetric

  13. TD-Learning with Expectimax At the leaf nodes of the expectimax search tree, we return the heuristic of the board. We replace the heuristic with the value, V(s), we retrieve in TD learning

  14. Result (in our environment) Highest Score: 255531 Average Score: 107833 Max Tile: 3072 192 Rate: 100% 384 Rate: 100% 768 Rate: 97% 1536 Rate: 86% 3072 Rate: 29% 6144 Rate: 0% Move Count: 81097 Time: 199.35

  15. Result (in contest server) Max Score : 246297​ Avg. Score : 110931​ 192 rate : 100%​ 384 rate : 100%​ 768 rate :99%​ 1536 rate : 86%​ 3072 rate : 31% 

  16. Thank you