Learning Strategic Features for General Games

1. Learning Strategic Features for General Games Dennis Soemers June 5, 2019 @DennisSoemers

2. Digital Ludeme Project • Computational study of traditional games throughout history • Model games in general game system (Ludii) • Generate plausible reconstructions of rulesets • Data-driven • AI self-play to ``play-test’’ generated rulesets

3. AI Requirements • Play approx. 1000 strategy games • and many more variants… • Need General Game Players! • Ideally strong, human-level AI • Do not need super-human AI • Automated strategy learning • Learning from self-play • Interpretable strategies

4. General Game Playing (GGP) Repeated X times • Monte Carlo tree search (MCTS) • Prevailing GGP approach • Can be improved with learned policies Selection Play-out Expansion Backpropagation

5. Policy Learning from Self-play • State of the art: Deep Learning • AlphaGo, AlphaGo Zero, AlphaZero…

6. Downsides of Deep Learning • Start learning from scratch per game • Difficult for 1000 games • Requires some domain knowledge • One policy output node per action • How many actions possible in <unknown game>? • Difficult for General Game Playing • Expensive

7. General Game Features • Binary features for state-action pairs • Local patterns • Use underlying graph-representation • Widely applicable • Single format, many games

8. General Game Features

9. Which features to use? • Learn features andweightssimultaneously • Start withatomic features • Simple patternswith a single “test” • Combine pairs of features • Maximise correlationwithpolicy’sobjective • Minimise correlationwithconstituents D. J. N. J. Soemers, É. Piette, C. Browne (2019). “Biasing MCTS with Features for General Games”. In 2019 IEEE Congress on EvolutionaryComputation.

10. Self-play Policy Learning Objectives • Minimise cross-entropybetweenlearned policy and MCTS visitcounts • AlphaGo Zero, AlphaZero, etc. • MCTS is exploratoryby design • Trained policy alsoexploratory!

11. Self-play Policy Learning Objectives • Do we want ourtrained policy tobeexploratory? • Bias MCTS Selection Yes • Bias MCTS Play-out  Maybe • Interpretlearnedstrategies  No • Usestrategyfor No “game distancefunction” D. J. N. J. Soemers, É. Piette, M. Stephenson, C. Browne (2019). “Learning PoliciesfromSelf-Play with Policy Gradientsand MCTS Value Estimates”. In 2019 IEEE Conference on Games.

12. Conclusion • Promisingresults on 10 games so far • Alltwo-player, full information, deterministic • Work in progress: • Scaling up to more games • Multi-player, hidden info, nondeterministic, … • Speeding up features • Interpretinglearnedstrategies

13. Thank you!

15. AI (UCT) without features in Gomoku

16. AI (UCT) without features in Gomoku

17. AI (Biased MCTS) with features in Gomoku

18. AI (Biased MCTS) with features in Gomoku

19. Learning Curves (CEC 2019)

20. Learning Curves - Pruned (CEC 2019)

21. Learning Curves (COG 2019)

22. Policy Entropy (COG 2019)