1 / 39

Pondering Probabilistic Play Policies for Pig

Pondering Probabilistic Play Policies for Pig. Todd W. Neller Gettysburg College. Sow What’s This All About?. The Dice Game “Pig” Odds and Ends Playing to Win “Piglet” Value Iteration Machine Learning. Pig: The Game. Object: First to score 100 points On your turn, roll until:

mercury
Download Presentation

Pondering Probabilistic Play Policies for Pig

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Pondering Probabilistic Play Policies for Pig Todd W. Neller Gettysburg College

  2. Sow What’s This All About? • The Dice Game “Pig” • Odds and Ends • Playing to Win • “Piglet” • Value Iteration • Machine Learning

  3. Pig: The Game • Object: First to score 100 points • On your turn, roll until: • You roll 1, and score NOTHING. • You hold, and KEEP the sum. • Simple game  simple strategy? • Let’s play…

  4. Playing to Score • Simple odds argument • Roll until you risk more than you stand to gain. • “Hold at 20” • 1/6 of time: -20  -20/6 • 5/6 of time: +4 (avg. of 2,3,4,5,6)  +20/6

  5. Hold at 20? • Is there a situation in which you wouldn’t want to hold at 20? • Your score: 99; you roll 2 • Case scenario • you: 79 opponent:99 • Your turn total stands at 20

  6. What’s Wrong With Playing to Score? • It’s mathematically optimal! • But what are we optimizing? • Playing to score  Playing to win • Optimizing score per turn  Optimizing probability of a win

  7. Piglet • Simpler version of Pig with a coin • Object: First to score 10 points • On your turn, flip until: • You flip tails, and score NOTHING. • You hold, and KEEP the # of heads. • Even simpler: play to 2 points

  8. Essential Information • What is the information I need to make a fully informed decision? • My score • The opponent’s score • My “turn score”

  9. A Little Notation • Pi,j,k – probability of a win ifi = my scorej = the opponent’s scorek = my turn score • Hold: Pi,j,k = 1 - Pj,i+k,0 • Flip: Pi,j,k = ½(1 - Pj,i,0) + ½ Pi,j,k+1

  10. Assume Rationality • To make a smart player, assume a smart opponent. • (To make a smarter player, know your opponent.) • Pi,j,k = max(1 - Pj,i+k,0, ½(1 - Pj,i,0 + Pi,j,k+1)) • Probability of win based on best decisions in any state

  11. The Whole Story P0,0,0 = max(1 – P0,0,0, ½(1 – P0,0,0 + P0,0,1)) P0,0,1 = max(1 – P0,1,0, ½(1 – P0,0,0 + P0,0,2)) P0,1,0 = max(1 – P1,0,0, ½(1 – P1,0,0 + P0,1,1)) P0,1,1 = max(1 – P1,1,0, ½(1 – P1,0,0 + P0,1,2)) P1,0,0 = max(1 – P0,1,0, ½(1 – P0,1,0 + P1,0,1)) P1,1,0 = max(1 – P1,1,0, ½(1 – P1,1,0 + P1,1,1))

  12. The Whole Story P0,0,0 = max(1 – P0,0,0, ½(1 – P0,0,0 + P0,0,1)) P0,0,1 = max(1 – P0,1,0, ½(1 – P0,0,0 + P0,0,2)) P0,1,0 = max(1 – P1,0,0, ½(1 – P1,0,0 + P0,1,1)) P0,1,1 = max(1 – P1,1,0, ½(1 – P1,0,0 + P0,1,2)) P1,0,0 = max(1 – P0,1,0, ½(1 – P0,1,0 + P1,0,1)) P1,1,0 = max(1 – P1,1,0, ½(1 – P1,1,0 + P1,1,1)) These are winning states!

  13. The Whole Story P0,0,0 = max(1 – P0,0,0, ½(1 – P0,0,0 + P0,0,1)) P0,0,1 = max(1 – P0,1,0, ½(1 – P0,0,0 + 1)) P0,1,0 = max(1 – P1,0,0, ½(1 – P1,0,0 + P0,1,1)) P0,1,1 = max(1 – P1,1,0, ½(1 – P1,0,0 + 1)) P1,0,0 = max(1 – P0,1,0, ½(1 – P0,1,0 + 1)) P1,1,0 = max(1 – P1,1,0, ½(1 – P1,1,0 + 1)) Simplified…

  14. The Whole Story P0,0,0 = max(1 – P0,0,0, ½(1 – P0,0,0 + P0,0,1)) P0,0,1 = max(1 – P0,1,0, ½(2 – P0,0,0)) P0,1,0 = max(1 – P1,0,0, ½(1 – P1,0,0 + P0,1,1)) P0,1,1 = max(1 – P1,1,0, ½(2 – P1,0,0)) P1,0,0 = max(1 – P0,1,0, ½(2 – P0,1,0)) P1,1,0 = max(1 – P1,1,0, ½(2 – P1,1,0)) And simplified more into a hamsome set of equations…

  15. How to Solve It? P0,0,0 = max(1 – P0,0,0, ½(1 – P0,0,0 + P0,0,1)) P0,0,1 = max(1 – P0,1,0, ½(2 – P0,0,0)) P0,1,0 = max(1 – P1,0,0, ½(1 – P1,0,0 + P0,1,1)) P0,1,1 = max(1 – P1,1,0, ½(2 – P1,0,0)) P1,0,0 = max(1 – P0,1,0, ½(2 – P0,1,0)) P1,1,0 = max(1 – P1,1,0, ½(2 – P1,1,0)) P0,1,0 depends on P0,1,1 depends on P1,0,0 depends on P0,1,0 depends on …

  16. P0,1,1 P0,0,1 P0,1,0 P1,1,0 P0,0,0 P1,0,0 A System of Pigquations Dependencies between non-winning states

  17. How Bad Is It? • The intersection of a set of bent hyperplanes in a hypercube • In the general case, no known method (read: PhD research) • Is there a method that works (without being guaranteed to work in general)? • Yes! Value Iteration!

  18. Value Iteration • Start out with some values (0’s, 1’s, random #’s) • Do the following until the values converge (stop changing): • Plug the values into the RHS’s • Recompute the LHS values • That’s easy. Let’s do it!

  19. Value Iteration P0,0,0 = max(1 – P0,0,0, ½(1 – P0,0,0 + P0,0,1)) P0,0,1 = max(1 – P0,1,0, ½(2 – P0,0,0)) P0,1,0 = max(1 – P1,0,0, ½(1 – P1,0,0 + P0,1,1)) P0,1,1 = max(1 – P1,1,0, ½(2 – P1,0,0)) P1,0,0 = max(1 – P0,1,0, ½(2 – P0,1,0)) P1,1,0 = max(1 – P1,1,0, ½(2 – P1,1,0)) • Assume Pi,j,k is 0 unless it’s a win • Repeat: Compute RHS’s, assign to LHS’s

  20. But That’s GRUNT Work! • So have a computer do it, slacker! • Not difficult – end of CS1 level • Fast! Don’t blink – you’ll miss it • Optimal play: • Compute the probabilities • Determine flip/hold from RHS max’s • (For our equations, always FLIP)

  21. Piglet Solved • Game to 10 • Play to Score: “Hold at 1” • Play to Win: Opponent You

  22. Pig Probabilities • Just like Piglet, but more possible outcomes • Pi,j,k = max(1 - Pj,i+k,0, 1/6(1 - Pj,i,0+ Pi,j,k+2 + Pi,j,k+3+ Pi,j,k+4 + Pi,j,k+5 + Pi,j,k+6))

  23. Solving Pig • 505,000 such equations • Same simple solution method (value iteration) • Speedup: Solve groups of interdependent probabilities • Watch and see!

  24. Pig Sow-lution

  25. Pig Sow-lution

  26. Reachable States Player 2 Score (j) = 30

  27. Reachable States

  28. Sow-lution forReachable States

  29. Probability Contours

  30. Summary • Playing to score is not playing to win. • A simple game is not always simple to play. • The computer is an exciting power tool for the mind!

  31. When Value IterationIsn’t Enough • Value Iteration assumes a model of the problem • Probabilities of state transitions • Expected rewards for transitions • Loaded die? • Optimal play vs. suboptimal player? • Game rules unknown?

  32. No Model? Then Learn! • Can’t write equations  can’t solve • Must learn from experience! • Reinforcement Learning • Learn optimal sequences of actions • From experience • Given positive/negative feedback

  33. Clever Mabel the Cat

  34. Clever Mabel the Cat • Mabel claws new La-Z-Boy  BAD! • Cats hate water  spray bottle = negative reinforcement • Mabel claws La-Z-Boy  Todd gets up  Todd sprays Mabel  Mabel gets negative feedback • Mabel learns…

  35. Clever Mabel the Cat • Mabel learns to run when Todd gets up. • Mabel first learns local causality: • Todd gets up  Todd sprays Mabel • Mabel eventually sees no correlation, learns indirect cause • Mabel happily claws carpet. The End.

  36. Backgammon • Tesauro’s Neurogammon • Reinforcement Learning + Neural Network (memory for learning) • Learned backgammon through self-play • Got better than all but a handful of people in the world! • Downside: Took 1.5 million games to learn

  37. Greased Pig • My continuous variant of Pig • Object: First to score 100 points • On your turn, generate a random number from 0.5 to 6.5 until: • Your rounded number is 1, and you score NOTHING. • You hold, and KEEP the sum. • How does this change things?

  38. Greased Pig Challenges • Infinite possible game states • Infinite possible games • Limited experience • Limited memory • Learning and approximation challenge

  39. Summary • Solving equations can only take you so far (but much farther than we can fathom). • Machine learning is an exciting area of research that can take us farther. • The power of computing will increasingly aid in our learning in the future.

More Related