CS 188: Artificial Intelligence Fall 2008

1. CS 188: Artificial IntelligenceFall 2008 Lecture 7: Expectimax Search 9/18/2008 Dan Klein – UC Berkeley Many slides over the course adapted from either Stuart Russell or Andrew Moore

2. Recap: Resource Limits • Cannot search to leaves • Depth-limited search • Instead, search a limited depth of tree • Replace terminal utilities with an eval function for non-terminal positions • Guarantee of optimal play is gone • Replanning agents: • Search to choose next action • Replan each new turn in response to new state max 4 min min -2 4 -1 -2 4 9 ? ? ? ?

3. Evaluation for Pacman [DEMO: thrashing, smart ghosts]

4. Iterative Deepening b Iterative deepening uses DFS as a subroutine: • Do a DFS which only searches for paths of length 1 or less. (DFS gives up on any path of length 2) • If “1” failed, do a DFS which only searches paths of length 2 or less. • If “2” failed, do a DFS which only searches paths of length 3 or less. ….and so on. This works for single-agent search as well! Why do we want to do this for multiplayer games? …

5. - Pruning Example

6. - Pruning • General configuration •  is the best value that MAX can get at any choice point along the current path • If n becomes worse than , MAX will avoid it, so can stop considering n’s other children • Define  similarly for MIN Player Opponent  Player Opponent n

7.  v - Pruning Pseudocode

8. - Pruning Properties • Pruning has no effect on final result • Good move ordering improves effectiveness of pruning • With “perfect ordering”: • Time complexity drops to O(bm/2) • Doubles solvable depth • Full search of, e.g. chess, is still hopeless! • A simple example of metareasoning, here reasoning about which computations are relevant

9. Expectimax Search Trees • What if we don’t know what the result of an action will be? E.g., • In solitaire, next card is unknown • In minesweeper, mine locations • In pacman, the ghosts act randomly • Can do expectimax search • Chance nodes, like min nodes, except the outcome is uncertain • Calculate expected utilities • Max nodes as in minimax search • Chance nodes take average (expectation) of value of children • Later, we’ll learn how to formalize the underlying problem as a Markov Decision Process max chance 10 4 5 7 [DEMO: minVsExp]

10. Maximum Expected Utility • Why should we average utilities? Why not minimax? • Principle of maximum expected utility: an agent should chose the action which maximizes its expected utility, given its knowledge • General principle for decision making • Often taken as the definition of rationality • We’ll see this idea over and over in this course! • Let’s decompress this definition…

11. Reminder: Probabilities • A random variable represents an event whose outcome is unknown • A probability distribution is an assignment of weights to outcomes • Example: traffic on freeway? • Random variable: T = whether there’s traffic • Outcomes: T in {none, light, heavy} • Distribution: P(T=none) = 0.25, P(T=light) = 0.55, P(T=heavy) = 0.20 • Some laws of probability (more later): • Probabilities are always non-negative • Probabilities over all possible outcomes sum to one • As we get more evidence, probabilities may change: • P(T=heavy) = 0.20, P(T=heavy | Hour=8am) = 0.60 • We’ll talk about methods for reasoning and updating probabilities later

12. What are Probabilities? • Objectivist / frequentist answer: • Averages over repeated experiments • E.g. empirically estimating P(rain) from historical observation • Assertion about how future experiments will go (in the limit) • New evidence changes the reference class • Makes one think of inherently random events, like rolling dice • Subjectivist / Bayesian answer: • Degrees of belief about unobserved variables • E.g. an agent’s belief that it’s raining, given the temperature • E.g. pacman’s belief that the ghost will turn left, given the state • Often learn probabilities from past experiences (more later) • New evidence updates beliefs (more later)

13. Uncertainty Everywhere • Not just for games of chance! • I’m snuffling: am I sick? • Email contains “FREE!”: is it spam? • Tooth hurts: have cavity? • 60 min enough to get to the airport? • Robot rotated wheel three times, how far did it advance? • Safe to cross street? (Look both ways!) • Why can a random variable have uncertainty? • Inherently random process (dice, etc) • Insufficient or weak evidence • Ignorance of underlying processes • Unmodeled variables • The world’s just noisy! • Compare to fuzzy logic, which has degrees of truth, or rather than just degrees of belief

14. Expectations • Real valued functions of random variables: • Expectation of a function of a random variable • Example: Expected value of a fair die roll

15. Utilities • Utilities are functions from outcomes (states of the world) to real numbers that describe an agent’s preferences • Where do utilities come from? • In a game, may be simple (+1/-1) • Utilities summarize the agent’s goals • Theorem: any set of preferences between outcomes can be summarized as a utility function (provided the preferences meet certain conditions) • In general, we hard-wire utilities and let actions emerge (why don’t we let agents decide their own utilities?) • More on utilities soon…

16. Expectimax Search • In expectimax search, we have a probabilistic model of how the opponent (or environment) will behave in any state • Model could be a simple uniform distribution (roll a die) • Model could be sophisticated and require a great deal of computation • We have a node for every outcome out of our control: opponent or environment • The model might say that adversarial actions are likely! • For now, assume for any state we magically have a distribution to assign probabilities to opponent actions / environment outcomes Having a probabilistic belief about an agent’s action does not mean that agent is flipping any coins!

17. 8 4 5 6 Expectimax Pseudocode def value(s) if s is a max node return maxValue(s) if s is an exp node return expValue(s) if s is a terminal node return evaluation(s) def maxValue(s) values = [value(s’) for s’ in successors(s)] return max(values) def expValue(s) values = [value(s’) for s’ in successors(s)] weights = [probability(s, s’) for s’ in successors(s)] return expectation(values, weights)

18. Expectimax for Pacman • Notice that we’ve gotten away from thinking that the ghosts are trying to minimize pacman’s score • Instead, they are now a part of the environment • Pacman has a belief (distribution) over how they will act • Quiz: Can we see minimax as a special case of expectimax? • Quiz: what would pacman’s computation look like if we assumed that the ghosts were doing 1-ply minimax and taking the result 80% of the time, otherwise moving randomly? • If you take this further, you end up calculating belief distributions over your opponents’ belief distributions over your belief distributions, etc… • Can get unmanageable very quickly!

19. Expectimax Pruning?

20. Expectimax Evaluation • For minimax search, evaluation function insensitive to monotonic transformations • We just want better states to have higher evaluations (get the ordering right) • For expectimax, we need the magnitudes to be meaningful as well • E.g. must know whether a 50% / 50% lottery between A and B is better than 100% chance of C • 100 or -10 vs 0 is different than 10 or -100 vs 0

21. Mixed Layer Types • E.g. Backgammon • Expectiminimax • Environment is an extra player that moves after each agent • Chance nodes take expectations, otherwise like minimax

22. Stochastic Two-Player • Dice rolls increase b: 21 possible rolls with 2 dice • Backgammon  20 legal moves • Depth 4 = 20 x (21 x 20)3 1.2 x 109 • As depth increases, probability of reaching a given node shrinks • So value of lookahead is diminished • So limiting depth is less damaging • But pruning is less possible… • TDGammon uses depth-2 search + very good eval function + reinforcement learning: world-champion level play

23. Non-Zero-Sum Games • Similar to minimax: • Utilities are now tuples • Each player maximizes their own entry at each node • Propagate (or back up) nodes from children • Can give rise to cooperation and competition dynamically… 1,2,6 4,3,2 6,1,2 7,4,1 5,1,1 1,5,2 7,7,1 5,4,5

24. Preferences • An agent chooses among: • Prizes: A, B, etc. • Lotteries: situations with uncertain prizes • Notation:

25. Rational Preferences • We want some constraints on preferences before we call them rational • For example: an agent with intransitive preferences can be induced to give away all its money • If B > C, then an agent with C would pay (say) 1 cent to get B • If A > B, then an agent with B would pay (say) 1 cent to get A • If C > A, then an agent with A would pay (say) 1 cent to get C

26. Rational Preferences • Preferences of a rational agent must obey constraints. • The axioms of rationality: • Theorem: Rational preferences imply behavior describable as maximization of expected utility

27. MEU Principle • Theorem: • [Ramsey, 1931; von Neumann & Morgenstern, 1944] • Given any preferences satisfying these constraints, there exists a real-valued function U such that: • Maximum expected likelihood (MEU) principle: • Choose the action that maximizes expected utility • Note: an agent can be entirely rational (consistent with MEU) without ever representing or manipulating utilities and probabilities • E.g., a lookup table for perfect tictactoe, reflex vacuum cleaner

28. Human Utilities • Utilities map states to real numbers. Which numbers? • Standard approach to assessment of human utilities: • Compare a state A to a standard lottery Lp between • ``best possible prize'' u+ with probability p • ``worst possible catastrophe'' u- with probability 1-p • Adjust lottery probability p until A ~ Lp • Resulting p is a utility in [0,1]

29. Utility Scales • Normalized utilities: u+ = 1.0, u- = 0.0 • Micromorts: one-millionth chance of death, useful for paying to reduce product risks, etc. • QALYs: quality-adjusted life years, useful for medical decisions involving substantial risk • Note: behavior is invariant under positive linear transformation • With deterministic prizes only (no lottery choices), only ordinal utility can be determined, i.e., total order on prizes

30. Example: Insurance • Consider the lottery [0.5,$1000; 0.5,$0] • What is its expected monetary value? ($500) • What is its certainty equivalent? • Monetary value acceptable in lieu of lottery • $400 for most people • Difference of $100 is the insurance premium • There’s an insurance industry because people will pay to reduce their risk • If everyone were risk-prone, no insurance needed!

31. Money • Money does not behave as a utility function • Given a lottery L: • Define expected monetary value EMV(L) • Usually U(L) < U(EMV(L)) • I.e., people are risk-averse • Utility curve: for what probability p am I indifferent between: • A prize x • A lottery [p,$M; (1-p),$0] for large M? • Typical empirical data, extrapolated with risk-prone behavior:

32. Example: Human Rationality? • Famous example of Allais (1953) • A: [0.8,$4k; 0.2,$0] • B: [1.0,$3k; 0.0,$0] • C: [0.2,$4k; 0.8,$0] • D: [0.25,$3k; 0.75,$0] • Most people prefer B > A, C > D • But if U($0) = 0, then • B > A  U($3k) > 0.8 U($4k) • C > D  0.8 U($4k) > U($3k)