- 55 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about 'Worlds with many intelligent agents' - eldora

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

### Worlds with many intelligent agents

An important consideration in AI, as well as games, distributed systems and networking, economics, sociology, political science, international relations, and other disciplines

Multi-agent Systems, or “Games”

Worlds with multiple agents, somtimes called “games”, are VASTLY more complicated than worlds with single agents.

They’re also more interesting and fun!

There are many different aspects to learn about:

- Representations (we will consider two, but there are MANY)
- Evaluation metrics (I’ll show you a few standard ones, but again there are MANY)
- Inference and planning (we will talk about a few techniques)
- Learning (We won’t really cover this)
- Communication (We won’t directly cover this)
- Computational Complexity (I’ll mention this in passing)
- Applications (We’ll look at several examples, but there are TONS)

Example Game (Prisoner’s Dilemma)

Student A and Student B have been naughty, cheating on an exam. Their teacher finds some evidence, and sends them to the dean’s office, where the dean grills them individually to find out more.

Outcome:

A rats out B

A says nothing

B rats out A

B says nothing

Example Game (Prisoner’s Dilemma)

Definitions: the set of choices available to each agent is called their action set, denoted Actions(A) or Actions(B).

Each square in the grid is called an outcome. Mathematically, let a be an action in Actions(A), and b in Actions(B). O(a, b) is the outcome square. OA(a, b) is the reward for A, and OB(a, b) is the reward for B.

This grid is called a normal formrepresentation of the game.

Outcome:

A rats out B

A says nothing

B rats out A

B says nothing

Quiz: Best outcome?

Which outcome is the best, and why?

Outcome:

A rats out B

A says nothing

B rats out A

B says nothing

Answer: Best outcome?

The answer depends. Let’s look at a few definitions of “best”.

Outcome:

A rats out B

A says nothing

B rats out A

B says nothing

Definition: Dominant Strategy

Definition: A strategy (or pure strategy) for player A (ditto for B) is a selection of one of the actions from Actions(A).

Definition: A’s strategy a ∊ Actions(A) is dominant if

∀b ∊ Actions(B), ∀a’≠a∊ Actions(A) . OA(a, b) > OA(a’, b)

Another way of saying this is: a is a dominant strategy for A if it is better than any other strategy a’ for A, no matter what strategy B chooses.

Quiz: Dominant Strategy

Definition: A’s strategy a ∊ Actions(A) is dominant if

∀b ∊ Actions(B), ∀a’≠a∊ Actions(A) . OA(a, b) > OA(a’, b)

Does A have a dominant strategy for the game below? What is it? What about B?

Outcome:

A rats out B

A says nothing

B rats out A

B says nothing

Answer: Dominant Strategy

Definition: A’s strategy a ∊ Actions(A) is dominant if

∀b ∊ Actions(B), ∀a’≠a∊ Actions(A) . OA(a, b) > OA(a’, b)

Does A have a dominant strategy for the game below? What is it? What about B?

Dominant strategy for A!

A rats out B

A says nothing

Dominant strategy for B!

B rats out A

B says nothing

Definition: Pareto Optimal

Definition: An outcome o is called pareto optimal if there is no other outcome o’ such that all players prefer o’ to o.

Notice: pareto optimality can lead to lots of optimal outcomes, not just one.

That’s because o’ is considered better than o only if ALL players prefer o’ to o.

Quiz: Pareto Optimal

Definition: An outcome o is called pareto optimal if there is no other outcome o’ such that all players prefer o’ to o.

Quiz: which outcome(s) is (are) pareto optimal?

Outcome:

A rats out B

A says nothing

B rats out A

B says nothing

Answer: Pareto Optimal

Definition: An outcome o is called pareto optimal if there is no other outcome o’ such that all players prefer o’ to o.

Quiz: which outcome(s) is (are) pareto optimal?

Outcome:

A rats out B

A says nothing

B rats out A

B says nothing

Definition: (Nash) Equilibrium

Definition: An outcome O(a, b) is called a Nash Equilibrium (or just equilibrium) if

- ∀a’≠a∊ Actions(A) . OA(a, b) >= OA(a’, b) and
- ∀b’≠b∊ Actions(B) . OB(a, b) >= OB(a, b’)

Another way of saying this is that an outcome is an equilibrium if each player is as happy with this outcome as any other available option, assuming the other player sticks with his equilibrium strategy.

Notice that this means an outcome is stable, in the sense that both players have no incentive to change their strategy.

Quiz: (Nash) Equilibrium

Definition: An outcome O(a, b) is called a Nash Equilibriumif

- ∀a’≠a∊ Actions(A) . OA(a, b) >= OA(a’, b) and
- ∀b’≠b∊ Actions(B) . OB(a, b) >= OB(a, b’)

Which outcome(s) is (are) a Nash equilibrium?

Outcome:

A rats out B

A says nothing

B rats out A

B says nothing

Answer: (Nash) Equilibrium

Definition: An outcome O(a, b) is called a Nash Equilibriumif

- ∀a’≠a∊ Actions(A) . OA(a, b) >= OA(a’, b) and
- ∀b’≠b∊ Actions(B) . OB(a, b) >= OB(a, b’)

Which outcome(s) is (are) a Nash equilibrium?

Outcome:

A rats out B

A says nothing

B rats out A

B says nothing

Answer: (Nash) Equilibrium

One reason that the prisoner’s dilemma is famous:

The only outcome that is a Nash equilibrium is also the only outcome that is NOT a pareto optimum.

This is not true in all games; it depends on the rewards!

For the prisoner’s dilemma, it means that the only stable solution is probably the worst possible solution for the players. (It’s sort of a depressing example.)

A rats out B

A says nothing

B rats out A

B says nothing

Quiz: The game of Chicken

(a.k.a. “Hawk-Dove”, “snow-drift”)

What are the dominant strategies, pareto optima, and Nash equlibria for the Chicken game?

Player 1

Swerve

Straight

Swerve

Player 2

Straight

Answer: The game of Chicken

(a.k.a. “Hawk-Dove”, “snow-drift”)

What are the dominant strategies, pareto optima, and Nash equlibria for the Chicken game?

Player 1

Nash Equilibrium

Pareto Optimum

No dominant strategies!

Swerve

Straight

Swerve

Player 2

Straight

Quiz: Coordination Games

What are the dominant strategies, pareto optima, and Nash equlibria for each of the games below?

Left

Right

Party

Home

Left

Party

Right

Home

Pure coordination game

Drive on which side of the road?

Party

Home

Stag

Hare

Party

Stag

Home

Hare

“Battle of the Sexes” game

“Stag Hunt” game

Dominant Strategy

Pareto Optimum

Answer: Coordination GamesWhat are the dominant strategies, pareto optima, and Nash equlibria for each of the games below?

Left

Right

Party

Home

Left

Party

Right

Home

Pure coordination game

Drive on which side of the road?

Party

Home

Stag

Hare

Party

Stag

Home

Hare

“Battle of the Sexes” game

“Stag Hunt” game

Humans don’t necessarily play Nash equlibria

The “Guess 2/3 of the Average” Game is famous because in experiments with people, they often don’t play Nash equilibrium strategies. (Other games like this are the centipede game and the prisoner’s dilemma.)

Thus Nash equilibria aren’t perfect models of human behavior.

Let’s try it. Here are the rules to “Guess 2/3 of the average”:

- Everyone guess a number between 0 and 100 (integer or real). Once everyone guesses, we’ll compute the average.
- The set of people who come closest to 2/3 of the average are the winners (payoff 1), everyone else loses (payoff 0).

Nash equilibrium for “Guess 2/3 of the Average”

If everyone guesses 100, then 2/3 of the average is 66.7, so 66.7 is the biggest possible result, so everyone has an incentive to guess 66.7 or less, so anything more than 66.7 can’t be a Nash equilibrium.

If everyone guesses 66.7, then 2/3 of the average will be around 44.4, so everyone has an incentive to guess lower, so guessing more than 44.4 can’t be a Nash equilibrium.

…

If everyone guesses 0, then we have a Nash equilibrium (they’d all guess right and win, and there’s no incentive for anyone to change their guess assuming everyone else remains at 0).

Constant-Sum Games

Player 1

Rock

Paper

Scissors

Notice: if you add up the reward for P1 and P2 in each square, you get 0.

The fact that all squares have the same sum makes this a “constant-sum” game.

This game is in fact a “zero-sum” game.

Rock

Paper

Player 2

Scissors

Constant-Sum Games

Player 1

Actually, any constant-sum game can be converted to an equivalent zero-sum game by subtracting a constant from every payoff.

They are “equivalent” in that this transformation preserves equilibria, optimal, and dominant strategies.

Rock

Paper

Scissors

Rock

Paper

Player 2

Scissors

Constant-Sum Games

Player 1

Rock

Paper

Scissors

In constant-sum games, any gain for one player is offset by a loss for the other player.

So these are hyper-competitive games.

Rock

Paper

Player 2

Scissors

Quiz: Pure Strategy Nash Equilibria

Player 1

Rock

Paper

Scissors

Can you find a Nash equilibrium for Rock-Paper-Scissors?

Rock

Paper

Player 2

Scissors

Answer: Pure Strategy Nash Equilibria

Player 1

Can you find a Nash equilibrium for Rock-Paper-Scissors?

No “pure” strategy for either player can lead to a Nash equilibrium for this game!

(Our definition of “strategy” is equivalent to “pure strategy”. We’ll talk about unpure strategies next.)

Rock

Paper

Scissors

Rock

Paper

Player 2

Scissors

Definition: Mixed Strategy

A mixed strategy for player A is a probability distribution over the set of available actions, Actions(A).

For instance, for Rock-Paper-Scissors, the distribution

P(Rock) = 1/3

P(Paper) = 1/3

P(Scissors) = 1/3

is a mixed strategy.

A pure strategy is also a mixed strategy, but with a probability distribution that places all the probability on one outcome.

E.g.: P(Rock) = 0, P(Paper) = 1, P(Scissors) = 0.

Definition: Mixed Strategy Nash Equilibrium

A mixed strategy profile for players A and B is a pair of mixed strategies, distribution PA for player A and distribution PBfor player B.

A mixed strategy Nash equilibrium is a mixed strategy profile (PA, PB) such that neither player would gain by choosing a different mixed strategy, assuming the other player’s mixed strategy stays the same.

Quiz: Mixed Strategy Nash Equilibria

Player 1

Rock

Paper

Scissors

Can you find a mixed-strategy Nash equilibrium for Rock Paper Scissors?

(This can be hard in general, but see if you can guess it for this game.)

Rock

Paper

Player 2

Scissors

Answer: Mixed Strategy Nash Equilibria

Player 1

Rock

Paper

Scissors

The mixed strategy profile where each player has 1/3 probability for each action is a Nash equilibrium.

(It’s the only Nash equilibrium for this game.)

Rock

Paper

Player 2

Scissors

Nash’s Theorem

Theorem (Nash, 1951): Every finite game (finite number of players, finite number of pure strategies) has at least one mixed-strategy Nash equilibrium.

Nash, John (1951) "Non-Cooperative Games" The Annals of Mathematics 54(2):286-295.

(John Nash did not call them “Nash equilibria”, that name came later.)

He shared the 1994 Nobel Memorial Prize in Economic Sciences with game theorists ReinhardSelten and John Harsanyi for his work on Nash equilibria.

He suffered from schizophrenia in the 1950s and 1960s, as depicted in the 1998 film, “A Beautiful Mind”. He nevertheless recovered enough to return to academia and continue his research.

More on Constant-Sum Games

Minimax Theorem(John von Neumann, 1928): For every two-person, zero-sum game with finitely many pure strategies, there exists a mixed strategy for each player and a value V such that:

- Given player 2’s strategy, the best possible payoff for player 1 is V
- Given player 1’s strategy, the best possible payoff for player 2 is –V.

The existence of strategies part is a special case of Nash’s theorem, and a precursor to it.

This basically says that player 1 can guarantee himself a payoff of at least V, and player 2 can guarantee himself a payoff of at least –V. If both players play optimally, that’s exactly what they will get.

It’s called “minimax” because the players get this value by pursuing a strategy that tries to minimize the maximum payoff of the other player. We’ll come back to this.

Definition: The value V is called thevalue of the game.

Eg: The value of Rock-paper-scissors is 0; the best that P1 can hope to achieve, assuming P2 plays optimally (1/3 probability of each action), is a payoff of 0.

Download Presentation

Connecting to Server..