Loading in 5 sec....

n -Player Stochastic Games with Additive TransitionsPowerPoint Presentation

n -Player Stochastic Games with Additive Transitions

- By
**alder** - Follow User

- 114 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about ' n -Player Stochastic Games with Additive Transitions' - alder

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

n-Player Stochastic Gameswith Additive Transitions

Frank ThuijsmanJános Flesch & Koos VriezeMaastricht University

European Journal of Operational Research 179 (2007) 483–497

Center for the Study of Rationality

Hebrew University of Jerusalem

Outline

- Model
- Brief History of Stochastic Games
- Additive Transitions
- Examples

Center for the Study of Rationality

Hebrew University of Jerusalem

Finite Stochastic Game

as = (a1s , a2s , , ans) joint action

rs(as) = (r1s(as), r2s(as), , rns(as)) rewards

ps(as) = (ps(1|as), ps(2|as), , ps(z|as)) transitions

1

s

z

- Infinite horizon
- Complete Information
- Perfect Recall
- Independent and Simultaneous Choices

rs(as)

ps(as)

Center for the Study of Rationality

Hebrew University of Jerusalem

3-Player Stochastic Game

F

3

0, 0, 0

0, 0, 0

N

L

2

R

(2/3, 0, 0, 1/3)

(1/3, 0, 1/3, 1/3)

0, 0, 0

0, 0, 0

0, 0, 0

0, 0, 0

T

(1, 0, 0, 0)

(2/3, 0, 1/3, 0)

1

(0, 1/3, 1/3, 1/3)

(1/3, 1/3, 0, 1/3)

0, 0, 0

0, 0, 0

1

B

(2/3, 1/3, 0, 0)

(1/3, 1/3, 1/3, 0)

1, 3, 0

0, 3, 1

3, 0, 1

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem

Strategies

general strategy i : N×S ×H→Xi

(k, s, h) →Xis

Markov strategy fi : N×S→Xi

(k, s) →Xis

stationary strategy xi : S→Xi

(s) →Xis

opponents’ strategy -i , f-i and x-i

mixed actions

Center for the Study of Rationality

Hebrew University of Jerusalem

Rewards

-Discounted rewards (with 0 << 1)

is() = Es((1-) k k-1Rik)

Limiting average rewards

is() = Es(limK→K-1Kk=1Rik)

Center for the Study of Rationality

Hebrew University of Jerusalem

MinMax Values

-Discounted minmax

vis = inf -isup i is()

Limiting average minmax

vis = inf -isup i is()

Highest rewards player i can defend

Center for the Study of Rationality

Hebrew University of Jerusalem

-Equilibrium

= (i)iN is an -equilibrium if

is(i, -i) ≤is() +

for all i, for all i and for all s.

Center for the Study of Rationality

Hebrew University of Jerusalem

Question

Any -equilibrium?

0, 0

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem

Highlights from (Finite)Stochastic Games History

Shapley, 1953

0-sum, “discounted”

Everett, 1957

0-sum, recursive, undiscounted

Gillette, 1957

0-sum, big match problem

Center for the Study of Rationality

Hebrew University of Jerusalem

Highlights from (Finite)Stochastic Games History

Fink, 1964 & Takahashi, 1964

n-player, discounted

Blackwell & Ferguson, 1968

0-sum, big match solution

Liggett & Lippmann, 1969

0-sum, perfect inf., undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem

Highlights from (Finite)Stochastic Games History

Kohlberg, 1974

0-sum, absorbing, undiscounted

Mertens & Neyman, 1981

0-sum, undiscounted

Sorin, 1986

2-player, Paris Match, undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem

Highlights from (Finite)Stochastic Games History

Vrieze & Thuijsman, 1989

2-player, absorbing, undiscounted

Thuijsman & Raghavan, 1997

n-player, perfect inf., undiscounted

Flesch, Thuijsman, Vrieze, 1997

3-player, absorbing example, undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem

Highlights from (Finite)Stochastic Games History

Solan, 1999

3-player, absorbing, undiscounted

Vieille, 2000

2-player, undiscounted

Solan & Vieille, 2001

n-player, quitting, undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem

Additive Transitions

ps(as) = ni=1is pis(ais)

pis(ais) transition probabilities controlled by player i in state s

istransition power of player i in state s

0 ≤is ≤1 and iis = 1 for each s

Center for the Study of Rationality

Hebrew University of Jerusalem

Example for 2-PlayerAdditive Transitions

ps(as) = ni=1is pis(ais)

21 = 0.7

p21(1)=(1, 0, 0)

p21(2)=(0, 1, 0)

11 = 0.3

(1, 0, 0)

(0.3, 0.7, 0)

p11(1) = (1, 0, 0)

p11(2) = (0, 0, 1)

(0.7, 0, 0.3)

(0, 0.7, 0.3)

1

Center for the Study of Rationality

Hebrew University of Jerusalem

Results

- 0-equilibria for n-player AT games (threats!)
- 0-opt. stationary strat. for 0-sum AT games
- Stat. -equilibria for 2-player abs. AT games
- Result 3 can not be strengthened,
neither to 3-player abs. AT games,

nor to 2-player non-abs. AT games,

nor to give stat. 0-equilibria

Center for the Study of Rationality

Hebrew University of Jerusalem

The Essential Observation

Additive Transitions

induce

a Complete Ordering of the Actions

If ais is “better” than bis against some strategy,

Then ais is “better” than bis against any strategy.

Center for the Study of Rationality

Hebrew University of Jerusalem

“Better”

Consider strategies ais and bis for player i

If, for some strategya-iswe have

t Sps(t | ais , a-is ) vit ≥ t Sps(t | bis , a-is ) vit

Then for all strategies b-iswe have

t Sps(t | ais , b-is ) vit ≥ t Sps(t | bis , b-is ) vit

Center for the Study of Rationality

Hebrew University of Jerusalem

Because ….

If t Sps(t | ais , a-is)vit ≥ t Sps(t | bis , a-is)vit

Then

ist Sps(t | ais) vit + j ijst Sps(t | a-js) vit ≥

ist Sps(t | bis) vit + j ijst Sps(t | a-js) vit

Center for the Study of Rationality

Hebrew University of Jerusalem

which implies that ….

ist Sps(t | ais) vit + j ijst Sps(t | b-js) vit ≥

ist Sps(t | bis) vit + j ijst Sps(t | b-js) vit

And therefore

t Sps(t | ais , b-is)vit ≥ t Sps(t | bis , b-is)vit

Center for the Study of Rationality

Hebrew University of Jerusalem

“Best”

The “best” actions for player i in state s

are those that maximize the expression

t Sps(t | ais) vit

Center for the Study of Rationality

Hebrew University of Jerusalem

The Restricted Game

Let G be the original AT game and

let G* be the restricted AT game,

where each player is restricted

to his “best” actions.

Center for the Study of Rationality

Hebrew University of Jerusalem

The Restricted Game

Now v*i≥ vi for each player i.

In G* : t Sps(t | a*s) v*it = v*isi, s, a*s

In G : t Sps(t | bis , a*-is ) vit < visi, s,

a*-is , bis

Center for the Study of Rationality

Hebrew University of Jerusalem

The Restricted Game

If x*iyields at least v*i in G*,

then x*iyields at least v*i in G as well.

Center for the Study of Rationality

Hebrew University of Jerusalem

Ex. 1: 2-Player Absorbing AT Game

0.5

(1, 0, 0)

(0, 1, 0)

0, 0

0, 0

(1, 0, 0)

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

0.5

(0.5, 0, 0.5)

(0, 0, 1)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

Center for the Study of Rationality

Hebrew University of Jerusalem

Ex. 1: 2-Player Absorbing AT Game

NO stationary 0-equilibrium

0, 0

0, 0

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

(0.5, 0, 0.5)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

B, L

T, L

T, L

T, R

B, R

-1, 3

0, 0

0, 0

-3, 1

-2, 2

Center for the Study of Rationality

Hebrew University of Jerusalem

Ex. 1: 2-Player Absorbing AT Game

1-/2

/2

stationary -equilibrium

with >0

0, 0

0, 0

0

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

1

(0.5, 0, 0.5)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

equilibrium rewards ≈ ((-1-, 3-), (-3,1), (-1,3))

Center for the Study of Rationality

Hebrew University of Jerusalem

Ex. 1: 2-Player Absorbing AT Game

non-stationary 0-equilibrium

Player 1: B, T, T, T, ….

Player 2: R, R, R, R, ….

0, 0

0, 0

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

(0.5, 0, 0.5)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

equilibrium rewards ((-2, 2), (-3, 1), (-1, 3))

Center for the Study of Rationality

Hebrew University of Jerusalem

Ex. 2: 2-Player Non-Absorbing AT Game

NO stationary -equilibrium

with >0

0, 0

p

q

1- q

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

1- p

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

p > 1 -

q > 0

p <

q = 0

q > 0

Center for the Study of Rationality

Hebrew University of Jerusalem

Ex. 2: 2-Player Non-Absorbing AT Game

non-stationary 0-equilibrium

Player 1: T, B, B, B, ….

Player 2: R, R, R, R, ….

0, 0

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

equilibrium rewards ((2, 1), (0, 0), (3, -1), 2, 1))

Center for the Study of Rationality

Hebrew University of Jerusalem

Ex. 2: 2-Player Non-Absorbing AT Game

alternative 0-equilibrium

Player 1: T, T, T, T, ….

Player 2: L, R, R, R, ….

0, 0

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

equilibrium rewards ((2, 1), (2, 1), (3, -1), 2, 1))

Center for the Study of Rationality

Hebrew University of Jerusalem

Ex. 3: 3-Player Absorbing AT Game

F

3

0, 0, 0

0, 0, 0

N

L

2

R

(2/3, 0, 0, 1/3)

(1/3, 0, 1/3, 1/3)

0, 0, 0

0, 0, 0

0, 0, 0

0, 0, 0

T

(1, 0, 0, 0)

(2/3, 0, 1/3, 0)

1

(0, 1/3, 1/3, 1/3)

(1/3, 1/3, 0, 1/3)

0, 0, 0

0, 0, 0

1

B

(2/3, 1/3, 0, 0)

(1/3, 1/3, 1/3, 0)

1, 3, 0

0, 1, 3

3, 0, 1

How to share 4 among three people

if only few solutions are allowed?

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem

Ex. 3: 3-Player Absorbing AT Game

F

3/2, 1/2, 2

3, 0, 1

N

L

R

(2/3, 0, 0, 1/3)

(1/3, 0, 1/3, 1/3)

0, 0, 0

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

(1, 0, 0, 0)

(2/3, 0, 1/3, 0)

(0, 1/3, 1/3, 1/3)

(1/3, 1/3, 0, 1/3)

1/2, 2, 3/2

1, 3, 0

1

B

(2/3, 1/3, 0, 0)

(1/3, 1/3, 1/3, 0)

1, 3, 0

0, 1, 3

3, 0, 1

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem

Ex. 3: 3-Player Absorbing AT Game

F

3/2, 1/2, 2

3, 0, 1

3, 0, 1

N

L

R

1/3 *

2/3 *

0, 0, 0

0, 1, 3

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

1/3 *

1 *

2/3 *

1/2, 2, 3/2

1, 3, 0

1, 3, 0

B

NO stationary -equilibrium

1/3 *

2/3 *

Center for the Study of Rationality

Hebrew University of Jerusalem

Ex. 3: 3-Player Absorbing AT Game

F

3/2, 1/2, 2

3, 0, 1

N

L

R

1/3 *

2/3 *

0, 0, 0

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

1/3 *

1 *

2/3 *

1/2, 2, 3/2

1, 3, 0

B

non-stationary 0-equilibrium

1/3 *

2/3 *

Player 1 on B: 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, ….

Player 2 on R:0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, ….

Player 3 on F:0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾,….

Center for the Study of Rationality

Hebrew University of Jerusalem

Ex. 3: 3-Player Absorbing AT Game

F

3/2, 1/2, 2

3, 0, 1

N

L

R

1/3 *

2/3 *

0, 0, 0

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

1/3 *

1 *

2/3 *

1/2, 2, 3/2

1, 3, 0

B

equilibrium rewards (1, 2, 1)

1/3 *

2/3 *

Player 1 on B: 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, ….

Player 2 on R:0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, ….

Player 3 on F:0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾,….

Center for the Study of Rationality

Hebrew University of Jerusalem

Results

- 0-equilibria for n-player AT games (threats!)
- 0-opt. stationary strat. for 0-sum AT games
- Stat. -equilibria for 2-player abs. AT games
- Result 3 can not be strengthened,
neither to 3-player abs. AT games,

nor to 2-player non-abs. AT games,

nor to give stat. 0-equilibria

Center for the Study of Rationality

Hebrew University of Jerusalem

Download Presentation

Connecting to Server..