Loading in 5 sec....

n -Player Stochastic Games with Additive TransitionsPowerPoint Presentation

n -Player Stochastic Games with Additive Transitions

- 100 Views
- Uploaded on
- Presentation posted in: General

n -Player Stochastic Games with Additive Transitions

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Frank ThuijsmanJános Flesch & Koos VriezeMaastricht University

European Journal of Operational Research 179 (2007) 483–497

Center for the Study of Rationality

Hebrew University of Jerusalem

- Model
- Brief History of Stochastic Games
- Additive Transitions
- Examples

Center for the Study of Rationality

Hebrew University of Jerusalem

as = (a1s , a2s , , ans) joint action

rs(as) = (r1s(as), r2s(as), , rns(as)) rewards

ps(as) = (ps(1|as), ps(2|as), , ps(z|as)) transitions

1

s

z

- Infinite horizon
- Complete Information
- Perfect Recall
- Independent and Simultaneous Choices

rs(as)

ps(as)

Center for the Study of Rationality

Hebrew University of Jerusalem

F

3

0, 0, 0

0, 0, 0

N

L

2

R

(2/3, 0, 0, 1/3)

(1/3, 0, 1/3, 1/3)

0, 0, 0

0, 0, 0

0, 0, 0

0, 0, 0

T

(1, 0, 0, 0)

(2/3, 0, 1/3, 0)

1

(0, 1/3, 1/3, 1/3)

(1/3, 1/3, 0, 1/3)

0, 0, 0

0, 0, 0

1

B

(2/3, 1/3, 0, 0)

(1/3, 1/3, 1/3, 0)

1, 3, 0

0, 3, 1

3, 0, 1

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem

general strategy i : N×S ×H→Xi

(k, s, h) →Xis

Markov strategy fi : N×S→Xi

(k, s) →Xis

stationary strategy xi : S→Xi

(s) →Xis

opponents’ strategy -i , f-i and x-i

mixed actions

Center for the Study of Rationality

Hebrew University of Jerusalem

-Discounted rewards (with 0 << 1)

is() = Es((1-) k k-1Rik)

Limiting average rewards

is() = Es(limK→K-1Kk=1Rik)

Center for the Study of Rationality

Hebrew University of Jerusalem

-Discounted minmax

vis = inf -isup i is()

Limiting average minmax

vis = inf -isup i is()

Highest rewards player i can defend

Center for the Study of Rationality

Hebrew University of Jerusalem

= (i)iN is an -equilibrium if

is(i, -i) ≤is() +

for all i, for all i and for all s.

Center for the Study of Rationality

Hebrew University of Jerusalem

Any -equilibrium?

0, 0

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem

Shapley, 1953

0-sum, “discounted”

Everett, 1957

0-sum, recursive, undiscounted

Gillette, 1957

0-sum, big match problem

Center for the Study of Rationality

Hebrew University of Jerusalem

Fink, 1964 & Takahashi, 1964

n-player, discounted

Blackwell & Ferguson, 1968

0-sum, big match solution

Liggett & Lippmann, 1969

0-sum, perfect inf., undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem

Kohlberg, 1974

0-sum, absorbing, undiscounted

Mertens & Neyman, 1981

0-sum, undiscounted

Sorin, 1986

2-player, Paris Match, undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem

Vrieze & Thuijsman, 1989

2-player, absorbing, undiscounted

Thuijsman & Raghavan, 1997

n-player, perfect inf., undiscounted

Flesch, Thuijsman, Vrieze, 1997

3-player, absorbing example, undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem

Solan, 1999

3-player, absorbing, undiscounted

Vieille, 2000

2-player, undiscounted

Solan & Vieille, 2001

n-player, quitting, undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem

ps(as) = ni=1is pis(ais)

pis(ais) transition probabilities controlled by player i in state s

istransition power of player i in state s

0 ≤is ≤1 and iis = 1 for each s

Center for the Study of Rationality

Hebrew University of Jerusalem

ps(as) = ni=1is pis(ais)

21 = 0.7

p21(1)=(1, 0, 0)

p21(2)=(0, 1, 0)

11 = 0.3

(1, 0, 0)

(0.3, 0.7, 0)

p11(1) = (1, 0, 0)

p11(2) = (0, 0, 1)

(0.7, 0, 0.3)

(0, 0.7, 0.3)

1

Center for the Study of Rationality

Hebrew University of Jerusalem

- 0-equilibria for n-player AT games (threats!)
- 0-opt. stationary strat. for 0-sum AT games
- Stat. -equilibria for 2-player abs. AT games
- Result 3 can not be strengthened,
neither to 3-player abs. AT games,

nor to 2-player non-abs. AT games,

nor to give stat. 0-equilibria

Center for the Study of Rationality

Hebrew University of Jerusalem

Additive Transitions

induce

a Complete Ordering of the Actions

If ais is “better” than bis against some strategy,

Then ais is “better” than bis against any strategy.

Center for the Study of Rationality

Hebrew University of Jerusalem

Consider strategies ais and bis for player i

If, for some strategya-iswe have

t Sps(t | ais , a-is ) vit ≥ t Sps(t | bis , a-is ) vit

Then for all strategies b-iswe have

t Sps(t | ais , b-is ) vit ≥ t Sps(t | bis , b-is ) vit

Center for the Study of Rationality

Hebrew University of Jerusalem

If t Sps(t | ais , a-is)vit ≥ t Sps(t | bis , a-is)vit

Then

ist Sps(t | ais) vit + j ijst Sps(t | a-js) vit ≥

ist Sps(t | bis) vit + j ijst Sps(t | a-js) vit

Center for the Study of Rationality

Hebrew University of Jerusalem

ist Sps(t | ais) vit + j ijst Sps(t | b-js) vit ≥

ist Sps(t | bis) vit + j ijst Sps(t | b-js) vit

And therefore

t Sps(t | ais , b-is)vit ≥ t Sps(t | bis , b-is)vit

Center for the Study of Rationality

Hebrew University of Jerusalem

The “best” actions for player i in state s

are those that maximize the expression

t Sps(t | ais) vit

Center for the Study of Rationality

Hebrew University of Jerusalem

Let G be the original AT game and

let G* be the restricted AT game,

where each player is restricted

to his “best” actions.

Center for the Study of Rationality

Hebrew University of Jerusalem

Now v*i≥ vi for each player i.

In G* : t Sps(t | a*s) v*it = v*isi, s, a*s

In G : t Sps(t | bis , a*-is ) vit < visi, s,

a*-is , bis

Center for the Study of Rationality

Hebrew University of Jerusalem

If x*iyields at least v*i in G*,

then x*iyields at least v*i in G as well.

Center for the Study of Rationality

Hebrew University of Jerusalem

0.5

(1, 0, 0)

(0, 1, 0)

0, 0

0, 0

(1, 0, 0)

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

0.5

(0.5, 0, 0.5)

(0, 0, 1)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

Center for the Study of Rationality

Hebrew University of Jerusalem

NO stationary 0-equilibrium

0, 0

0, 0

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

(0.5, 0, 0.5)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

B, L

T, L

T, L

T, R

B, R

-1, 3

0, 0

0, 0

-3, 1

-2, 2

Center for the Study of Rationality

Hebrew University of Jerusalem

1-/2

/2

stationary -equilibrium

with >0

0, 0

0, 0

0

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

1

(0.5, 0, 0.5)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

equilibrium rewards ≈ ((-1-, 3-), (-3,1), (-1,3))

Center for the Study of Rationality

Hebrew University of Jerusalem

non-stationary 0-equilibrium

Player 1: B, T, T, T, ….

Player 2: R, R, R, R, ….

0, 0

0, 0

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

(0.5, 0, 0.5)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

equilibrium rewards ((-2, 2), (-3, 1), (-1, 3))

Center for the Study of Rationality

Hebrew University of Jerusalem

NO stationary -equilibrium

with >0

0, 0

p

q

1- q

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

1- p

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

p > 1 -

q > 0

p <

q = 0

q > 0

Center for the Study of Rationality

Hebrew University of Jerusalem

non-stationary 0-equilibrium

Player 1: T, B, B, B, ….

Player 2: R, R, R, R, ….

0, 0

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

equilibrium rewards ((2, 1), (0, 0), (3, -1), 2, 1))

Center for the Study of Rationality

Hebrew University of Jerusalem

alternative 0-equilibrium

Player 1: T, T, T, T, ….

Player 2: L, R, R, R, ….

0, 0

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

equilibrium rewards ((2, 1), (2, 1), (3, -1), 2, 1))

Center for the Study of Rationality

Hebrew University of Jerusalem

F

3

0, 0, 0

0, 0, 0

N

L

2

R

(2/3, 0, 0, 1/3)

(1/3, 0, 1/3, 1/3)

0, 0, 0

0, 0, 0

0, 0, 0

0, 0, 0

T

(1, 0, 0, 0)

(2/3, 0, 1/3, 0)

1

(0, 1/3, 1/3, 1/3)

(1/3, 1/3, 0, 1/3)

0, 0, 0

0, 0, 0

1

B

(2/3, 1/3, 0, 0)

(1/3, 1/3, 1/3, 0)

1, 3, 0

0, 1, 3

3, 0, 1

How to share 4 among three people

if only few solutions are allowed?

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem

F

3/2, 1/2, 2

3, 0, 1

N

L

R

(2/3, 0, 0, 1/3)

(1/3, 0, 1/3, 1/3)

0, 0, 0

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

(1, 0, 0, 0)

(2/3, 0, 1/3, 0)

(0, 1/3, 1/3, 1/3)

(1/3, 1/3, 0, 1/3)

1/2, 2, 3/2

1, 3, 0

1

B

(2/3, 1/3, 0, 0)

(1/3, 1/3, 1/3, 0)

1, 3, 0

0, 1, 3

3, 0, 1

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem

F

3/2, 1/2, 2

3, 0, 1

3, 0, 1

N

L

R

1/3 *

2/3 *

0, 0, 0

0, 1, 3

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

1/3 *

1 *

2/3 *

1/2, 2, 3/2

1, 3, 0

1, 3, 0

B

NO stationary -equilibrium

1/3 *

2/3 *

Center for the Study of Rationality

Hebrew University of Jerusalem

F

3/2, 1/2, 2

3, 0, 1

N

L

R

1/3 *

2/3 *

0, 0, 0

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

1/3 *

1 *

2/3 *

1/2, 2, 3/2

1, 3, 0

B

non-stationary 0-equilibrium

1/3 *

2/3 *

Player 1 on B: 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, ….

Player 2 on R:0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, ….

Player 3 on F:0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾,….

Center for the Study of Rationality

Hebrew University of Jerusalem

F

3/2, 1/2, 2

3, 0, 1

N

L

R

1/3 *

2/3 *

0, 0, 0

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

1/3 *

1 *

2/3 *

1/2, 2, 3/2

1, 3, 0

B

equilibrium rewards (1, 2, 1)

1/3 *

2/3 *

Player 1 on B: 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, ….

Player 2 on R:0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, ….

Player 3 on F:0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾,….

Center for the Study of Rationality

Hebrew University of Jerusalem

- 0-equilibria for n-player AT games (threats!)
- 0-opt. stationary strat. for 0-sum AT games
- Stat. -equilibria for 2-player abs. AT games
- Result 3 can not be strengthened,
neither to 3-player abs. AT games,

nor to 2-player non-abs. AT games,

nor to give stat. 0-equilibria

Center for the Study of Rationality

Hebrew University of Jerusalem

?

frank@math.unimaas.nl

Center for the Study of Rationality

Hebrew University of Jerusalem

GAME VER

Center for the Study of Rationality

Hebrew University of Jerusalem

GAME VER

Center for the Study of Rationality

Hebrew University of Jerusalem

GAME VER

Center for the Study of Rationality

Hebrew University of Jerusalem

GAME VER

Center for the Study of Rationality

Hebrew University of Jerusalem