N player stochastic games with additive transitions
Sponsored Links
This presentation is the property of its rightful owner.
1 / 43

n -Player Stochastic Games with Additive Transitions PowerPoint PPT Presentation


  • 94 Views
  • Uploaded on
  • Presentation posted in: General

n -Player Stochastic Games with Additive Transitions. Frank Thuijsman János Flesch & Koos Vrieze Maastricht University. European Journal of Operational Research 179 (2007) 483–497. Outline. Model Brief History of Stochastic Games Additive Transitions Examples. Finite Stochastic Game.

Download Presentation

n -Player Stochastic Games with Additive Transitions

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


n-Player Stochastic Gameswith Additive Transitions

Frank ThuijsmanJános Flesch & Koos VriezeMaastricht University

European Journal of Operational Research 179 (2007) 483–497

Center for the Study of Rationality

Hebrew University of Jerusalem


Outline

  • Model

  • Brief History of Stochastic Games

  • Additive Transitions

  • Examples

Center for the Study of Rationality

Hebrew University of Jerusalem


Finite Stochastic Game

as = (a1s , a2s ,  , ans) joint action

rs(as) = (r1s(as), r2s(as),  , rns(as)) rewards

ps(as) = (ps(1|as), ps(2|as),  , ps(z|as)) transitions

1

s

z

  • Infinite horizon

  • Complete Information

  • Perfect Recall

  • Independent and Simultaneous Choices

rs(as)

ps(as)

Center for the Study of Rationality

Hebrew University of Jerusalem


3-Player Stochastic Game

F

3

0, 0, 0

0, 0, 0

N

L

2

R

(2/3, 0, 0, 1/3)

(1/3, 0, 1/3, 1/3)

0, 0, 0

0, 0, 0

0, 0, 0

0, 0, 0

T

(1, 0, 0, 0)

(2/3, 0, 1/3, 0)

1

(0, 1/3, 1/3, 1/3)

(1/3, 1/3, 0, 1/3)

0, 0, 0

0, 0, 0

1

B

(2/3, 1/3, 0, 0)

(1/3, 1/3, 1/3, 0)

1, 3, 0

0, 3, 1

3, 0, 1

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem


Strategies

general strategy i : N×S ×H→Xi

(k, s, h) →Xis

Markov strategy fi : N×S→Xi

(k, s) →Xis

stationary strategy xi : S→Xi

(s) →Xis

opponents’ strategy -i , f-i and x-i

mixed actions

Center for the Study of Rationality

Hebrew University of Jerusalem


Rewards

-Discounted rewards (with 0 << 1)

 is() = Es((1-) k k-1Rik)

Limiting average rewards

 is() = Es(limK→K-1Kk=1Rik)

Center for the Study of Rationality

Hebrew University of Jerusalem


MinMax Values

-Discounted minmax

vis = inf -isup  i is()

Limiting average minmax

vis = inf -isup  i is()

Highest rewards player i can defend

Center for the Study of Rationality

Hebrew University of Jerusalem


-Equilibrium

 = (i)iN is an -equilibrium if

 is(i, -i) ≤is() + 

for all i, for all i and for all s.

Center for the Study of Rationality

Hebrew University of Jerusalem


Question

Any -equilibrium?

0, 0

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem


Highlights from (Finite)Stochastic Games History

Shapley, 1953

0-sum, “discounted”

Everett, 1957

0-sum, recursive, undiscounted

Gillette, 1957

0-sum, big match problem

Center for the Study of Rationality

Hebrew University of Jerusalem


Highlights from (Finite)Stochastic Games History

Fink, 1964 & Takahashi, 1964

n-player, discounted

Blackwell & Ferguson, 1968

0-sum, big match solution

Liggett & Lippmann, 1969

0-sum, perfect inf., undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem


Highlights from (Finite)Stochastic Games History

Kohlberg, 1974

0-sum, absorbing, undiscounted

Mertens & Neyman, 1981

0-sum, undiscounted

Sorin, 1986

2-player, Paris Match, undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem


Highlights from (Finite)Stochastic Games History

Vrieze & Thuijsman, 1989

2-player, absorbing, undiscounted

Thuijsman & Raghavan, 1997

n-player, perfect inf., undiscounted

Flesch, Thuijsman, Vrieze, 1997

3-player, absorbing example, undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem


Highlights from (Finite)Stochastic Games History

Solan, 1999

3-player, absorbing, undiscounted

Vieille, 2000

2-player, undiscounted

Solan & Vieille, 2001

n-player, quitting, undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem


Additive Transitions

ps(as) = ni=1is pis(ais)

pis(ais) transition probabilities controlled by player i in state s

istransition power of player i in state s

0 ≤is ≤1 and iis = 1 for each s

Center for the Study of Rationality

Hebrew University of Jerusalem


Example for 2-PlayerAdditive Transitions

ps(as) = ni=1is pis(ais)

21 = 0.7

p21(1)=(1, 0, 0)

p21(2)=(0, 1, 0)

11 = 0.3

(1, 0, 0)

(0.3, 0.7, 0)

p11(1) = (1, 0, 0)

p11(2) = (0, 0, 1)

(0.7, 0, 0.3)

(0, 0.7, 0.3)

1

Center for the Study of Rationality

Hebrew University of Jerusalem


Results

  • 0-equilibria for n-player AT games (threats!)

  • 0-opt. stationary strat. for 0-sum AT games

  • Stat. -equilibria for 2-player abs. AT games

  • Result 3 can not be strengthened,

    neither to 3-player abs. AT games,

    nor to 2-player non-abs. AT games,

    nor to give stat. 0-equilibria

Center for the Study of Rationality

Hebrew University of Jerusalem


The Essential Observation

Additive Transitions

induce

a Complete Ordering of the Actions

If ais is “better” than bis against some strategy,

Then ais is “better” than bis against any strategy.

Center for the Study of Rationality

Hebrew University of Jerusalem


“Better”

Consider strategies ais and bis for player i

If, for some strategya-iswe have

t Sps(t | ais , a-is ) vit ≥ t Sps(t | bis , a-is ) vit

Then for all strategies b-iswe have

t Sps(t | ais , b-is ) vit ≥ t Sps(t | bis , b-is ) vit

Center for the Study of Rationality

Hebrew University of Jerusalem


Because ….

If t Sps(t | ais , a-is)vit ≥ t Sps(t | bis , a-is)vit

Then

ist Sps(t | ais) vit + j  ijst Sps(t | a-js) vit ≥

ist Sps(t | bis) vit + j  ijst Sps(t | a-js) vit

Center for the Study of Rationality

Hebrew University of Jerusalem


which implies that ….

ist Sps(t | ais) vit + j  ijst Sps(t | b-js) vit ≥

ist Sps(t | bis) vit + j  ijst Sps(t | b-js) vit

And therefore

t Sps(t | ais , b-is)vit ≥ t Sps(t | bis , b-is)vit

Center for the Study of Rationality

Hebrew University of Jerusalem


“Best”

The “best” actions for player i in state s

are those that maximize the expression

t Sps(t | ais) vit

Center for the Study of Rationality

Hebrew University of Jerusalem


The Restricted Game

Let G be the original AT game and

let G* be the restricted AT game,

where each player is restricted

to his “best” actions.

Center for the Study of Rationality

Hebrew University of Jerusalem


The Restricted Game

Now v*i≥ vi for each player i.

In G* : t Sps(t | a*s) v*it = v*isi, s, a*s

In G : t Sps(t | bis , a*-is ) vit < visi, s,

a*-is , bis

Center for the Study of Rationality

Hebrew University of Jerusalem


The Restricted Game

If x*iyields at least v*i in G*,

then x*iyields at least v*i in G as well.

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex. 1: 2-Player Absorbing AT Game

0.5

(1, 0, 0)

(0, 1, 0)

0, 0

0, 0

(1, 0, 0)

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

0.5

(0.5, 0, 0.5)

(0, 0, 1)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex. 1: 2-Player Absorbing AT Game

NO stationary 0-equilibrium

0, 0

0, 0

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

(0.5, 0, 0.5)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

B, L

T, L

T, L

T, R

B, R

-1, 3

0, 0

0, 0

-3, 1

-2, 2

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex. 1: 2-Player Absorbing AT Game

1-/2

/2

stationary -equilibrium

with  >0

0, 0

0, 0

0

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

1

(0.5, 0, 0.5)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

equilibrium rewards ≈ ((-1-, 3-), (-3,1), (-1,3))

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex. 1: 2-Player Absorbing AT Game

non-stationary 0-equilibrium

Player 1: B, T, T, T, ….

Player 2: R, R, R, R, ….

0, 0

0, 0

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

(0.5, 0, 0.5)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

equilibrium rewards ((-2, 2), (-3, 1), (-1, 3))

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex. 2: 2-Player Non-Absorbing AT Game

NO stationary -equilibrium

with  >0

0, 0

p

q

1- q

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

1- p

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

p > 1 -

q > 0

p < 

q = 0

q > 0

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex. 2: 2-Player Non-Absorbing AT Game

non-stationary 0-equilibrium

Player 1: T, B, B, B, ….

Player 2: R, R, R, R, ….

0, 0

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

equilibrium rewards ((2, 1), (0, 0), (3, -1), 2, 1))

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex. 2: 2-Player Non-Absorbing AT Game

alternative 0-equilibrium

Player 1: T, T, T, T, ….

Player 2: L, R, R, R, ….

0, 0

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

equilibrium rewards ((2, 1), (2, 1), (3, -1), 2, 1))

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex. 3: 3-Player Absorbing AT Game

F

3

0, 0, 0

0, 0, 0

N

L

2

R

(2/3, 0, 0, 1/3)

(1/3, 0, 1/3, 1/3)

0, 0, 0

0, 0, 0

0, 0, 0

0, 0, 0

T

(1, 0, 0, 0)

(2/3, 0, 1/3, 0)

1

(0, 1/3, 1/3, 1/3)

(1/3, 1/3, 0, 1/3)

0, 0, 0

0, 0, 0

1

B

(2/3, 1/3, 0, 0)

(1/3, 1/3, 1/3, 0)

1, 3, 0

0, 1, 3

3, 0, 1

How to share 4 among three people

if only few solutions are allowed?

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex. 3: 3-Player Absorbing AT Game

F

3/2, 1/2, 2

3, 0, 1

N

L

R

(2/3, 0, 0, 1/3)

(1/3, 0, 1/3, 1/3)

0, 0, 0

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

(1, 0, 0, 0)

(2/3, 0, 1/3, 0)

(0, 1/3, 1/3, 1/3)

(1/3, 1/3, 0, 1/3)

1/2, 2, 3/2

1, 3, 0

1

B

(2/3, 1/3, 0, 0)

(1/3, 1/3, 1/3, 0)

1, 3, 0

0, 1, 3

3, 0, 1

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex. 3: 3-Player Absorbing AT Game

F

3/2, 1/2, 2

3, 0, 1

3, 0, 1

N

L

R

1/3 *

2/3 *

0, 0, 0

0, 1, 3

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

1/3 *

1 *

2/3 *

1/2, 2, 3/2

1, 3, 0

1, 3, 0

B

NO stationary -equilibrium

1/3 *

2/3 *

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex. 3: 3-Player Absorbing AT Game

F

3/2, 1/2, 2

3, 0, 1

N

L

R

1/3 *

2/3 *

0, 0, 0

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

1/3 *

1 *

2/3 *

1/2, 2, 3/2

1, 3, 0

B

non-stationary 0-equilibrium

1/3 *

2/3 *

Player 1 on B: 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, ….

Player 2 on R:0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, ….

Player 3 on F:0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾,….

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex. 3: 3-Player Absorbing AT Game

F

3/2, 1/2, 2

3, 0, 1

N

L

R

1/3 *

2/3 *

0, 0, 0

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

1/3 *

1 *

2/3 *

1/2, 2, 3/2

1, 3, 0

B

equilibrium rewards (1, 2, 1)

1/3 *

2/3 *

Player 1 on B: 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, ….

Player 2 on R:0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, ….

Player 3 on F:0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾,….

Center for the Study of Rationality

Hebrew University of Jerusalem


Results

  • 0-equilibria for n-player AT games (threats!)

  • 0-opt. stationary strat. for 0-sum AT games

  • Stat. -equilibria for 2-player abs. AT games

  • Result 3 can not be strengthened,

    neither to 3-player abs. AT games,

    nor to 2-player non-abs. AT games,

    nor to give stat. 0-equilibria

Center for the Study of Rationality

Hebrew University of Jerusalem


?

[email protected]

Center for the Study of Rationality

Hebrew University of Jerusalem


GAME VER

Center for the Study of Rationality

Hebrew University of Jerusalem


GAME VER

Center for the Study of Rationality

Hebrew University of Jerusalem


GAME VER

Center for the Study of Rationality

Hebrew University of Jerusalem


GAME VER

Center for the Study of Rationality

Hebrew University of Jerusalem


  • Login