N player stochastic games with additive transitions
This presentation is the property of its rightful owner.
Sponsored Links
1 / 43

n -Player Stochastic Games with Additive Transitions PowerPoint PPT Presentation


  • 71 Views
  • Uploaded on
  • Presentation posted in: General

n -Player Stochastic Games with Additive Transitions. Frank Thuijsman János Flesch & Koos Vrieze Maastricht University. European Journal of Operational Research 179 (2007) 483–497. Outline. Model Brief History of Stochastic Games Additive Transitions Examples. Finite Stochastic Game.

Download Presentation

n -Player Stochastic Games with Additive Transitions

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


N player stochastic games with additive transitions

n-Player Stochastic Gameswith Additive Transitions

Frank ThuijsmanJános Flesch & Koos VriezeMaastricht University

European Journal of Operational Research 179 (2007) 483–497

Center for the Study of Rationality

Hebrew University of Jerusalem


Outline

Outline

  • Model

  • Brief History of Stochastic Games

  • Additive Transitions

  • Examples

Center for the Study of Rationality

Hebrew University of Jerusalem


Finite stochastic game

Finite Stochastic Game

as = (a1s , a2s ,  , ans) joint action

rs(as) = (r1s(as), r2s(as),  , rns(as)) rewards

ps(as) = (ps(1|as), ps(2|as),  , ps(z|as)) transitions

1

s

z

  • Infinite horizon

  • Complete Information

  • Perfect Recall

  • Independent and Simultaneous Choices

rs(as)

ps(as)

Center for the Study of Rationality

Hebrew University of Jerusalem


3 player stochastic game

3-Player Stochastic Game

F

3

0, 0, 0

0, 0, 0

N

L

2

R

(2/3, 0, 0, 1/3)

(1/3, 0, 1/3, 1/3)

0, 0, 0

0, 0, 0

0, 0, 0

0, 0, 0

T

(1, 0, 0, 0)

(2/3, 0, 1/3, 0)

1

(0, 1/3, 1/3, 1/3)

(1/3, 1/3, 0, 1/3)

0, 0, 0

0, 0, 0

1

B

(2/3, 1/3, 0, 0)

(1/3, 1/3, 1/3, 0)

1, 3, 0

0, 3, 1

3, 0, 1

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem


Strategies

Strategies

general strategy i : N×S ×H→Xi

(k, s, h) →Xis

Markov strategy fi : N×S→Xi

(k, s) →Xis

stationary strategy xi : S→Xi

(s) →Xis

opponents’ strategy -i , f-i and x-i

mixed actions

Center for the Study of Rationality

Hebrew University of Jerusalem


Rewards

Rewards

-Discounted rewards (with 0 << 1)

 is() = Es((1-) k k-1Rik)

Limiting average rewards

 is() = Es(limK→K-1Kk=1Rik)

Center for the Study of Rationality

Hebrew University of Jerusalem


Minmax values

MinMax Values

-Discounted minmax

vis = inf -isup  i is()

Limiting average minmax

vis = inf -isup  i is()

Highest rewards player i can defend

Center for the Study of Rationality

Hebrew University of Jerusalem


Equilibrium

-Equilibrium

 = (i)iN is an -equilibrium if

 is(i, -i) ≤is() + 

for all i, for all i and for all s.

Center for the Study of Rationality

Hebrew University of Jerusalem


Question

Question

Any -equilibrium?

0, 0

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem


Highlights from finite stochastic games history

Highlights from (Finite)Stochastic Games History

Shapley, 1953

0-sum, “discounted”

Everett, 1957

0-sum, recursive, undiscounted

Gillette, 1957

0-sum, big match problem

Center for the Study of Rationality

Hebrew University of Jerusalem


Highlights from finite stochastic games history1

Highlights from (Finite)Stochastic Games History

Fink, 1964 & Takahashi, 1964

n-player, discounted

Blackwell & Ferguson, 1968

0-sum, big match solution

Liggett & Lippmann, 1969

0-sum, perfect inf., undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem


Highlights from finite stochastic games history2

Highlights from (Finite)Stochastic Games History

Kohlberg, 1974

0-sum, absorbing, undiscounted

Mertens & Neyman, 1981

0-sum, undiscounted

Sorin, 1986

2-player, Paris Match, undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem


Highlights from finite stochastic games history3

Highlights from (Finite)Stochastic Games History

Vrieze & Thuijsman, 1989

2-player, absorbing, undiscounted

Thuijsman & Raghavan, 1997

n-player, perfect inf., undiscounted

Flesch, Thuijsman, Vrieze, 1997

3-player, absorbing example, undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem


Highlights from finite stochastic games history4

Highlights from (Finite)Stochastic Games History

Solan, 1999

3-player, absorbing, undiscounted

Vieille, 2000

2-player, undiscounted

Solan & Vieille, 2001

n-player, quitting, undiscounted

Center for the Study of Rationality

Hebrew University of Jerusalem


Additive transitions

Additive Transitions

ps(as) = ni=1is pis(ais)

pis(ais) transition probabilities controlled by player i in state s

istransition power of player i in state s

0 ≤is ≤1 and iis = 1 for each s

Center for the Study of Rationality

Hebrew University of Jerusalem


Example for 2 player additive transitions

Example for 2-PlayerAdditive Transitions

ps(as) = ni=1is pis(ais)

21 = 0.7

p21(1)=(1, 0, 0)

p21(2)=(0, 1, 0)

11 = 0.3

(1, 0, 0)

(0.3, 0.7, 0)

p11(1) = (1, 0, 0)

p11(2) = (0, 0, 1)

(0.7, 0, 0.3)

(0, 0.7, 0.3)

1

Center for the Study of Rationality

Hebrew University of Jerusalem


Results

Results

  • 0-equilibria for n-player AT games (threats!)

  • 0-opt. stationary strat. for 0-sum AT games

  • Stat. -equilibria for 2-player abs. AT games

  • Result 3 can not be strengthened,

    neither to 3-player abs. AT games,

    nor to 2-player non-abs. AT games,

    nor to give stat. 0-equilibria

Center for the Study of Rationality

Hebrew University of Jerusalem


The essential observation

The Essential Observation

Additive Transitions

induce

a Complete Ordering of the Actions

If ais is “better” than bis against some strategy,

Then ais is “better” than bis against any strategy.

Center for the Study of Rationality

Hebrew University of Jerusalem


Better

“Better”

Consider strategies ais and bis for player i

If, for some strategya-iswe have

t Sps(t | ais , a-is ) vit ≥ t Sps(t | bis , a-is ) vit

Then for all strategies b-iswe have

t Sps(t | ais , b-is ) vit ≥ t Sps(t | bis , b-is ) vit

Center for the Study of Rationality

Hebrew University of Jerusalem


Because

Because ….

If t Sps(t | ais , a-is)vit ≥ t Sps(t | bis , a-is)vit

Then

ist Sps(t | ais) vit + j  ijst Sps(t | a-js) vit ≥

ist Sps(t | bis) vit + j  ijst Sps(t | a-js) vit

Center for the Study of Rationality

Hebrew University of Jerusalem


Which implies that

which implies that ….

ist Sps(t | ais) vit + j  ijst Sps(t | b-js) vit ≥

ist Sps(t | bis) vit + j  ijst Sps(t | b-js) vit

And therefore

t Sps(t | ais , b-is)vit ≥ t Sps(t | bis , b-is)vit

Center for the Study of Rationality

Hebrew University of Jerusalem


N player stochastic games with additive transitions

“Best”

The “best” actions for player i in state s

are those that maximize the expression

t Sps(t | ais) vit

Center for the Study of Rationality

Hebrew University of Jerusalem


The restricted game

The Restricted Game

Let G be the original AT game and

let G* be the restricted AT game,

where each player is restricted

to his “best” actions.

Center for the Study of Rationality

Hebrew University of Jerusalem


The restricted game1

The Restricted Game

Now v*i≥ vi for each player i.

In G* : t Sps(t | a*s) v*it = v*isi, s, a*s

In G : t Sps(t | bis , a*-is ) vit < visi, s,

a*-is , bis

Center for the Study of Rationality

Hebrew University of Jerusalem


The restricted game2

The Restricted Game

If x*iyields at least v*i in G*,

then x*iyields at least v*i in G as well.

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex 1 2 player absorbing at game

Ex. 1: 2-Player Absorbing AT Game

0.5

(1, 0, 0)

(0, 1, 0)

0, 0

0, 0

(1, 0, 0)

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

0.5

(0.5, 0, 0.5)

(0, 0, 1)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex 1 2 player absorbing at game1

Ex. 1: 2-Player Absorbing AT Game

NO stationary 0-equilibrium

0, 0

0, 0

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

(0.5, 0, 0.5)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

B, L

T, L

T, L

T, R

B, R

-1, 3

0, 0

0, 0

-3, 1

-2, 2

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex 1 2 player absorbing at game2

Ex. 1: 2-Player Absorbing AT Game

1-/2

/2

stationary -equilibrium

with  >0

0, 0

0, 0

0

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

1

(0.5, 0, 0.5)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

equilibrium rewards ≈ ((-1-, 3-), (-3,1), (-1,3))

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex 1 2 player absorbing at game3

Ex. 1: 2-Player Absorbing AT Game

non-stationary 0-equilibrium

Player 1: B, T, T, T, ….

Player 2: R, R, R, R, ….

0, 0

0, 0

(1, 0, 0)

(0.5, 0.5, 0)

-3, 1

-1, 3

0, 0

0, 0

(0.5, 0, 0.5)

(0, 0.5, 0.5)

(0, 1, 0)

(0, 0, 1)

2

1

3

equilibrium rewards ((-2, 2), (-3, 1), (-1, 3))

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex 2 2 player non absorbing at game

Ex. 2: 2-Player Non-Absorbing AT Game

NO stationary -equilibrium

with  >0

0, 0

p

q

1- q

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

1- p

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

p > 1 -

q > 0

p < 

q = 0

q > 0

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex 2 2 player non absorbing at game1

Ex. 2: 2-Player Non-Absorbing AT Game

non-stationary 0-equilibrium

Player 1: T, B, B, B, ….

Player 2: R, R, R, R, ….

0, 0

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

equilibrium rewards ((2, 1), (0, 0), (3, -1), 2, 1))

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex 2 2 player non absorbing at game2

Ex. 2: 2-Player Non-Absorbing AT Game

alternative 0-equilibrium

Player 1: T, T, T, T, ….

Player 2: L, R, R, R, ….

0, 0

(0, 0, 0, 1)

0, 0

3, -1

0, 0

0, 0

2,1

(0, 0.5, 0.5, 0)

(1, 0, 0, 0)

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

1

3

4

equilibrium rewards ((2, 1), (2, 1), (3, -1), 2, 1))

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex 3 3 player absorbing at game

Ex. 3: 3-Player Absorbing AT Game

F

3

0, 0, 0

0, 0, 0

N

L

2

R

(2/3, 0, 0, 1/3)

(1/3, 0, 1/3, 1/3)

0, 0, 0

0, 0, 0

0, 0, 0

0, 0, 0

T

(1, 0, 0, 0)

(2/3, 0, 1/3, 0)

1

(0, 1/3, 1/3, 1/3)

(1/3, 1/3, 0, 1/3)

0, 0, 0

0, 0, 0

1

B

(2/3, 1/3, 0, 0)

(1/3, 1/3, 1/3, 0)

1, 3, 0

0, 1, 3

3, 0, 1

How to share 4 among three people

if only few solutions are allowed?

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex 3 3 player absorbing at game1

Ex. 3: 3-Player Absorbing AT Game

F

3/2, 1/2, 2

3, 0, 1

N

L

R

(2/3, 0, 0, 1/3)

(1/3, 0, 1/3, 1/3)

0, 0, 0

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

(1, 0, 0, 0)

(2/3, 0, 1/3, 0)

(0, 1/3, 1/3, 1/3)

(1/3, 1/3, 0, 1/3)

1/2, 2, 3/2

1, 3, 0

1

B

(2/3, 1/3, 0, 0)

(1/3, 1/3, 1/3, 0)

1, 3, 0

0, 1, 3

3, 0, 1

(0, 1, 0, 0)

(0, 0, 1, 0)

(0, 0, 0, 1)

2

3

4

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex 3 3 player absorbing at game2

Ex. 3: 3-Player Absorbing AT Game

F

3/2, 1/2, 2

3, 0, 1

3, 0, 1

N

L

R

1/3 *

2/3 *

0, 0, 0

0, 1, 3

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

1/3 *

1 *

2/3 *

1/2, 2, 3/2

1, 3, 0

1, 3, 0

B

NO stationary -equilibrium

1/3 *

2/3 *

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex 3 3 player absorbing at game3

Ex. 3: 3-Player Absorbing AT Game

F

3/2, 1/2, 2

3, 0, 1

N

L

R

1/3 *

2/3 *

0, 0, 0

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

1/3 *

1 *

2/3 *

1/2, 2, 3/2

1, 3, 0

B

non-stationary 0-equilibrium

1/3 *

2/3 *

Player 1 on B: 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, ….

Player 2 on R:0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, ….

Player 3 on F:0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾,….

Center for the Study of Rationality

Hebrew University of Jerusalem


Ex 3 3 player absorbing at game4

Ex. 3: 3-Player Absorbing AT Game

F

3/2, 1/2, 2

3, 0, 1

N

L

R

1/3 *

2/3 *

0, 0, 0

0, 1, 3

4/3, 4/3, 4/3

2, 3/2, 1/2

T

1/3 *

1 *

2/3 *

1/2, 2, 3/2

1, 3, 0

B

equilibrium rewards (1, 2, 1)

1/3 *

2/3 *

Player 1 on B: 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, ….

Player 2 on R:0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾, 0, 0, ….

Player 3 on F:0, 0, 0, 0, 1, ¾, 0, 0, 0, 0, 1, ¾,….

Center for the Study of Rationality

Hebrew University of Jerusalem


Results1

Results

  • 0-equilibria for n-player AT games (threats!)

  • 0-opt. stationary strat. for 0-sum AT games

  • Stat. -equilibria for 2-player abs. AT games

  • Result 3 can not be strengthened,

    neither to 3-player abs. AT games,

    nor to 2-player non-abs. AT games,

    nor to give stat. 0-equilibria

Center for the Study of Rationality

Hebrew University of Jerusalem


N player stochastic games with additive transitions

?

[email protected]

Center for the Study of Rationality

Hebrew University of Jerusalem


N player stochastic games with additive transitions

GAME VER

Center for the Study of Rationality

Hebrew University of Jerusalem


N player stochastic games with additive transitions

GAME VER

Center for the Study of Rationality

Hebrew University of Jerusalem


N player stochastic games with additive transitions

GAME VER

Center for the Study of Rationality

Hebrew University of Jerusalem


N player stochastic games with additive transitions

GAME VER

Center for the Study of Rationality

Hebrew University of Jerusalem


  • Login