bayesian ranking from xbox live to computer go l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Bayesian Ranking: From Xbox Live to Computer Go PowerPoint Presentation
Download Presentation
Bayesian Ranking: From Xbox Live to Computer Go

Loading in 2 Seconds...

play fullscreen
1 / 30

Bayesian Ranking: From Xbox Live to Computer Go - PowerPoint PPT Presentation


  • 548 Views
  • Uploaded on

Bayesian Ranking: From Xbox Live to Computer Go Ralf Herbrich and Thore Graepel Overview Motivation: Ranking in Video Games Bayesian Player Ranking: TrueSkill™ Skill Belief, Likelihood, Update Equation Applications in Online Gaming Numerical Results on Halo 2 data

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Bayesian Ranking: From Xbox Live to Computer Go' - benjamin


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
bayesian ranking from xbox live to computer go

Bayesian Ranking:From Xbox Live to Computer Go

Ralf Herbrich and Thore Graepel

overview
Overview
  • Motivation: Ranking in Video Games
  • Bayesian Player Ranking: TrueSkill™
    • Skill Belief, Likelihood, Update Equation
    • Applications in Online Gaming
    • Numerical Results on Halo 2 data
  • Ranking Moves: Computer Go
  • Conclusion
motivation
Motivation
  • Microsoft is the leader in online video gaming (Xbox Live).
    • Centralised game-independent service.
    • Gamercard, Achievements, TrueSkill™, etc.
  • What makes playing online games fun?
    • Good network connection (Broadband).
    • Seamless setup (Xbox Live).
    • Competitive matches (Ranking!).
ranking in video games
Ranking in Video Games
  • Problem Setting:
    • k teams of n1,…,nk many players compete.
    • The outcome is a ranking between the teams (including draws).
  • Questions:
    • Skill si of each player such that
    • Global ranking between all players.
    • High quality of match between k teams.
overview5
Overview
  • Motivation: Ranking in Video Games
  • Bayesian Player Ranking: TrueSkill™
    • Skill Belief, Likelihood, Update Equation
    • Applications in Online Gaming
    • Numerical Results on Halo 2 data
  • Ranking Moves: Computer Go
  • Conclusion
the bayesian approach
The Bayesian Approach
  • Classical logic deals with certain statements.
  • In the real world, uncertainty is abundant.
  • Degree of Belief (logic: 0% or 100%).

“Player i’s skill is between 35 and 40” and

“Player j’s skill is between 30 and 35”

“Player i won against Player j”

Bayesian Approach = Probability for Logic under Uncertainty

P(A|B) = P(B|A) P(A) / P(B)

Posterior

Prior

Likelihood

Evidence

overview7
Overview
  • Motivation: Ranking in Video Games
  • Bayesian Player Ranking: TrueSkill™
    • Skill Belief, Likelihood, Update Equation
    • Applications in Online Gaming
    • Numerical Results on Halo 2 data
  • Ranking Moves: Computer Go
  • Conclusion
trueskill skill belief
TrueSkill™: Skill Belief
  • Track two numbers per player:
    • μ = Average skill of player
    • σ = Uncertainty about skill of player
  • Benefits:
    • Faster Skill Learning
    • Better Matchmaking
    • Accurate Prediction of Game Outcomes

σ

Belief in Skill Level

μ

10

15

20

25

30

35

40

Skill Level

trueskill likelihood
TrueSkill™: Likelihood
  • Likelihood:
  • Game outcomes are all permutations including draws between pairs of teams.
  • Latent performance model for players:

P(game outcome|s1,…,sn)=

P(ti’s are in game outcome order| s1,…,sn)

  • Team performance, ti, is sum of players’ performances in the team.
likelihood example two players
Likelihood Example: Two Players

7

0.2

Player 2 wins

0.18

6

Players 1 and 2 draw

0.16

5

0.14

0.12

4

Performance of Player 2

Probability density

0.1

3

0.08

0.06

2

Players 1 and 2 draw

0.04

Player 1 wins

1

0.02

Player 2 wins

Player 1 wins

0

0

0

1

2

3

4

5

6

7

-8

-6

-4

-2

0

2

4

Performance of Player 1

- x

x

1

2

trueskill skill updates

3rd Place

1st Place

2nd Place

TrueSkill™: Skill Updates

Game Outcome

Belief in Skill Level

0

10

20

30

40

50

overview12
Overview
  • Motivation: Ranking in Video Games
  • Bayesian Player Ranking: TrueSkill™
    • Skill Belief, Likelihood, Update Equation
    • Applications in Online Gaming
    • Numerical Results on Halo 2 data
  • Ranking Moves: Computer Go
  • Conclusion
the true skill system applications
The True Skill System: Applications
  • Leaderboard:
    • (Conservative) skill estimate : μ- 3·σ
  • Matchmaking:
    • Competitive game = Fun game!
    • Match quality = Probability of a draw
  • Team Balancing:
    • Maximise match quality by greedy search.
trueskill matchmaking

0.1

0.1

0.08

0.08

0.06

0.06

Probability density

Probability density

0.04

0.04

0.02

0.02

0

0

-5

-5

0

0

5

5

10

10

15

15

20

20

25

25

30

30

Skill

Skill

0.8

0.7

0.6

0.5

Probability density

0.4

0.3

0.2

0.1

0

-5

0

5

10

15

20

25

30

Skill

TrueSkill™: Matchmaking

Lobby

Possible Matches

?

overview15
Overview
  • Motivation: Ranking in Video Games
  • Bayesian Player Ranking: TrueSkill™
    • Skill Belief, Likelihood, Update Equation
    • Applications in Online Gaming
    • Numerical Results on Halo 2 data
  • Ranking Moves: Computer Go
  • Conclusion
results halo 2 multiplayer beta
Results: Halo 2 Multiplayer Beta
  • 5 different “hoppers”
    • Free-For-All: 60261 games (5946 players)
    • 1 vs. 1: 6240 games (1672 players)
  • 5 maps, 3 different game variants.
  • Matchmaking was relaxed to level gap 9.
  • Parameters in all experiments:
    • Performance variation factor: 60%
    • Draw Probability: 5%
    • Dynamics variation factor: 2%
free for all char vs sqlwildman
Free-For-All: char vs. SQLwildman?

40

35

30

25

Level

20

15

)

char (

TrueSkill

10

)

SQLwildman (

TrueSkill

char (Halo 2)

5

SQLwildman (Halo 2)

0

0

100

200

300

400

Number of games played

free for all char vs sqlwildman18
Free-For-All: char vs. SQLwildman?

100%

char wins

SQLwildman wins

80%

Both players draw

60%

Winning probability

40%

20%

5/8 games won by char

0%

0

100

200

300

400

500

Number of games played

near normal level distribution 650 000 players
Near Normal Level Distribution – 650,000 Players

TrueSkill™ Analysis of Halo 2 (Nov. – Dec. 2004)

50

40

30

Level

20

10

1

3

2.5

2

1.5

1

0.5

0

Level Occupancy

4

x 10

x 10

trueskill
TrueSkill™
  • Skill based ranking instead of experience based ranking for better matchmaking.
  • TrueSkill™ system is
    • a generalisation of ELO
    • tracks a belief distribution
    • can deal with multiple team/players/draws
  • Every Xbox 360 Live game uses TrueSkill™ ranking & matchmaking!
overview21
Overview
  • Motivation: Ranking in Video Games
  • Bayesian Player Ranking: TrueSkill™
    • Skill Belief, Likelihood, Update Equation
    • Applications in Online Gaming
    • Numerical Results on Halo 2 data
  • Ranking Moves: Computer Go
  • Conclusion
liberty fast pattern based computer go
Liberty: Fast Pattern Based Computer Go
  • Go: Simple rules yet very complex game.
  • Computer Go:
    • Infancy (best programs at weak amateur level).
    • Problems: Evaluation and Branching factor (≈250).
    • New grand challenge of AI (replacing Chess).
  • Idea: Learning good moves from expert play.
  • Applications:
    • Reduce branching factor for search.
    • Fast pattern based Go engine.
from local patterns to probability of moves

Pattern Urgency Table

σ

Belief in Urgency

μ

10

15

20

25

30

35

40

Urgency

From Local Patterns to Probability of Moves

=25,  = 5.2

Not in database!

Black to move

harvesting and learning
Harvesting and Learning
  • Two processes:
      • Harvesting patterns.
      • Ranking patterns.
  • We learn the value of these patterns using a modified Bayesian TrueSkill™ ranking system.
  • Partial ranking from every expert move:
    • Move made wins over any other move available on the board.
    • Nothing is known about the ranking of the un-played moves.
better than state of the art
Better than State-of-the-Art

1

Liberty (120K games)

Liberty (20K games)

Liberty (20K games)

Werf et al. (2002)

0.8

0.6

55% in top 5

cumulative probability

0.4

32% top

0.2

0

1

5

10

15

20

25

30

expert move rank

bayesian ranking for go
Bayesian Ranking for Go
  • Bayesian ranking makes full use of the information available from expert moves.
  • Simple features used in the approach already beats state-of-the-art prediction methods.
  • Approach is ideal for server-side Go AI:
    • Very fast at move selection time.
    • Large memory footprint.
  • Planned extension to 1,000,000 game records and context-aware patterns.
conclusions
Conclusions
  • Bayesian Ranking is a powerful technique
    • TrueSkill™generalises ELO and has a large influence on the online gaming experience of Xbox gamers.
    • Provides a principled and efficient way for learning the value of local patterns in the game of Go.