Bayesian ranking from xbox live to computer go
Download
1 / 30

Bayesian Ranking: From Xbox Live to Computer Go - PowerPoint PPT Presentation


  • 527 Views
  • Uploaded on

Bayesian Ranking: From Xbox Live to Computer Go Ralf Herbrich and Thore Graepel Overview Motivation: Ranking in Video Games Bayesian Player Ranking: TrueSkill™ Skill Belief, Likelihood, Update Equation Applications in Online Gaming Numerical Results on Halo 2 data

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Bayesian Ranking: From Xbox Live to Computer Go' - benjamin


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Bayesian ranking from xbox live to computer go l.jpg

Bayesian Ranking:From Xbox Live to Computer Go

Ralf Herbrich and Thore Graepel


Overview l.jpg
Overview

  • Motivation: Ranking in Video Games

  • Bayesian Player Ranking: TrueSkill™

    • Skill Belief, Likelihood, Update Equation

    • Applications in Online Gaming

    • Numerical Results on Halo 2 data

  • Ranking Moves: Computer Go

  • Conclusion


Motivation l.jpg
Motivation

  • Microsoft is the leader in online video gaming (Xbox Live).

    • Centralised game-independent service.

    • Gamercard, Achievements, TrueSkill™, etc.

  • What makes playing online games fun?

    • Good network connection (Broadband).

    • Seamless setup (Xbox Live).

    • Competitive matches (Ranking!).


Ranking in video games l.jpg
Ranking in Video Games

  • Problem Setting:

    • k teams of n1,…,nk many players compete.

    • The outcome is a ranking between the teams (including draws).

  • Questions:

    • Skill si of each player such that

    • Global ranking between all players.

    • High quality of match between k teams.


Overview5 l.jpg
Overview

  • Motivation: Ranking in Video Games

  • Bayesian Player Ranking: TrueSkill™

    • Skill Belief, Likelihood, Update Equation

    • Applications in Online Gaming

    • Numerical Results on Halo 2 data

  • Ranking Moves: Computer Go

  • Conclusion


The bayesian approach l.jpg
The Bayesian Approach

  • Classical logic deals with certain statements.

  • In the real world, uncertainty is abundant.

  • Degree of Belief (logic: 0% or 100%).

“Player i’s skill is between 35 and 40” and

“Player j’s skill is between 30 and 35”

“Player i won against Player j”

Bayesian Approach = Probability for Logic under Uncertainty

P(A|B) = P(B|A) P(A) / P(B)

Posterior

Prior

Likelihood

Evidence


Overview7 l.jpg
Overview

  • Motivation: Ranking in Video Games

  • Bayesian Player Ranking: TrueSkill™

    • Skill Belief, Likelihood, Update Equation

    • Applications in Online Gaming

    • Numerical Results on Halo 2 data

  • Ranking Moves: Computer Go

  • Conclusion


Trueskill skill belief l.jpg
TrueSkill™: Skill Belief

  • Track two numbers per player:

    • μ = Average skill of player

    • σ = Uncertainty about skill of player

  • Benefits:

    • Faster Skill Learning

    • Better Matchmaking

    • Accurate Prediction of Game Outcomes

σ

Belief in Skill Level

μ

10

15

20

25

30

35

40

Skill Level


Trueskill likelihood l.jpg
TrueSkill™: Likelihood

  • Likelihood:

  • Game outcomes are all permutations including draws between pairs of teams.

  • Latent performance model for players:

P(game outcome|s1,…,sn)=

P(ti’s are in game outcome order| s1,…,sn)

  • Team performance, ti, is sum of players’ performances in the team.


Likelihood example two players l.jpg
Likelihood Example: Two Players

7

0.2

Player 2 wins

0.18

6

Players 1 and 2 draw

0.16

5

0.14

0.12

4

Performance of Player 2

Probability density

0.1

3

0.08

0.06

2

Players 1 and 2 draw

0.04

Player 1 wins

1

0.02

Player 2 wins

Player 1 wins

0

0

0

1

2

3

4

5

6

7

-8

-6

-4

-2

0

2

4

Performance of Player 1

- x

x

1

2


Trueskill skill updates l.jpg

3rd Place

1st Place

2nd Place

TrueSkill™: Skill Updates

Game Outcome

Belief in Skill Level

0

10

20

30

40

50


Overview12 l.jpg
Overview

  • Motivation: Ranking in Video Games

  • Bayesian Player Ranking: TrueSkill™

    • Skill Belief, Likelihood, Update Equation

    • Applications in Online Gaming

    • Numerical Results on Halo 2 data

  • Ranking Moves: Computer Go

  • Conclusion


The true skill system applications l.jpg
The True Skill System: Applications

  • Leaderboard:

    • (Conservative) skill estimate : μ- 3·σ

  • Matchmaking:

    • Competitive game = Fun game!

    • Match quality = Probability of a draw

  • Team Balancing:

    • Maximise match quality by greedy search.


Trueskill matchmaking l.jpg

0.1

0.1

0.08

0.08

0.06

0.06

Probability density

Probability density

0.04

0.04

0.02

0.02

0

0

-5

-5

0

0

5

5

10

10

15

15

20

20

25

25

30

30

Skill

Skill

0.8

0.7

0.6

0.5

Probability density

0.4

0.3

0.2

0.1

0

-5

0

5

10

15

20

25

30

Skill

TrueSkill™: Matchmaking

Lobby

Possible Matches

?


Overview15 l.jpg
Overview

  • Motivation: Ranking in Video Games

  • Bayesian Player Ranking: TrueSkill™

    • Skill Belief, Likelihood, Update Equation

    • Applications in Online Gaming

    • Numerical Results on Halo 2 data

  • Ranking Moves: Computer Go

  • Conclusion


Results halo 2 multiplayer beta l.jpg
Results: Halo 2 Multiplayer Beta

  • 5 different “hoppers”

    • Free-For-All: 60261 games (5946 players)

    • 1 vs. 1: 6240 games (1672 players)

  • 5 maps, 3 different game variants.

  • Matchmaking was relaxed to level gap 9.

  • Parameters in all experiments:

    • Performance variation factor: 60%

    • Draw Probability: 5%

    • Dynamics variation factor: 2%


Free for all char vs sqlwildman l.jpg
Free-For-All: char vs. SQLwildman?

40

35

30

25

Level

20

15

)

char (

TrueSkill

10

)

SQLwildman (

TrueSkill

char (Halo 2)

5

SQLwildman (Halo 2)

0

0

100

200

300

400

Number of games played


Free for all char vs sqlwildman18 l.jpg
Free-For-All: char vs. SQLwildman?

100%

char wins

SQLwildman wins

80%

Both players draw

60%

Winning probability

40%

20%

5/8 games won by char

0%

0

100

200

300

400

500

Number of games played


Near normal level distribution 650 000 players l.jpg
Near Normal Level Distribution – 650,000 Players

TrueSkill™ Analysis of Halo 2 (Nov. – Dec. 2004)

50

40

30

Level

20

10

1

3

2.5

2

1.5

1

0.5

0

Level Occupancy

4

x 10

x 10


Trueskill l.jpg
TrueSkill™

  • Skill based ranking instead of experience based ranking for better matchmaking.

  • TrueSkill™ system is

    • a generalisation of ELO

    • tracks a belief distribution

    • can deal with multiple team/players/draws

  • Every Xbox 360 Live game uses TrueSkill™ ranking & matchmaking!


Overview21 l.jpg
Overview

  • Motivation: Ranking in Video Games

  • Bayesian Player Ranking: TrueSkill™

    • Skill Belief, Likelihood, Update Equation

    • Applications in Online Gaming

    • Numerical Results on Halo 2 data

  • Ranking Moves: Computer Go

  • Conclusion


Liberty fast pattern based computer go l.jpg
Liberty: Fast Pattern Based Computer Go

  • Go: Simple rules yet very complex game.

  • Computer Go:

    • Infancy (best programs at weak amateur level).

    • Problems: Evaluation and Branching factor (≈250).

    • New grand challenge of AI (replacing Chess).

  • Idea: Learning good moves from expert play.

  • Applications:

    • Reduce branching factor for search.

    • Fast pattern based Go engine.


From local patterns to probability of moves l.jpg

Pattern Urgency Table

σ

Belief in Urgency

μ

10

15

20

25

30

35

40

Urgency

From Local Patterns to Probability of Moves

=25,  = 5.2

Not in database!

Black to move


Harvesting and learning l.jpg
Harvesting and Learning

  • Two processes:

    • Harvesting patterns.

    • Ranking patterns.

  • We learn the value of these patterns using a modified Bayesian TrueSkill™ ranking system.

  • Partial ranking from every expert move:

    • Move made wins over any other move available on the board.

    • Nothing is known about the ranking of the un-played moves.


  • Better than state of the art l.jpg
    Better than State-of-the-Art

    1

    Liberty (120K games)

    Liberty (20K games)

    Liberty (20K games)

    Werf et al. (2002)

    0.8

    0.6

    55% in top 5

    cumulative probability

    0.4

    32% top

    0.2

    0

    1

    5

    10

    15

    20

    25

    30

    expert move rank



    Bigger patterns early l.jpg

    Later Stage

    Early Stage

    Bigger Patterns Early


    Bigger patterns better predictions l.jpg

    Error decreases for increasing pattern size

    Bigger Patterns  Better Predictions


    Bayesian ranking for go l.jpg
    Bayesian Ranking for Go

    • Bayesian ranking makes full use of the information available from expert moves.

    • Simple features used in the approach already beats state-of-the-art prediction methods.

    • Approach is ideal for server-side Go AI:

      • Very fast at move selection time.

      • Large memory footprint.

    • Planned extension to 1,000,000 game records and context-aware patterns.


    Conclusions l.jpg
    Conclusions

    • Bayesian Ranking is a powerful technique

      • TrueSkill™generalises ELO and has a large influence on the online gaming experience of Xbox gamers.

      • Provides a principled and efficient way for learning the value of local patterns in the game of Go.


    ad