slide1 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Graham Kendall Automated Scheduling, Optimisation and Planning Research Group (ASAP) PowerPoint Presentation
Download Presentation
Graham Kendall Automated Scheduling, Optimisation and Planning Research Group (ASAP)

Loading in 2 Seconds...

play fullscreen
1 / 90

Graham Kendall Automated Scheduling, Optimisation and Planning Research Group (ASAP) - PowerPoint PPT Presentation


  • 182 Views
  • Uploaded on

Graham Kendall Automated Scheduling, Optimisation and Planning Research Group (ASAP). MIU, July 2004. Contents. Checkers: Why was it considered “beaten”? Two approaches to Checkers Poker (if time). 1959. Arthur Samuel started to look at Checkers 2

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

Graham Kendall Automated Scheduling, Optimisation and Planning Research Group (ASAP)


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Graham KendallAutomated Scheduling, Optimisation and Planning Research Group (ASAP)

MIU, July 2004

slide2

Contents

  • Checkers: Why was it considered “beaten”?
  • Two approaches to Checkers
  • Poker (if time)
slide3

1959. Arthur Samuel started to look at Checkers2

    • The determination of weights through self-play
    • 39 Features
    • Included look-ahead via mini-max

2 Samuel A. Some studies in machine learning using the game of checkers. IBM J. Res. Develop. 3 (1959), 210-229

slide4

Samuels’s program defeated Robert Nealy, although the victory is surrounded in controversy

    • Was he state champion?
    • Did he lose the game or did Samuel win?
slide5

Checkers Starting Position

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

slide6

Checkers Moves

1

2

4

3

5

6

8

7

Pieces can only move diagonally forward

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

slide7

Jumps are forced

Checkers Forced Jumps

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

slide8

K

1

Red (Samuel’s Program)

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

Getting to the back row gives a King

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide9

Red (Samuel’s Program)

1

2

4

3

5

6

8

7

Forced Jump

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide10

Red (Samuel’s Program)

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide11

Strong

(Try to keep)

Trapped

Only advance to Square 28

Red (Samuel’s Program)

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide12

Red (Samuel’s Program)

1

2

4

3

What Move?

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide13

Red (Samuel’s Program)

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide14

Red (Samuel’s Program)

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide15

This was a very poor move.

    • It allowed Samual to retain his “Triangle of Oreo”
    • AND.. By moving his checker from 19 to 24 it guaranteed Samuel a King
    • This questioned how strong a player Nealy really was
slide16

Red (Samuel’s Program)

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide17

This was a very poor move.

    • It allowed Samual to retain his “Triangle of Oreo”
    • AND.. By moving his checker from 19 to 24 it guaranteed Samuel a King
    • This questioned how strong a player Nealy really was
slide18

Red (Samuel’s Program)

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide19

K

23

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide20

K

23

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

24

21

22

25

28

26

What Move (5, 13 or 16)?

27

29

30

31

32

White (Nealey)

slide21

K

23

Computers & Game Playing : A Potted History

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide22

16-12 then 5-1, Chinook said would be a draw

K

23

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide23

K

23

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide24

K

23

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide25

K

23

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide26

K

23

Computers & Game Playing : A Potted History

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide27

K

23

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide28

K

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

This checker is lost

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide29

K

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide30

K

K

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

What Move (3, 6 or 19)?

29

30

31

32

White (Nealey)

slide31

K

K

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

This checker could run (to 10) but..

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide32

K

K

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide33

K

K

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide34

K

K

Red (Samuel’s Program) : After Move 25

1

2

Forced Jump

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide35

K

K

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide36

K

K

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide37

Victory

K

Red (Samuel’s Program) : After Move 25

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

White (Nealey)

slide38

Two Mistakes by Nealy

    • Allowing Samuel to get a King
    • Playing a move that led to defeat when there was a draw available
slide39

The next year a six match rematch was won by Nealy 5-1.

  • Three years later (1966) the two world championship challengers (Walter Hellman and Derek Oldbury) played four games each against Samuel’s program. They won every game.
slide40

Checkers

    • Chinook
    • Blondie 24 (aka Anaconda)
slide41

Types of Games

  • Perfect
    • Each Player has complete knowledge of the game state
    • Usually only two players, who take alternate turns
    • Examples include Chess, Checkers, Awari, Connect-Four, Go, Othello
slide42

Types of Games

  • Imperfect
    • Some of the game state is hidden
    • Examples include Poker, Cribbage, Bridge
slide43

Types of Games

  • Games with an element of chance
    • The game moves have some stochastic element
    • For example, Backgammon
slide44

Types of Games

6 Jaap van den Herik H., Uiterwijk and van Rijswijck J. Games Solved: Now and in the future. Artificial Intelligence 134 (2002) 277-311

slide45

Case Study 1: Checkers

  • Samuel’s work, perhaps, restricted the research into Checkers until 1989 when Jonathan Schaeffer began working on Chinook
  • He had two aims
    • To develop the worlds best checkers player
    • To “solve” the game of checkers
slide46

Case Study 1: Checkers

  • Chinook, at its heart, had an evaluation function
    • Piece count (+30% for a King)
    • Runaway checker
    • “Dog Hole”
  • The weights were hand-tuned
slide47

Case Study 1: Checkers

  • Opening game database from published work (with corrections they found)
  • Initially 4000 openings, leading to an eventual 40,000
  • “Cooks” – innovative lines of play that could surprise an opponent
  • The aim was to take opponents into unknown territory
slide48

Case Study 1: Checkers

  • Endgame database: Started writing in May 1989
  • The 8-piece endgame database finished on February 20th 1994
slide52

Case Study 1: Checkers

  • With a 4-piece database Chinook won the 1989 Computer Olympiad
  • In the 1990 US National Checkers Championship Chinook was using a 6-piece database.
  • It came second, to Marion Tinsley, defeating Don Lafferty on the way who was regarded at the worlds second best player.
slide53

Case Study 1: Checkers

  • Marion Tinsley
  • Held the world championship from 1951 to 1994
  • Before playing Chinook, Tinsley only lost 4 competitive games (no matches)
slide54

Case Study 1: Checkers

  • The winner of the US Championship has the right to play for the world championship. Finishing second (with Tinsley first) entitled Chinook to play for the world championship
  • The American Checkers Federation (ACF) and the European Draughts Association (ADF) refused to let a machine compete for the title.
slide55

Case Study 1: Checkers

  • In protest, Tinsley resigned
  • The ACF and EDF, created a new world championship, “man versus machine” and named Tinsley as the world champion.
  • At this time Tinsley was rated at 2812, Chinook was rated at 2706
slide56

Case Study 1: Checkers

  • The match took place 17-29 Aug 1992.
  • The $300,000 computer used in the tournament ran at about half the speed of a 1GHz PC
  • The match finished 4-2 in favour of Tinsley (with 34 draws)
slide57

Case Study 1: Checkers

  • A 32 game rematch was held in 1994
  • 8-piece end game
  • Processors four times as fast (resulted in a factor of 2 speed up due to more complex evaluation function and the overhead of parallel processing)
  • Opening book of 40,000 moves
  • In preparation Chinook no losses in 94 games against Grandmasters
slide58

Case Study 1: Checkers

  • Six games in (1-1, with 4 draws) Tinsley resigned for health reasons. His symptoms were later diagnosed as pancreatic cancer.
  • Tinsley died on 3rd April 1995 (aged 68). Undoubtedly the best player ever to have lived
  • Chinook was crowned the man versus machine champion. The first automated game player to have achieved this.
  • A 20-match with Don Lafferty resulted in a draw (1-1, with 18 draws)
slide59

Case Study 1: Checkers

…defeating the world who

had held the title

for 40 years

Opening Game Database

(40,000) moves

Hand Crafted

Evaluation

Function

(a/b search)

Schaeffer J. One Jump Ahead:

Challenging Human Supremacy

in checkers, Springer, 1997

Won the World (Man

Versus Machine)

Championship

in 1994…

Marion Tinsley lost his 5th,

6th and 7th games to

Chinook

End Game Database

(8-pieces)

slide60

Case Study 2: Anaconda

  • Project started in the summer of 1998, following a conversation between David Fogel and Kumar Chellapilla
  • It was greatly influenced by the recent defeat of Kasparov by Deep Blue
  • Chess was seen as too complex so “draughts” was chosen instead
  • The aim is to evolve a player – rather than build in knowledge
slide61

Case Study 2: Anaconda

  • Reject inputting into a neural network what humans think might be important
  • Reject inputting any direct knowledge into the program
  • Reject trying to optimise the weights for an evaluation function
slide62

Case Study 2: Anaconda

  • The Gedanken Experiment
  • I offer to sit down and play a game with you. We sit across an 8x8 board and I tell you the legal moves
  • We play five games, only then do I say “You got 7 points.”I don’t tell you if you won or lost
  • We play another five games and I say “You got 5 points”
  • You only know “higher is better”
slide63

Case Study 2: Anaconda

  • The Gedanken Experiment
  • How long would it take you to become an expert at this game?
  • We cannot conduct this experiment but we can get a computer to do it
slide64

Case Study 2: Anaconda

  • Samuel’s Challenge: “Can we design a program that would invent its own features in a game of checkers and learn how to play, even up to the level of an expert?”
slide65

Case Study 2: Anaconda

  • Newell’s Challenge: “Could the program learn just by playing games against itself and receiving feedback, not after each game, but only after a series of games, even to the point where the program wouldn’t even know which programs had been won or lost?”
  • Newell (and Minsky)7 believed that this was not possible, arguing that the way forward was to solve the credit assignment problem.

7 Minsky M. Steps Towards Artificial Intelligence. Proceedings of the IRE, 1961, 8-30

slide66

.

.

.

.

.

.

.

.

.

Later changed to

an explicit piece differential

Case Study 2: Anaconda

HL21

HL11

I1

Evaluation used for MiniMax

O

I32

HL210

HL140

# weights=1741

slide67

K

Case Study 2: Anaconda

1

2

4

3

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

slide68

+1

+1

+1

-K

-1

-1

Case Study 2: Anaconda

1

7

8

14

25

27

All other neurons have an value of zero

slide69

Case Study 2: Anaconda

  • Algorithm
  • Initialise 30 Networks
  • Each network played 5 games as red against random opponents
  • Games were played to completion or until 100 moves had been made (a draw)
  • +2 for a win, 0 for a draw, -1 for a loss
  • 15 best performing networks were saved for the next generation and copies were mutated
slide70

Case Study 2: Anaconda

  • Observations
  • The points for a win, lose draw were set such that wins were encouraged. No experimentation with different values were tried
  • Players could play a different number of games. This was, purposefully, not taken into account
  • Mutation was carried out using an evolutionary strategy
slide71

Case Study 2: Anaconda

  • After 10 Generations
  • After 10 generations the best neural network was able to beat both its creators and a simple (undergraduate project) program which, by the authors admission was “weak”
  • Note: 400MHz PC
slide73

Case Study 2: Anaconda

  • After 100 Generations
  • Playing on zone.com
  • Initial rating = 1600
  • Beat a player ranked at 1800 but lost against a player in the mid 1900’s
  • After 10 games their ranking had improved to 1680. After 100 games it had improved to 1750
  • Typically a 6-ply search but often 8-ply
slide74

Case Study 2: Anaconda

  • Observations
  • The highest rating it achieved was 1825
  • The evolved King value was 1.4, which agrees with perceived wisdom that a king is worth about 1.5 of a checker
  • In 100 generations a neural network had been created that was competitive with humans
  • It surpassed Samuel’s program
  • The challenge set by Newell had been met
slide75

Case Study 2: Anaconda

  • The Next Development
  • Alpha-Beta Pruning introduced and evolved over 250 generations
  • Over a series of games, Obi_WanThe Jedi defeated a player rated at 2134 (48 out of 40,000 registered) and also beat a player rated 2207 (ranked 18)
  • Final rating was 1902 (taking into account the different orderings of the games)
slide76

Case Study 2: Anaconda

  • The Next Development
  • Spatial nature of the board was introduced as at the moment it just “saw” the board as a vector of length 32
slide77

Case Study 2: Anaconda

1

2

4

3

36 3x3 Overlapping squares

5

6

8

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

slide78

Case Study 2: Anaconda

1

2

4

3

5

6

8

25 4x4 Overlapping squares

7

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

slide79

Case Study 2: Anaconda

1

2

4

3

5

6

8

7

16 5x5 Overlapping squares

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

slide80

Case Study 2: Anaconda

1

2

4

3

5

6

8

7

9 6x6 Overlapping squares

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

slide81

Case Study 2: Anaconda

1

2

4

3

5

6

8

7

4 7x7 Overlapping squares

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

slide82

Case Study 2: Anaconda

1

2

4

3

5

6

8

7

1 8x8 Overlapping squares

11

9

10

12

15

16

14

13

17

18

19

20

23

24

21

22

25

28

26

27

29

30

31

32

slide83

Case Study 2: Anaconda

  • The Next Development
  • 36+25+16+9+4+1 = 91 inputs
  • 5,046 weights
slide84

.

.

.

Case Study 2: Anaconda

HL1

(91 nodes)

HL2

(40 nodes)

HL3

(10 nodes)

36 3x3

I1

25 4x4

16 5x5

O

9 6x6

I32

4 7x7

1 8x8

Sum of 32 Board Inputs

# weights=5046

slide85

Case Study 2: Anaconda

  • 2 months and 230 generations later!!
  • After 100 games the rating was 1929
  • A 27 point increase over the previous network. Nice but not decisive
  • Maybe it was due to there being three times more weights but the training period was the same?
slide86

Case Study 2: Anaconda

  • 6 months and 840 generations later!!
  • After 165 games it was rated at 2045.85 (sd 33.94)
  • Rated in the top 500 at zone.com (of the 120,000 players now registered)
  • That is better than 99.61% of the players
slide87

Case Study 2: Anaconda

  • Playing Chinook8
  • In a ten match series against Chinnok novice level it had two wins, two losses and 4 draws

8 Fogel D. B. and Chellapilla K. Verifying Anaconda’s expert rating by competing against Chinook: experiments in co-evolving a neural checkers player, Neurocomputing 42 (2002) 69-86

slide88

Case Study 2: Anaconda

  • Blondie
  • The neural checkers player went through a number of names
    • David0111
    • Anaconda
    • Blondie24
slide90

Case Study 2: Anaconda

  • References
  • Fogel D.B. Blondie24: Playing at the Edge of AI, Morgan Kaufmann, 2002
  • Fogel D. B. and Chellapilla K. Verifying Anaconda’s expert rating by competing against Chinook: experiments in co-evolving a neural checkers player, Neurocomputing 42 (2002) 69-86
  • Chellapilla K and Fogel D. B. Evolving neural networks to play checkers without expert knowledge. IEEE Trans. Neural Networks 10(6):1382-1391, 1999
  • Chellapilla K and Fogel D. B.Evolution, neural networks, games, and intelligence, Proc. IEEE 87(9):1471-1496. 1999
  • Chellapilla K and Fogel D. B. Evolving an expert checkers playing program without relying on human expertise. IEEE Trans. Evolutionary Computation, 2001
  • Chellapilla K and Fogel D. B. Anaconda Defeats Hoyle 6-0: A Case Study Competing an Evolved Checkers Program Against Commercially Available Software. Proc. Of CEC 2000:857-863