A Scene Learning and Recognition Framework for RoboCup

1. A Scene Learning and Recognition Framework for RoboCup M.A.Sc Thesis Defense Kevin Lam 100215552 Carleton University September 6, 2005

2. Presentation Overview • RoboCup Overview • Objective and Motivation • Contributions • Methodology • Scene Representation • Scene Matching • What We Built • Experimental Results • Future Work

3. RoboCup Overview

4. Objective & Motivation Can an agent learn by observing and imitating another, with little or no intervention from designers or domain experts? • Based on Paul Marlow’s work • Observe other agents instead of humans • Lots of RoboCup teams to choose from (subject to restrictions) • Advantages: reduce development time, access existing body of knowledge

5. Contributions • An extensible framework for research in RoboCup agent imitation • a conversion process (raw logs  higher-level representation) • customizable scene recognition and matching using k-nearest-neighbor • a RoboCup client agent based on this scene recognition algorithm • Semi-automated imitation results • Student contributions (SYSC 5103, Fall 2004) Can an agent learn by observing and imitating another, with little or no intervention from designers or domain experts? … Yes! (with caveats)

6. Current Approaches • Most agent development is: • Hard coded or scripted (e.g. Krislet) • High-level behaviour descriptions (Steffens) • Supervised learning situations (Stone) • Some attempts at learning by observation • ILP rule inducer (Matsui) • Results not directly reused; complex rules, “OK” results • COTS data mining (Weka-based) • Problems: complex trees, hard to describe complex behaviours - tuning/pruning needed

7. Methodology • Model agent behaviour as function of inputs and output: y = ƒ(a, b, c) • Assumptions • Deterministic • Stateless and memory-less (no memory or internal status kept)

8. Methodology • Observation of Existing Agents • Scenario Representation • “Scene” spatial knowledge representation • Scenario Recognition • K-nearest-neighbor search • “Distance” metric definition • Scene and Action selection

9. (see 267 ((f c) 19.7 5 -0 -0) ((f l t) 79.8 28) (turn 85.0) (sense_body 312 (view_mode high normal) (see 312 ((f c b) 38.1 12) ((f b 0) 42.5 8) (dash 70.0) (see 993 ((p "Canada2" 1 goalie) 7.4 32 ) (dash 70.0) . . . Agent Observation • Logged messages describe: • Objects seen • Actions sent • Agent internal states

10. Scene Representation • A “scene” is a snapshot of space at a given time • Up to 6,000 time slices in typical game • In RoboCup, this means a list of objects: • Players • Ball • Goals • Lines • Flags Distance Direction Velocity Attributes (team, uniform number etc.)

11. Scene Representation  2723 in: “see …” out: “kick”

12. Discretized Representation • Can discretize scenes into segments • Size of “slices”  degree of generalization • Logical notions; consistent with simulator • Reduced storage (or better coverage) vs generalization

13. Scene Recognition “What should I do in this situation?” • Keep stored scenes from observed agent • Find best match between current situation and stored scenes • Use k-nearest-neighbor search “What did the observed agent do when faced with a situation like this?”

14. “Distance” Metric • Object Pairing • a  c, b  d • Separate by type • Continuous vs. Discrete Distance • Cosine Law (Continuous) • Euclidian (Cell-based)

15. Action Selection • Get k-nearest matching scene-action pairs • Only one action must be selected • Choices: • Random • First available • Weighted majority • Also attributes (direction, power)

16. What We Built • Agent based on Krislet • New Brain algorithm: • Load scene file at startup • When new info arrives, convert to scene • Compare with every stored scene • Pick the “best match” and reuse action • Validator (cross-validation testing)

17. Krislet-Scenes Architecture

18. “Distance” Calculation Random Selection Continuous Distance Object Calculation Discretized Distance Object Calculation Discretized Ball-Goal Calculation (student contribution) Action Selection Random Selection First Available Weighted Vote Vote with Randomness Selection Algorithms Object Matching • Simple heuristic (sorted by distance to player)

19. Experiments • Logged three RoboCup agents • Krislet, NewKrislet, CMUnited • Experimental Parameters • Distance calculation algorithms • Action selection algorithms • k-value  {1, 5, 15} • Object weights  {0, 1} • Discretization size fixed at (5, 3) • Quantitative (validation, real) vs Qualitative

20. Experimental Results Best Statistical Results Best Qualitative Results

21. Experimental Results • Best parameters: • Continuous seems to work better than discretization • k  5 with random selection, or k=1 • Object weighting is critical! • Can successfully imitate Krislet client (difficult for a human observer to distinguish) • Slightly less responsive, slower to “decide” • Imitates many aspects of NewKrislet Attacker • Copies only very basic behaviours of CMUnited

22. Limitations • Simplistic object matching algorithm • Need way to match detailed objects like players, flags • Works best on stateless, deterministic, reactive agents • Does not consider memory, field position, internal state • “Single-layered” logic approach • Performance (speed and coverage) • Limited parameter exploration in our tests • Not yet fully automated

23. Future Work • Application of a “minimum weight perfect matching” algorithm (David Tudino) • Hierarchical scene storage for improved search and storage performance • State detection and learning (e.g. Hidden Markov Model) • Pattern mining within scenes • Better qualitative evaluation metrics (needed for automation)

24. Conclusions • We contributed an extensible framework for research in RoboCup agent imitation • a conversion process (raw logs  higher-level representation) • customizable scene recognition and matching using k-nearest-neighbor • a RoboCup client agent based on this scene recognition algorithm • Encouraging imitation results (semi-automated) • Lots of direction for future work

25. Questions?

26. Krislet Behaviour • Simple decision tree logic • Turn and look for ball • Run toward ball • Turn and look for goal • Kick ball to goal • No concept of teams or strategies • Stateless • Deterministic

27. NewKrislet Behaviour • Implement strategies as state machine • “AntiSimpleton” prepackaged strategy • Attackers wait near center line until ball is near; then kicks the ball toward the goals • Defenders wait along the field; if the ball comes near, they kick it to the attacker and return to their position • Deterministic

28. CMUnited Behaviour • World Cup Championship Winner! • Layered-learning model • Passing, dribbling skills at low level • Strategies at higher level • Formations • Communications • Coach • Player Modeling

29. Stateless Behaviour? • Model agent as function f(a, b, c) = x • If inputs (a, b, c) usually results in x, the agent is probably stateless • If inputs (a, b, c) sometimes produces x, but other times produces y, there might be two states involved • Subject to probability modeling

30. Discretizing Scenes: Issues • Potential problems • Bias introduced • Boundary values/edging • Overfitting