reinforcement learning for the soccer dribbling task n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Reinforcement Learning for the Soccer Dribbling Task PowerPoint Presentation
Download Presentation
Reinforcement Learning for the Soccer Dribbling Task

Loading in 2 Seconds...

play fullscreen
1 / 22

Reinforcement Learning for the Soccer Dribbling Task - PowerPoint PPT Presentation


  • 177 Views
  • Uploaded on

Reinforcement Learning for the Soccer Dribbling Task. Arthur Carvalho Renato Oliveira. Introduction. RoboCup soccer simulation Scoring “A Data Mining Approach to Solve the Goal Scoring Problem” Passing “A New Passing Strategy Based on Q-Learning Algorithm in  RoboCup ” Dribbling ?.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Reinforcement Learning for the Soccer Dribbling Task' - kizzy


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
introduction
Introduction
  • RoboCup soccer simulation
    • Scoring
      • “A Data Mining Approach to Solve the Goal Scoring Problem”
    • Passing
      • “A New Passing Strategy Based on Q-Learning Algorithm in RoboCup”
    • Dribbling
      • ?
outline
Outline
  • The soccer dribbling task as a RL problem
  • RL solution
  • Experiments
  • Conclusion
the soccer dribbling task as a rl problem
The Soccer Dribbling Task as a RL Problem
  • Coach
    • Setting positions
      • Dribbler is placed in the center-left region together with the ball
      • Adversary is placed in a random position
    • Manage the play
      • Adversary wins when he gains possession or when the ball goes out of the field
      • Dribbler wins when he crosses the field with the ball
the soccer dribbling task as a rl problem1
The Soccer Dribbling Task as a RL Problem
  • When an episode ends, the coach starts a new one
  • RoboCup soccer simulator operates in discrete time steps
  • Episodic reinforcement-learning framework
the soccer dribbling task as a rl problem2
The Soccer Dribbling Task as a RL Problem
  • Actions
    • HoldBall()
    • Dribble(α, k)
      • Dribble(30, 5), Dribble(330, 5), Dribble(0, 5), Dribble(0, 10)
      • The dribbler can kick the ball forward (strongly and weakly), diagonally upward, and diagonally downward.
outline1
Outline
  • The soccer dribbling task as a RL problem
  • RL solution
  • Experiments
  • Conclusion
rl solution
RL Solution
  • Sarsa + CMAC
  • Sarsa estimates the action-value function for the current policy and for all state-action pairs
rl solution1
RL Solution
  • CMAC
    • Partitioning the state space into several receptive fields (hyper-rectangles)
      • Each one is associated with a weight
    • Multiple partitions of the state space (layers) are usually used
    • The CMAC’s response to a given input is equal to the sum of the weights of the excited receptive fields
outline2
Outline
  • The soccer dribbling task as a RL problem
  • RL solution
  • Experiments
  • Conclusion
experiments
Experiments
  • RoboCup soccer simulator
    • Version 14.0.3, protocol 9.3
  • 20m x 20m region
  • -greedy policy
    • = 0.01
  • Parameters
    • Learning rate:
    • Discount rate:
experiments1
Experiments
  • Adversary
    • Fixed policy
    • It computes a near-optimal interception point (UvATrilearn 2003 team)
  • Two phases
    • Training
    • Testing
experiments2
Experiments
  • Training Phase: 5 independent runs, each one lasting 50,000 episodes

53%

experiments3
Experiments
  • Qualitatively
    • Rule #1
experiments4
Experiments
  • Qualitatively
    • Rule #2
experiments5
Experiments
  • Testing:
    • 10,000 episodes
    • Weights with the highest success rate
    • The weights were not updated
    • , i.e., the dribbler always selected the action with the highest estimated value
    • The dribbler won 5,795 episodes (58%)
outline3
Outline
  • The soccer dribbling task as a RL problem
  • RL solution
  • Experiments
  • Conclusion
conclusion
Conclusion
  • Dribble
    • Soccer dribbling task
    • Reinforcement learning solution
  • Benchmark
  • Start point for dribbling tasks in other sports games
    • E.g., hockey, basketball, and football
thank you
Thank you!

Source code available at:

http://sites.google.com/site/soccerdribbling

Arthur Carvalho Renato Oliveira

a3carval@cs.uwaterloo.ca rmo@cin.ufpe.br