evaluation of human robot interaction in the nist reference search and rescue test arenas l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Evaluation of Human-Robot Interaction in the NIST Reference Search and Rescue Test Arenas PowerPoint Presentation
Download Presentation
Evaluation of Human-Robot Interaction in the NIST Reference Search and Rescue Test Arenas

Loading in 2 Seconds...

play fullscreen
1 / 14

Evaluation of Human-Robot Interaction in the NIST Reference Search and Rescue Test Arenas - PowerPoint PPT Presentation


  • 318 Views
  • Uploaded on

Evaluation of Human-Robot Interaction in the NIST Reference Search and Rescue Test Arenas Jean Scholtz Brian Antonishek Jeff Young Outline of Talk NIST Reference Search and Rescue Test Arenas Human-Robot Interaction (HRI) Challenges Case studies from USAR Competitions Methods Metrics

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Evaluation of Human-Robot Interaction in the NIST Reference Search and Rescue Test Arenas' - paul


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
evaluation of human robot interaction in the nist reference search and rescue test arenas
Evaluation of Human-Robot Interaction in the NIST Reference Search and Rescue Test Arenas

Jean Scholtz

Brian Antonishek

Jeff Young

Permis 2004

outline of talk
Outline of Talk
  • NIST Reference Search and Rescue Test Arenas
  • Human-Robot Interaction (HRI)
    • Challenges
  • Case studies from USAR Competitions
    • Methods
    • Metrics
  • Guidelines
  • Recommendations

Permis 2004

nist usar reference test arenas
NIST USAR Reference Test Arenas
  • Provides a repeatable way to evaluate a search and rescue system (robot + operator + human-robot interaction)
  • Score depends on
    • Number of victims located
    • Difficulty of arena in which victims are located
    • Accuracy of victim location
    • Accuracy of victim assessment
    • Penalties incurred in locating victims
  • Autonomy levels, HRI, platform mobility, sensor packages are left to the participants’ discretion

Permis 2004

usar competitions
USAR Competitions
  • Success depends on:
    • Mobility of platforms
    • Skill of the operator
    • Affordances, ease of use of the user interface
    • Sensor packages
    • Communications
    • System robustness
  • Currently we do not evaluate the various components separately but use the overall system performance in determining the winners of the competitions

Permis 2004

hri evaluation
HRI Evaluation
  • Challenge:
    • To determine the contribution of the human-robot interaction design to the overall performance
    • And in the process, to develop both metrics for HRI and guidelines for the design
  • HRI
    • More than just the visual interface
    • Includes the design of the interaction dialogue between the robot(s) and the operator(s)

Permis 2004

data collection for hri at usar competitions
Data Collection for HRI at USAR Competitions
  • Have collected data from 6 major competitions since 2002
  • Offers wide range of HRI designs
  • Operators are robotics researchers, hence best case
  • Limited in our ability to interview/ control conditions
  • Data collected include:
    • Video of robot in arena (ground truth)
    • Video of what operator sees
    • Video of operator actions (in some cases)
    • Maps of coverage of arenas

Permis 2004

data analysis
Data Analysis
  • Hypothesis: Systems that are able to cover more of the arena should be more successful
  • Analyzed % of time spent in
    • Navigation
    • Victim identification
    • Logistics
    • Failures
  • Looked for correlations between coverage, where time was spent, success in competition
    • More time spent navigating, more victims found
    • Correlation with coverage is difficult to compute; time between arenas, difficulty of arenas; difficulty in assessing

Permis 2004

data analysis cont
Data Analysis, cont.
  • Human-robot awareness
    • The knowledge the human has of the location, status, and behavior of the robot
  • Indirect measures necessary
  • Used Critical Incident analysis
    • Global navigation
    • Local navigation
    • Obstacle evaluation
    • Vehicle state
    • Victim ID

Permis 2004

data analysis cont11
Data Analysis, cont.
  • What contributed to fewer critical incidents?
    • Local navigation
      • Frame of reference provided – overhead camera; 2 degree of freedom camera used to see wheels of robot in relation to environment
    • Obstacle encounters
      • Front and rear cameras
      • Ability to move robot and camera at same time
    • Vehicle state
      • Top down view of robot may have helped
      • Audio also helped (but noise in arena was excessive at times)
  • How did this correlate with success in competition?
    • Obstacle encounters were the best predictor but too little data to generalize

Permis 2004

data analysis cont12
Data Analysis, cont.
  • Robocup 2004
    • Allowed us to compare overhead camera use with automatic mapping

Permis 2004

guidelines for hri design
Guidelines for HRI Design
  • Information for effective situation awareness should include:
    • a frame of reference to determine the position of the robot relative to the surrounding environment
    • indicators of vehicle state, such as pitch, roll, traction indicators, indicators of sensor status, and camera positions relative to the robot body.
    • a map to provide global navigation information
  • Minimize the number of windows provided to the operator.
  • Provide a fused view of sensor information.
  • Support multiple robot operators in a single display.
  • Provide help from the robot in determining what mode of autonomy is most useful.

Permis 2004

conclusions recommendations
Conclusions/ Recommendations
  • Awareness assessment provides insights about information needed by operators to avoid critical incidents
  • Indirect evaluation is problematic
    • Takes lots of resources to evaluate; hence cannot produce feedback for robotics researchers in timely fashion
  • Potential solution for more direct assessment
    • “compulsory figures” evaluation for USAR competitions
    • Place robots in a number of situations and measure time/accuracy needed for operators to assess and describe the situation
    • Eliminates execution of the situation (operator skill, platform mobility)
    • Could also provide a benchmark system for comparison

Permis 2004