Evaluation of human robot interaction in the nist reference search and rescue test arenas
Download
1 / 14

Evaluation of Human-Robot Interaction in the NIST Reference Search and Rescue Test Arenas - PowerPoint PPT Presentation


  • 291 Views
  • Updated On :

Evaluation of Human-Robot Interaction in the NIST Reference Search and Rescue Test Arenas Jean Scholtz Brian Antonishek Jeff Young Outline of Talk NIST Reference Search and Rescue Test Arenas Human-Robot Interaction (HRI) Challenges Case studies from USAR Competitions Methods Metrics

Related searches for Evaluation of Human-Robot Interaction in the NIST Reference Search and Rescue Test Arenas

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Evaluation of Human-Robot Interaction in the NIST Reference Search and Rescue Test Arenas' - paul


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Evaluation of human robot interaction in the nist reference search and rescue test arenas l.jpg
Evaluation of Human-Robot Interaction in the NIST Reference Search and Rescue Test Arenas

Jean Scholtz

Brian Antonishek

Jeff Young

Permis 2004


Outline of talk l.jpg
Outline of Talk Search and Rescue Test Arenas

  • NIST Reference Search and Rescue Test Arenas

  • Human-Robot Interaction (HRI)

    • Challenges

  • Case studies from USAR Competitions

    • Methods

    • Metrics

  • Guidelines

  • Recommendations

Permis 2004


Nist usar reference test arenas l.jpg
NIST USAR Reference Test Arenas Search and Rescue Test Arenas

  • Provides a repeatable way to evaluate a search and rescue system (robot + operator + human-robot interaction)

  • Score depends on

    • Number of victims located

    • Difficulty of arena in which victims are located

    • Accuracy of victim location

    • Accuracy of victim assessment

    • Penalties incurred in locating victims

  • Autonomy levels, HRI, platform mobility, sensor packages are left to the participants’ discretion

Permis 2004


Examples of nist usar arenas l.jpg
Examples of NIST USAR Arenas Search and Rescue Test Arenas

Permis 2004


Usar competitions l.jpg
USAR Competitions Search and Rescue Test Arenas

  • Success depends on:

    • Mobility of platforms

    • Skill of the operator

    • Affordances, ease of use of the user interface

    • Sensor packages

    • Communications

    • System robustness

  • Currently we do not evaluate the various components separately but use the overall system performance in determining the winners of the competitions

Permis 2004


Hri evaluation l.jpg
HRI Evaluation Search and Rescue Test Arenas

  • Challenge:

    • To determine the contribution of the human-robot interaction design to the overall performance

    • And in the process, to develop both metrics for HRI and guidelines for the design

  • HRI

    • More than just the visual interface

    • Includes the design of the interaction dialogue between the robot(s) and the operator(s)

Permis 2004


Data collection for hri at usar competitions l.jpg
Data Collection for HRI at USAR Competitions Search and Rescue Test Arenas

  • Have collected data from 6 major competitions since 2002

  • Offers wide range of HRI designs

  • Operators are robotics researchers, hence best case

  • Limited in our ability to interview/ control conditions

  • Data collected include:

    • Video of robot in arena (ground truth)

    • Video of what operator sees

    • Video of operator actions (in some cases)

    • Maps of coverage of arenas

Permis 2004


Data analysis l.jpg
Data Analysis Search and Rescue Test Arenas

  • Hypothesis: Systems that are able to cover more of the arena should be more successful

  • Analyzed % of time spent in

    • Navigation

    • Victim identification

    • Logistics

    • Failures

  • Looked for correlations between coverage, where time was spent, success in competition

    • More time spent navigating, more victims found

    • Correlation with coverage is difficult to compute; time between arenas, difficulty of arenas; difficulty in assessing

Permis 2004


Data analysis cont l.jpg
Data Analysis, cont. Search and Rescue Test Arenas

  • Human-robot awareness

    • The knowledge the human has of the location, status, and behavior of the robot

  • Indirect measures necessary

  • Used Critical Incident analysis

    • Global navigation

    • Local navigation

    • Obstacle evaluation

    • Vehicle state

    • Victim ID

Permis 2004


Data analysis cont10 l.jpg
Data Analysis, cont. Search and Rescue Test Arenas

Permis 2004


Data analysis cont11 l.jpg
Data Analysis, cont. Search and Rescue Test Arenas

  • What contributed to fewer critical incidents?

    • Local navigation

      • Frame of reference provided – overhead camera; 2 degree of freedom camera used to see wheels of robot in relation to environment

    • Obstacle encounters

      • Front and rear cameras

      • Ability to move robot and camera at same time

    • Vehicle state

      • Top down view of robot may have helped

      • Audio also helped (but noise in arena was excessive at times)

  • How did this correlate with success in competition?

    • Obstacle encounters were the best predictor but too little data to generalize

Permis 2004


Data analysis cont12 l.jpg
Data Analysis, cont. Search and Rescue Test Arenas

  • Robocup 2004

    • Allowed us to compare overhead camera use with automatic mapping

Permis 2004


Guidelines for hri design l.jpg
Guidelines for HRI Design Search and Rescue Test Arenas

  • Information for effective situation awareness should include:

    • a frame of reference to determine the position of the robot relative to the surrounding environment

    • indicators of vehicle state, such as pitch, roll, traction indicators, indicators of sensor status, and camera positions relative to the robot body.

    • a map to provide global navigation information

  • Minimize the number of windows provided to the operator.

  • Provide a fused view of sensor information.

  • Support multiple robot operators in a single display.

  • Provide help from the robot in determining what mode of autonomy is most useful.

Permis 2004


Conclusions recommendations l.jpg
Conclusions/ Recommendations Search and Rescue Test Arenas

  • Awareness assessment provides insights about information needed by operators to avoid critical incidents

  • Indirect evaluation is problematic

    • Takes lots of resources to evaluate; hence cannot produce feedback for robotics researchers in timely fashion

  • Potential solution for more direct assessment

    • “compulsory figures” evaluation for USAR competitions

    • Place robots in a number of situations and measure time/accuracy needed for operators to assess and describe the situation

    • Eliminates execution of the situation (operator skill, platform mobility)

    • Could also provide a benchmark system for comparison

Permis 2004