Heterogeneous multi robot dialogues for search tasks
Download
1 / 22

Heterogeneous Multi-Robot Dialogues for Search Tasks - PowerPoint PPT Presentation


  • 48 Views
  • Uploaded on

Heterogeneous Multi-Robot Dialogues for Search Tasks. Thomas K Harris, Satanjeev (Bano) Banerjee Alexander Rudnicky. AAAI Spring Symposium 2005: Dialogical Robots. Communication among autonomous robots and humans.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Heterogeneous Multi-Robot Dialogues for Search Tasks' - hadassah-nolan


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Heterogeneous multi robot dialogues for search tasks

Heterogeneous Multi-Robot Dialogues for Search Tasks

Thomas K Harris, Satanjeev (Bano) Banerjee Alexander Rudnicky

AAAI Spring Symposium 2005: Dialogical Robots


Communication among autonomous robots and humans
Communication among autonomous robots and humans

  • Embodied (not necessarily robotic, or even anthropomorphic) agents will become ubiquitous

  • NL dialogue will be a useful modality

  • Agent’s dialogue and task models will remain tightly coupled, and independent from other agents

  • We will be talking to multiple agents without a (completely) shared dialog system

AAAI Spring ’05 Symposium: Dialogical Robots


Gedankenexperiment
Gedankenexperiment

  • The problem of many robots: sweeper, furniture mover, baby monitor.

  • Task: clean up house; don’t wake baby up.

  • Who do you talk to?

  • What do you say?

  • Who talks back?

AAAI Spring ’05 Symposium: Dialogical Robots


Issues for polylogical systems
Issues for polylogical systems

  • What’s a reasonable architecture?

    • A single system controlling multiple robots

    • A single DM directing autonomous robots

    • Shared understanding/generation

    • Shared microphone and speakers

    • Robots self-contained: Nothing shared

  • How to communicate with multiple agents?

    • Directions to the entire team? (“talk into the air”)

    • Mediation through a designated team leader? (foreman)

    • Instructions to each team member? (direct management)

    • To a proxy agent within/without the team? (personal assistant)

AAAI Spring ’05 Symposium: Dialogical Robots


What do we have today
What do we have today?

  • A platform that supports multiple (robot) participants

    • Sphinx II ASR; Festival TTS

    • RavenClaw dialogue manager

    • Galaxy-II message-passing architecture

    • Several “back-end” interfaces

  • A basic corridor movement domain

    • Commands to move robots along in vectors and towards named locations

    • Mechanisms for describing robot status and location to human interlocutor

AAAI Spring ’05 Symposium: Dialogical Robots


Current architecture
Current Architecture

HUMAN INTERFACE

ASR

AGENT1

DM1

Robot1

Interpret1

AGENT2

Interpret2

DM2

Robot2

Multi-voice TTS

AAAI Spring ’05 Symposium: Dialogical Robots


Implementation
Implementation

HUMAN INTERFACE

AGENT1

Sphinx II

Phoenix

Helios

Ravenclaw

Bashful

Bashful

Robot Communication

System

Galaxy

AGENT2

Ravenclaw

Clyde

Clyde

Swift

Rosetta

AAAI Spring ’05 Symposium: Dialogical Robots


Human robot treasure hunt
Human-Robot Treasure Hunt

  • One Human

  • Two robots

    • Segway

    • Pioneer

  • Treasures “hidden” in large space

  • TASK: Retrieve all treasures

AAAI Spring ’05 Symposium: Dialogical Robots


Scenario for search
Scenario for Search

We found it!

We are at <x,y>

AAAI Spring ’05 Symposium: Dialogical Robots


Issues in h r communication
Issues in H-R communication

  • How do people decompose the task into sub-tasks?

  • What language do people use to get the tasks performed by the robots?

  • Given a human command, what is the expected robot behavior?

  • Explore using Wizard of Oz experiments

AAAI Spring ’05 Symposium: Dialogical Robots


Woz design
WOZ design

  • 1 experimenter, 2 robot actors, 1 participant.

  • Experimenter places treasure, simulates robot treasure sensors.

  • 1st robot actor is blind, but carries a webcam for the participant’s consumptions.

  • 2nd robot actor can only move by crawling.

Communication takes place by normal speech, walkie-talkie, and through the webcam.

AAAI Spring ’05 Symposium: Dialogical Robots


Annotation and analysis
Annotation and analysis

Data transcribed and annotated

Utterances classified into functional categories

394 utterances

20 utterance categories

8 major categories

Carnegie Mellon MockBrow annotation tool

AAAI Spring ’05 Symposium: Dialogical Robots


Utterance task breakdown

Controlling team behaviors

Grounding

Positive/negative feedback

Informing robot of it’s state or the world

Explanations of commands

Orientation Grounding

Navigation

Simple Navigation commands

Spatial Referential Navigation

Object Referential Navigation

Manipulation

Manipulating the environment

Manipulating treasure

Coverage

Manipulating the webcam view

Object coverage commands

Generic coverage

Asking about the robot’s abilities

Filler

Real-time command modifications

Utterance/Task Breakdown

AAAI Spring ’05 Symposium: Dialogical Robots


Summary future work
Summary / Future Work

  • Architecture for human-robot teams

  • WoZ study of language requirements

  • Different WoZ scenarios

  • Implement language for robot team

  • “Field” testing

AAAI Spring ’05 Symposium: Dialogical Robots


Controlling team behaviors
Controlling team behaviors

  • “you guys get together”

  • “T- you go first and B- follow”

AAAI Spring ’05 Symposium: Dialogical Robots


Grounding
Grounding

  • Positive/negative feedback

    • “ok that’s better”

  • Informing robot of state

    • “so that’s up”

    • “I don’t see anything there”

  • Explanations of commands

    • “so I can see which direction is up”

  • Orientation Grounding

    • “What you’re facing now with the camera – is that the vehicle that you just circumnavigated”

    • “I can tell you’re going in the wrong direction, stop”

AAAI Spring ’05 Symposium: Dialogical Robots


Navigation
Navigation

  • Simple Navigation commands

    • “so um T- turn to you left”

    • “T- I want you to turn right 90 degrees”

    • “can you go in that general direction”

    • “can you proceed in that direction”

  • Spatial Referential Navigation

    • “go to that open area”

    • “continue around the periphery of that open area”

    • “back out of that alley”

    • “proceed in that direction until you find an opening to turn left”

  • Object Referential Navigation

    • “go over by T-”

    • “can you go on the other side of that vehicle”

    • “go over by the posters”

AAAI Spring ’05 Symposium: Dialogical Robots


Manipulation
Manipulation

  • Manipulating the environment

    • “T- why don’t you move the trash can”

  • Manipulating treasure

    • “T- bring the coin to me”

  • Manipulating the webcam view

    • “ok B- look to your left”

    • “B- can you look around with the camera a little”

AAAI Spring ’05 Symposium: Dialogical Robots


Coverage
Coverage

  • Object coverage commands

    • “ok so examine the shelf”

    • “do you see something on that shelf in front of B-”

    • “can you look over by that table over there”

  • Generic Coverage

    • “do you see anything that looks interesting”

AAAI Spring ’05 Symposium: Dialogical Robots


Asking about the robot s abilities
Asking about the robot’s abilities

  • “is that possible”

AAAI Spring ’05 Symposium: Dialogical Robots


Filler
Filler

  • “and now um”

  • “ok um”

AAAI Spring ’05 Symposium: Dialogical Robots


Real time command modifications
Real-time command modifications

  • “keep going”

  • “stop”

  • “a little more”

  • “change of plans”

  • “other direction”

AAAI Spring ’05 Symposium: Dialogical Robots


ad