shared control for dexterous telemanipulation with haptic feedback l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Shared Control for Dexterous Telemanipulation with Haptic Feedback PowerPoint Presentation
Download Presentation
Shared Control for Dexterous Telemanipulation with Haptic Feedback

Loading in 2 Seconds...

play fullscreen
1 / 76

Shared Control for Dexterous Telemanipulation with Haptic Feedback - PowerPoint PPT Presentation


  • 372 Views
  • Uploaded on

Shared Control for Dexterous Telemanipulation with Haptic Feedback Weston B. Griffin Dissertation Defense Presentation May 1, 2003 Telemanipulation First systems developed ~ 1940’s handling radioactive materials Can provide access to dangerous environments

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Shared Control for Dexterous Telemanipulation with Haptic Feedback' - paul


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
shared control for dexterous telemanipulation with haptic feedback

Shared Control for Dexterous Telemanipulation with Haptic Feedback

Weston B. Griffin

Dissertation Defense Presentation

May 1, 2003

telemanipulation
Telemanipulation
  • First systems developed ~ 1940’s
    • handling radioactive materials
  • Can provide access todangerous environments
  • Benefit from natural human abilities

operator

master

slave

environment

[The E1 developed by Goertz at Argonne National Lab]

telemanipulation3
Telemanipulation
  • Applications include:
    • underwater salvage
    • nuclear waste handling
    • space station repair
    • minimally invasive surgery

[Intuitive Surgical, Canadian Space Agency, Oceaneering International]

telemanipulation frameworks

position

force

Telemanipulation Frameworks
  • computer controlled electro-mechanical systems

remote controlled robot <> feeding back information

  • several different architectures

Operator

Master System

Slave Controller

Slave Manipulator

extends a person’s sensing and/or manipulation ability to a remote location

manipulation

position

force

Manipulation
  • Desire to leverage human manipulation skills
    • immersive hand/finger based system

Master System

Slave Manipulator

movie
Movie
  • Remote Control by Andy Shocken
    • filmed 2002 in our lab
    • narrated by Mark Cutkosky
issues in telemanipulation

Master system design

    • difficult task considering complexity of human hands
    • active area of research
  • Enhance slave controller
    • by sharing control between operator and slave system

- shared control

Issues in Telemanipulation

operator may feel remotely present

BUT

is not getting normal manipulation cues

  • Current telemanipulation limitations
    • force feedback (limited accuracy and fidelity)
    • limited tactile display
contributions
Contributions
  • Development a human-to-robot mapping method
    • map glove-based hand motions to a planar robot hand
  • Development and implementation of a shared control framework for dexterous telemanipulation
    • combining operator commands with a semi-autonomous controller
  • Investigation of an experimental telemanipulation system
    • results demonstrate benefits of shared control and need to choose carefully types of feedback to achieve a real improvement
outline
Outline
  • System overview
  • Human-to-robot mapping
  • Shared control framework
  • Experimental investigation

Development of dexterous telemanipulation system

improving telemanipulation
Improving Telemanipulation
  • Take advantage of the slave controller and local sensor information for improved dexterity
    • add “low-level” intelligence

Why?

  • can feedback sensor information by other means
  • robot can intervene in certain situations (fast response)
  • human and robot can share control for improved performance
shared control

force

position

commands

sensor feedback

Shared Control

bilateraltelemanipulation

high level

commands & feedback

semi-autonomous dexterous manipulation

shared control12
Shared Control

combining operator high level and low level commands

with a remote controller

for improved manipulation

master system

Hand tracker

CyberGlove

CyberGrasp

Master System
  • CyberGlove™ instrumented glove
    • 22 bend sensors
    • calibrated for dexterous manipulation [Turner 2001]
  • CyberGrasp™ fingertip force feedback
    • lightweight exo-skeleton
    • uni-directional force feedback
  • Logitech hand tracker
    • ultrasonic transducers and sensors
    • 6 d.o.f. position and orientation

[CyberGlove and CyberGrasp are products of Immersion Corporation]

slave system
Slave System
  • Custom built robot hand
    • two fingers, two d.o.f. per finger
    • low inertia DC motors
    • cable capstan drive
  • Robot arm
    • Adept industrial arm, five d.o.f.
    • enlarges task workspace
  • Fingertip sensors
    • two-axis force sensors
    • contact location sensors
system architecture

1000 Hz

200 Hz

1000 Hz

50 Hz

200 Hz

7 Hz

63 Hz

1000 Hz / 200 Hz

1000 Hz

System Architecture

Master

CyberGrasp

CyberGlove

Indirect Feedback

Wrist Tracker

GUI

QNX Node-to-Node

QNX-Node 1

QNX-Node 2

Adept Control

Slave

Slave Control

outline16
Outline
  • System overview
  • Human-to-robot mapping
  • Shared control framework
  • Experimental investigation

Development of dexterous telemanipulation system

human to robot mapping
Human-to-Robot Mapping
  • Robot is non-anthropomorphic, symmetric, and planar
  • joint-to-joint mapping not possible
  • very different workspace
human to robot mapping18
Human-to-robot Mapping
  • How do you control a non-anthropomorphic robot hand using a human hand and glove?

? ?

virtual object mapping
Virtual Object Mapping
  • Interpret human fingertip motions to be imparting motions to a virtual object held between the fingers
  • Virtual object parameters are mapped to robot
    • to produce fingertip positions OR motions of a grasped object
  • Parameters independently modified
    • to account for kinematic and workspace differences
virtual object mapping20

0

-0.05

-0.1

-0.15

Mapped Pinch Point Position

-0.2

-0.25

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

Index Mapped Positions

Left Finger Boundary

Thumb Mapped Positions

Right Finger Boundary

Virtual Object Mapping
  • Match natural human manipulation motions to corresponding robot hand motions
  • good mapping?
    • operator can intuitively control robot and utilize robots workspace
outline21
Outline
  • System overview
  • Human-to-robot mapping
  • Shared control framework
  • Experimental investigation

Development of dexterous telemanipulation system

shared control22
Shared Control
  • Hannford et al. [1991]
    • force feedback joystick controlling robot arm/gripper
    • improved task completion time and resulted in lower forces
  • Michelman and Allen [1994]
    • sequencing primitives for dexterous hand control
      • joystick control, no provisions for haptic feedback
  • Williams et al. [2002]
    • NASA’s Robonaut project - robot arm and dexterous hand
    • force feedback joystick for control
      • reduced task peak forces
shared control23
Shared Control

Next step: using shared control in a dexterous telemanipulation system with fingertip force feedback

How?

  • implement a semi-autonomous controller capable of dexterous manipulation
    • robot has force and tactile sensors and specialized control laws for manipulation
dexterous manipulation
Dexterous Manipulation
  • What does it mean to autonomously manipulate an object?
    • with sensors robot can detect the object and determine proper fingertip forces for:

manipulation

dexterous manipulation25
Dexterous Manipulation
  • What does it mean to autonomously manipulate an object?
    • with sensors robot can detect the object and determine proper fingertip forces for:

manipulation

dexterous manipulation26
Dexterous Manipulation
  • What does it mean to autonomously manipulate an object?
    • with sensors robot can detect the object and determine proper fingertip forces for:

manipulation

grasp force regulation

object manipulation control

Velocity Grasp Transform

Ts

+

+

ZOH

Tactile Based Object Tracking

-

Tactile Sensing

+

Object ImpedanceController

+

+

Forward Grasp Transform

Finger Controller

RobotFinger

Internal ForceController

+

-

Internal ForceDecomposition

Object Manipulation Control
  • Utilize the Grasp Transform to determine robot fingertip forces [Mason & Salisbury 1985]
object manipulation control28
Object Manipulation Control
  • Controlling internal force

Velocity Grasp Transform

Ts

+

+

ZOH

Tactile Based Object Tracking

-

Tactile Sensing

+

Object ImpedanceController

+

+

Forward Grasp Transform

Finger Controller

RobotFinger

Internal ForceController

+

-

Internal ForceDecomposition

object manipulation control29

Velocity Grasp Transform

Ts

+

+

ZOH

Tactile Based Object Tracking

-

Tactile Sensing

+

Object ImpedanceController

+

+

Forward Grasp Transform

Finger Controller

RobotFinger

Internal ForceController

+

-

Internal ForceDecomposition

Object Manipulation Control
  • Controlling object position
shared control telemanipulation
Shared Control Telemanipulation
  • What are the advantages to programming robot for dexterous manipulation?
    • robot can monitor operator’s object manipulation
    • if necessary, robot can intervene (take over control of object manipulation)
      • impedance modification, limit motion, prevent release
    • robot can warn/inform operator of manipulation status through indirect methods
      • using other feedback modalities (visual indicators, audio, or augmented haptic feedback)
shared control telemanipulation31
Shared Control Telemanipulation
  • What are the advantages to letting robot take control over force regulation and/or object manipulation?
    • operator can focus on behavior of grasped object or tool
    • master commands are no longer essential to prevent unwanted slip or damaged objects
    • operator can still override to release or grasp more tightly
shared control telemanipulation32
Shared Control Telemanipulation

Shared control implementation issues

    • as the robot assumes more control
      • concern the operator’s sense of presence will be reduced
        • we want to keep the operator “in the loop”
    • preserve operator’s intent
    • what type of indirect feedback is most effective?
    • does sharing control improve performance in an immersive fingertip force feedback system?
  • To answer these questions we perform a set a controlled experiments
outline33
Outline
  • System overview
  • Human-to-robot mapping
  • Shared control framework
  • Experimental investigation

Development of dexterous telemanipulation system

previous experimental studies
Previous Experimental Studies
  • force feedback evaluation
    • Turner et al. 2000: block stacking and knob turning
      • force feedback with CyberGrasp not always a benefit
    • Howe & Kontarinis 1992: fragile peg insertion task
      • audio buzzer sounded if grasp force excessive
      • operators were not able to reduce force
  • shared control evaluation
    • Hannaford et al. 1991: peg insertion task
      • operator’s controlled position, shared orientation control
      • reduction in task completion time and insertion forces
experimental hypothesis
Experimental Hypothesis
  • Addition of a dexterous shared control framework will increase an operator’s ability to handle objects delicately and securely compared to direct telemanipulation
experiment description
Experiment Description
  • Motivating scenario: recovering an ancient Greek vase on the sea floor

“fragile object handling” - user’s asked to carry an object with minimal force but without dropping the object

experiment description38

If operator’s desired (commanded) force is too low

robot can monitor and warn the operator

OR

robot can intervene and regulate grasp force

to prevent object dropping

Experiment Description
  • To assist operator in fragile object handling taskthe robot computes the minimum grasp force required
shared controlled task
Shared Controlled Task
  • Operator maintains manipulation control
shared controlled task40
Shared Controlled Task
  • Operator maintains manipulation control
  • Robot and operator share control over internal force
    • robot monitors excessive force
shared controlled task41
Shared Controlled Task
  • Operator maintains manipulation control
  • Robot and operator share control over internal force
    • robot monitors excessive force
    • robot can apply minimum internal force required to prevent slip
sharing control in fragile task
Sharing Control in Fragile Task
  • Target window with intervention can be wider: desired force can drop below fint,min without adverse effects
  • In theory, it is possible to always do better without intervention
question that arise
Question that arise...
  • Does warning the operator of a possible failure help?
  • Does task performance improve with robot intervention?
  • If robot intervenes, is it necessary to inform operator?
  • Is it helpful to feed back information of impending state changes (such as object release)?
  • With haptic feedback in a force control task, what forces should be fed back?
case effects
Case Effects
  • Audio Alarms - when operator’s desired force is too high or too low
  • Robot Intervention - robot assumes control when operator’s desired force falls below a threshold (safe minimum internal force)
  • Visual Indicator (fingertip LEDs) - to inform the operator of robot intervention
  • Force Feedback: actual vs. commanded - during robot intervention, forces to operator’s fingertip are reduced (reduced force feedback)
experimental procedure
Experimental Procedure
  • Diverse set of subjects
    • 11 subjects total
    • 8 males and 3 females
  • Two sessions
    • first - calibration and training
    • second - four trials for each case
  • Case order randomized
    • reduce possible learning and fatigue effects
evaluating performance
Evaluating Performance
  • Objective data analysis
    • measured internal force applied to object
      • fragile object task - lower is better
    • task failures (number of drops)
    • task completion time
  • Subjective data analysis
    • operator’s expressed preference
    • operator’s perceived difficulty
typical subject data

Measured Internal ForceDesired Internal Force

Minimum Internal Force

Case 1

4

Force [N]

2

0

0

5

10

15

20

25

Time [sec]

Measured Internal ForceDesired Internal Force

101% of Minimum Internal Force

Case 2

Excessive Force Warning (Low Tone)

Object Slip Warning (High Tone)

4

Force [N]

2

0

0

5

10

15

20

25

Measured Internal ForceDesired Internal Force

110% of Minimum Internal Force

Excessive Force Warning (Low Tone)Object Release Warning (High Tone)

Robot Intervention

C

Case 6

A

4

D

E

B

Force [N]

2

0

0

5

10

15

20

25

Time [sec]

Typical Subject Data
data analysis

3.0

Average of allsubjects for each case

2.8

2.6

2.4

Internal Force [N]

2.2

2.0

1.8

average minimum internal force to prevent object slippage

1.6

1

2

3

4

5

6

7

Case Number

Data Analysis
  • Measured internal force applied to the object
    • averages of each subject for each case (trial failures excluded)
  • Boxplot
    • medians and quartiles
    • observe trends
  • Is there a significant effect?
statistical analysis
Statistical Analysis
  • ANOVA - determines the probability that these results (differences in averages) are really due to random variation in data
  • Apply to averaged measured internal force
    • p = 0.003 (<< 0.05), indicating that there is a difference between the means
    • but which ones are different
  • Can’t use a simple t-test for multiple comparisons
      • increase probability of false-positive
    • Dunnett’s method - comparison to a control (Case 1)
      • Cases 4, 6, 7 have statistically different mean than Case 1
      • a reduction of approximately 15%
task failures
Task Failures
  • Number of failures that occurred for each case (dropped object)

Number of Failures in Each Case - All Subjects

8

6

Case 5 and 6 had least number of failures

Number of Failures

4

2

0

1

2

3

4

5

6

7

Case Number

sub1

sub2

Number of Failures in Each Case

N/A

sub3

sub4

8

sub5

Case failures not dominated by one subject

sub6

sub7

6

sub8

N/A

sub9

Number of Failures

sub10

4

sub11

2

0

1

2

3

4

5

6

7

Case Number

objective data analysis results

Internal Force [N]

3.0

2.8

2.6

2.4

2.2

2.0

1.8

1.6

1

2

3

4

5

6

7

Case Number

Objective Data Analysis Results
  • Robot intervention improves performance
    • presence and type of direct and indirect feedback had an effect
      • Cases 4, 6, and 7 had lower internal force
      • Case 3 and 5 did not
analysis results
Analysis Results
  • Robot intervention improved performance
    • presence and type of direct and indirect feedback had an effect
      • Cases 4, 6, and 7 had lower internal force
      • Case 3 and 5 did not
    • only informing of intervention not adequate
      • Case 7 had most failures
      • indicating alarms were helpful

Number of Failures

8

6

4

2

0

1

2

3

4

5

6

7

Case Number

analysis results55
Analysis Results
  • Robot intervention improved performance
    • presence and type of direct and indirect feedback had an effect
      • Cases 4, 6, and 7 had lower internal force
      • Case 3 and 5 did not
    • only informing of intervention not adequate
      • Case 7 had most failures
      • indicating alarms were helpful
  • Reduced force feedback
    • compare Case 3 to 5
    • slight improvement in measured internal force (6%)
    • fewer failures in Case 5
      • Cases 4 and 6 show similar results

Number of Failures

8

6

4

2

0

1

2

3

4

5

6

7

Case Number

task time

Average of allsubjects for each case

Task Time
  • May reveal any physical or mental difficulties associated with the various conditions

35

no obvious trends

p = 0.82 (i.e., no difference in means)

30

25

Time [Sec]

shared controldid not improve task completion time BUT did not make it worse

20

15

10

1

2

3

4

5

6

7

Case Number

results
Results
  • Given objective data analysis performance criteria

minimizing internal force but preventing failures:

provided best overall performance compared to bilateral case

Case 6 - shared control with multi-modal feedback

  • In post experiment surveys, subjects also generally ranked Case 6 highest in preferenceand ease-of-use
conclusions
Conclusions

Answering our hypothesis

  • Can the addition of a dexterous shared control framework increase an operator’s ability to handle objects delicately and securely compared to direct telemanipulation?

YES, shared control gives better performance but you need to:

a) let the operator know when the intervention is active

b) let the operator know of impending state changes

c) feed back force based on commanded force and not actual forces (during intervention)

summary of contributions
Summary of Contributions
  • Development a human-to-robot mapping method
    • map glove-based hand motions to a planar robot hand that allows for intuitive hand control
  • Development and implementation of a shared control framework for dexterous telemanipulation
    • combining operator commands with a semi-autonomous controller
  • Investigation of an experimental telemanipulation system
    • results demonstrate benefits of shared control and need to choose carefully types of feedback to achieve a real improvement
future work
Future Work
  • Do the benefits of shared control extend to other situations and applications?
    • assembly tasks
    • e.g., steer-by-wire vehicles
  • Do the same requirements for shared control improvement hold?
    • informing the operator of intervention
    • notifying of impending state changes
    • modifying the forces fed back
acknowledgements
Acknowledgements
  • Mark Cutkosky
  • Defense Committee
  • Will Provancher
  • The DML
  • Eric (setting the pace in the final days)
shared control for dexterous telemanipulation with haptic feedback62

Shared Control for Dexterous Telemanipulation with Haptic Feedback

Weston B. Griffin

Dissertation Defense Presentation

May 1, 2003

one slide statistics review
One Slide Statistics Review
  • statistical analysis
    • two competing hypotheses
      • null: cases have no real effect (all the means are the same)
      • alternate: at least one case is different (all means are NOT the same)
one slide statistics review65
One Slide Statistics Review
  • statistical analysis
    • two competing hypotheses
      • null: cases have no real effect (all the means are the same)
      • alternate: at least one case is different (all means are NOT the same)
  • ANOVA - analysis of variance
    • tests if difference in means of several samples is significant based on variances

if ratio small then: all means are the same

within

if ratio large: at least one mean is different

  • how likely is it to have a t.s. as extreme as observed (p-Value)
  • compare to a significance level (95%)(e.g., reject null if p < 0.05)

performance quantity

between

Case

manipulation66
Manipulation
  • Desire to leverage human manipulation skills
    • immersive hand/finger based system

position

Human-to-robot Mapping

Operator

Slave Manipulator

force

Master System

telemanipulation67
Telemanipulation
  • Glove based
    • Brunner et al. 1994, DLR dexterous robot hand
    • Li et al. 1996 - NASA DART project
    • Ambrose et al. 2000, NASA Robonaut project
  • Teleoperation / telemanipulation
    • Lawn and Hannaford 1993
    • Lawrence et al. 1993
    • Daniel and McAree 1998
    • Sherman et al. 2000
    • Speich and Goldfarb 2002
control architectures
Control Architectures
  • general four-channel one d.o.f. framework

Fh

C3

Ve

C4

-

+

Ve

Fh

Vm

+

+

+

Zm-1

C1

Zs-1

Fh*

-

-

-

-

-

Cm

Cs

Ze

Zh

+

Fe

Vh

C2

+

Fe

Fe*

Human Operator

MasterSystem

Comm.Link

SlaveSystem

Environ-ment

[Lawrence 1993]

mapping background

DLR hand

Mapping Background
  • anthropomorphic
    • linear joint-to-joint [Kyriakopoulos et. al 1997]
    • fingertip position mapping
      • scaling [Fisher et a. 1998]
  • semi-anthropomorphic
    • pose matching [Pao and Speeter 1989]
      • joint angle transformation matrix
    • fingertip position mapping [Speeter 1992, Rholing et al. 1993]
      • forward kinematics, inverse kinematics
  • non-anthropomorphic
    • greater dissimilarities
      • grammar based
      • functional mapping[Speeter 1992]

Utah/MIT hand

JPL/Salisbury hand

Dexter hand

point to point mapping
Point-to-Point Mapping
  • initial approach
    • planar projection of fingertip positions
    • standard planar frame transformation
mapping implementation
Mapping Implementation
  • compute virtual object parameters
    • 3D size to capture thumb motion
    • planar reduction
  • computing robot positions
    • based on planar virtual object
transformation to robot frame
Transformation to Robot Frame
  • must modify and scale parameters for desired correspondence

kinematics

  • v.o. orientation
    • angular offset
  • v.o. midpoint
    • frame transformation

workspace

  • v.o. midpoint & size
    • scaled
parameter determination
Parameter Determination
  • based on individual’s recorded hand motion
    • three simple poses/motions
  • defining
    • orientation offset
    • midpoint transformation variables
    • midpoint scaling
    • size scaling
mapping results

0

Point-to-Point Mapping

-0.05

-0.1

Y-axis, [m]

-0.15

-0.2

0

-0.25

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

X-axis, [m]

-0.05

-0.1

Y-axis, [m]

Virtual Object Mapping

-0.15

-0.2

-0.25

Mapped Pinch Point Position

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

X-axis, [m]

Index Mapped Positions

Left Finger Workspace Boundary

Thumb Mapped Positions

Right Finger Workspace Boundary

Mapping Results

Virtual Object Mapping

  • improved achievable positions
  • pinch-point can be mapped to any point
  • fundamentally analytical
    • continuous, smooth, and predictable
  • fingertip-to-fingertip correspondence
modeling
Modeling
  • Model: averaged percent difference in measured internal force compared to Case 1

Percent Difference (from Case 1 for each subject) in Mean Internal Force

Means with Error Bars of Two Standard Deviations

15

10

5

0

Percent Difference, [%]

-5

-10

-15

-20

-25

-30

1

2

3

4

5

6

7

Case Number

model analysis
Model Analysis
  • Look at residuals:

Residuals due to Task Order (Learning and Fatigue Effects)

Means with Error Bars of Two Standard Deviations

0.4

0.3

0.2

0.1

Force, [N]

0

-0.1

-0.2

-0.3

-0.4

1

2

3

4

5

6

7

Order