psychophysiology based affective communication for implicit human robot interaction n.
Skip this Video
Loading SlideShow in 5 Seconds..
Psychophysiology-Based Affective Communication for Implicit Human-Robot Interaction PowerPoint Presentation
Download Presentation
Psychophysiology-Based Affective Communication for Implicit Human-Robot Interaction

Loading in 2 Seconds...

play fullscreen
1 / 62

Psychophysiology-Based Affective Communication for Implicit Human-Robot Interaction - PowerPoint PPT Presentation

  • Uploaded on

Ph.D. Defense. Psychophysiology-Based Affective Communication for Implicit Human-Robot Interaction. Pramila Rani , 2005 October 24, 2005. Committee: Dr. Nilanjan Sarkar (Chair) Dr. Mitch Wilkes, Dr. Richard Shiavi, Dr. Eric Vanman, and Dr. Michael Goldfarb. Some Definitions.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Psychophysiology-Based Affective Communication for Implicit Human-Robot Interaction' - johana

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
psychophysiology based affective communication for implicit human robot interaction

Ph.D. Defense

Psychophysiology-Based Affective Communication for Implicit Human-Robot Interaction

Pramila Rani, 2005

October 24, 2005


Dr. Nilanjan Sarkar (Chair)

Dr. Mitch Wilkes, Dr. Richard Shiavi,

Dr. Eric Vanman, and Dr. Michael Goldfarb

some definitions
Some Definitions
  • Human-Robot Interaction
    • The study of humans, robots and the ways in which they influence each other
  • Psychophysiology
    • Science of understanding the link between psychology and physiology
  • Affective Communication
    • Communication relating to, arising from, or influencing feelings or emotions

Pramila Rani

research focus
Research Focus

This dissertation involves developing an intuitive affect*-sensitive human-robot interaction framework where

  • robot interacts with a human based on his/her probable affectivestate
  • affective states are inferred from the human's physiological signals
  • robot adapts its behavior in response to the human's affective state


Pramila Rani

  • Motivation
  • Research Hypotheses
  • Main Components
  • Results
  • Discussion
  • Conclusion

Pramila Rani

  • The Robot “Invasion”
    • There is a projected increase of 1,145% in the number of personal service robots in use within a year
    • According to World robotics 2004 report, at the end of 2003, about 610,000 autonomous vacuum cleaners and lawn-mowing robots were in operation
    • In 2004-2007, more than 4 million new units are forecasted to be added!!!
  • Need for Natural and Intuitive Human-Robot Communication
    • Unlike industrial robots, personal and professional service robots will need to communicate more naturally and spontaneously with people around
    • Robots will be expected to be understanding, emphatic and intelligent

Pramila Rani

  • Attempt to mimic Human-Human Interaction
    • More than 70% of communication is non-verbal or implicit
    • Emotions are a significant part of communication
    • 7% percent of the emotional meaning of a message is communicated verbally. About 38% by paralanguage and 55% via nonverbal channels [1]
  • Most Significant Channels of Implicit Communication in Humans
    • Facial Expressions
    • Vocal Intonation
    • Gestures and Postures
  • Physiology

[1] Mehrabian, A. (1971). Silent Messages. Wadsworth, Belmont, California

Pramila Rani


Giving Robots Emotional Intelligence

  • Robots should be capable of implicit communication with humans
  • They should detect human emotions
  • They should modify their behavior to adapt to human emotions

Pramila Rani

application areas
Application Areas

Some Potential Application Areas of Affect-Sensitive Robots

Pramila Rani

research challenges
Research Challenges
  • Human–Centric Technology
  • Affective Robots- Emotion Expression and Perception
  • Physiology-Based Affective Computing
  • Challenges of Affect Recognition
  • Robot Control Architecture
  • Robots and Real-Time Affective Feedback

Pramila Rani

human centric technology
Human–Centric Technology
  • Technological advancement so far has been more machine-centric
  • Now, new areas of robot application are emerging (e.g., battlefield, space, personal assistance, search & rescue)
  • There is a need to synergistically combine various capabilities of robotic systems with human intelligence
  • Most robots lack implicit channel of communication with humans

There is an evident need for technological innovation in HRI that permits implicit communication between humans and robots so that we can begin to build affective or emotionally-intelligent robots.

Pramila Rani

affective robots
Affective Robots
  • Two-fold capability of an affective robot
    • Perceive emotions in humans,
    • Express its own emotions in a manner understandable to humans.
  • There exist robots that can express their emotions using human-like facial expressions and affective speech
  • Need for real understanding of human emotions
    • detecting anxiety, frustration, engagement, boredom etc.
    • reacting to these emotions

For intelligent and intuitive human-robot interaction, it is imperative that the robot should be capable of perceiving human psychological states and adapting its behavior appropriately to address such a perception.

Pramila Rani

physiology based affective computing
Physiology-Based Affective Computing
  • Affect Recognition via Facial Expressions and Vocal Intonation
    • Work under highly constrained conditions
    • Dependent on gender, age, culture
    • Under voluntary control, hence manipulable
    • Computationally expensive and not designed for real-time affect recognition.
  • Advantages of using Physiology for Affect Recognition
    • Largely involuntary
    • Reasonably independent of cultural, gender and age related biases.
    • Continuously available and are not dependent on overt emotion expression
  • Technological Advancement in Physiological Sensing
    • Smaller, noninvasive , better sensors
    • Wireless communication
    • High-speed signal processing and pattern recognition capabilities

Given the strong relationship between physiology and affective states, and the continuous and involuntary nature of physiological phenomena, advanced signal processing and machine learning techniques can be effectively employed to determine an individual's underlying affective states in real-time.

Pramila Rani

physiology and affect
Physiology and Affect
  • Current Physiology-Based Affect-Recognition Systems
    • Vyzas et. al., Kim et. Al., Nasoz, Hayakawa et. al.
  • Limitations of Current Systems
    • Distinguish between discrete affective states
    • Affect elicitation usually involves audio/visual stimuli, or in some cases deliberate emotion expression
    • Very few systems work online
    • No systematic investigation of the relationship between a comprehensive set of physiological signals, their features and the affective states

It would be useful to develop an online affect-recognition system based on features derived from multiple physiological signals, that can detect arousal of specific emotions of individuals while they are engaged in real-life task experiments.

Pramila Rani

machine learning
Machine Learning
  • The data sets are extremely constrained:
    • Noisy
    • Small size
    • Missing predictor variables
    • High input dimensionality
    • Possible redundancy in the input domain
  • Machine learning techniques employed by other works
    • Fuzzy Logic, Neural Network, Hidden Markov Models, and Bayesian learning
  • Regression Tree based affect-recognition not been investigated till now

It would be worthwhile to empirically study the classification performance, advantages and disadvantages of few key machine learning techniques when applied to the domain of affect recognition using physiological signals.

Pramila Rani

real time affective feedback
Real-Time Affective Feedback

Requirements of Robot Control Architecture

  • Support channels for Explicit and Implicit Communication
  • Interpret affective input in the task context
  • Adapt Robot functionality to accommodate the affective states of the human
  • Allows mixed-initiative interaction between the human and robot

Till date there is no human-robot interaction system available in which real-time physiology-based feedback is utilized by a robot to interpret the underlying psychological state of the human and modify or adapt its (robot's) behavior as a result.

Pramila Rani

research hypotheses
Research Hypotheses
  • It is possible to detect distinct affective states and further differentiate within varying levels of each affective state using multiple indices derived from physiological signals in real-time
  • Such a channel of implicit-communication can be integrated within a machine's control architecture to make it capable of detecting human affective states and responding to them appropriately
  • Such systems are expected to improve human performance, while lowering the user's anxiety and increasing task challenge.

Pramila Rani

research components
Research Components
  • Theoretical
    • Psychological states relevant in implicit communication
    • Physiological signals to be monitored
    • Control Architecture to accommodate implicit communication
    • Task design for training (Phase I) and validation (Phase II) phases
  • Computational
    • Signal conditioning and processing
    • Machine learning for affect recognition
  • System Development
    • Phase I and Phase II
  • Experimental
    • Phase I & Phase II

Pramila Rani

psychological states selected
Psychological States Selected

Anxiety, Engagement, Boredom, Frustration, and Anger

  • These psychological states play an important role in human-machine interaction.
  • The affective states identified above were mainly chosen from the domain of negative affective states since they can be more closely related to performance and mental health of humans while working with machines.
  • Discussion with Psychologists, review of research works done in psychophysiology and human factors, and preliminary piloting was instrumental in this selection.

Pramila Rani

physiological signals
Physiological Signals
  • Impedance Cardiogram (ICG)
  • Electrocardiogram (PCG)
  • Pulseplethysmogram (PPG)
  • Phonocardiogram (PCG)
  • Electromyogram (EMG)
    • Corrugator Supercilii
    • Zygomaticus Major
    • Upper Trapezius
  • Peripheral Temperature
  • Electrodermal Activity (EDA)

Pramila Rani

impedance cardiogram
Impedance Cardiogram

Q Point


B Point

  • Relationship with Affective States
    • Pre-Ejection Period (PEP) is most heavily influenced by sympathetic innervation of the heart.
    • Reduced PEP is a marker of negative affect states – specifically anxiety
  • Features Extracted
    • Mean PEP
    • Mean IBI

ECG Signal

dZ/dt, where Z = ICG Signal

Pramila Rani

ecg and ppg




  • Relationship with Affective States
    • ECG influenced by frustration, anger and anxiety
    • PPG modulated by anxiety, fear of harm
    • Negative affect dimension specifically associated with increased sympathetic arousal
  • Features extracted
    • Mean Interbeat Interval (IBI)
    • Std. of IBI
    • Sympathetic power
    • Parasympathetic power
    • Ratio of Sympathetic to Parasympathetic power
    • Mean amp. of the peak values of the BVP signal
    • Standard deviation (Std.) of the peak values of the BVP signal
    • Mean Pulse Transit Time

ECG Signal

Pulse Transit Time

Pramila Rani

phonocardiogram heart sound
Phonocardiogram (Heart Sound)

Features extracted

  • Mean of the 3rd,4th, and 5th level coefficients of the Daubechies wavelet transform of heart sound signal
  • Standard deviation of the 3rd,4th, and 5th level coefficients of the Daubechies wavelet transform of heart sound signal

Pramila Rani

  • Relationship with Affective States
    • Facial displays (frowns, grimaces, smiles etc.) of affective reactions are obvious overt behaviors associated with expression of emotions
    • The Corrugator Supercilii muscles (responsible for lowering and contraction of the brows) considered as a measure of distress
    • EMG activity in the Zygomaticus Major occurs when the cheek is drawn back or tightened. This activity has been found to increase with expression of pleasure.
  • Features Extracted
    • Mean of EMG activity
    • Std. of EMG activity
    • Slope. of EMG activity
    • Mean Interbeat Interval of blink activity
    • Mean amplitude of blink activity
    • Mean and Median frequency of Corrugator, Zygomaticus and Trapezius

EMG Signal Bio112/muscles%20fac...

Pramila Rani

electrodermal activity
Electrodermal Activity
  • Relationship with Affective States
    • Tonic SC can be a useful index of a process related to energy mobilization or regulation
    • SC response is produced by social stimulation that invokes stress, tension, anxiety or cognitive reactions.
    • Significantly smaller values of SC response associated with neutral states than with sadness, anger, fear, disgust, and amusement
  • Features Extracted:
    • Mean tonic activity level
    • Slope of tonic activity
    • Mean amplitude of skin conductance response (phasic activity)
    • Maximum amplitude of skin conductance response
    • Rate of phasic activity

Typical Skin Conductance Response

Skin Conductance Signal

Pramila Rani

peripheral temperature
Peripheral Temperature
  • Relationship with Affective States
    • Peripheral temperature is an indirect index of peripheral vasoconstriction.
    • Skin temperature can vary by 1-2 degrees Fahrenheit depending upon the emotional state of a person
    • In the flight/fight stress response peripheral nervous system shunts the blood away from one’s extremities and into the brain, heart and lungs, to aid in optimum performance in order to eliminate the acute stress.
  • Features Extracted
    • Mean
    • Slope, and
    • Standard deviation of temperature recording

Pramila Rani

task design phase i
Task Design (Phase I)


  • The anagram solving task has been previously employed to explore relationships between both electrodermal and cardiovascular activity with mental anxiety.
  • In this task, emotional responses were manipulated by presenting the participant with anagrams of varying difficulty levels, as established through pilot work.
  • Affective states such as engagement, boredom, anger, frustration and anxiety were induced by manipulating the difficulty of anagrams
  • All these conditions were well tested during the task design and development stage and piloting.

Pramila Rani

task design phase i1
Task Design (Phase I)


  • Pong game has been used in the past by researchers to study anxiety, performance, and gender differences
  • Various parameters of the game were manipulated to elicit the required affective responses. These included:
    • ball speed and size,
    • paddle speed and size,
    • sluggish or over-responsive keyboard,
    • random keyboard response.
  • The relative difficulties of various trial configurations were established through pilot work.

Pramila Rani

task design phase ii
Task Design (Phase II)
  • Pong
    • Interactive Pong
    • Real-time feedback regarding player anxiety provided to machine
    • Performance-based game adaptation
    • Anxiety-based game adaptation

Pramila Rani

task design phase ii1
Task Design (Phase II)
  • Robot Basketball Game
    • A basketball hoop attached to a robotic manipulator
    • The difficulty of the task varied by controlling parameters such as robot arm speed and direction of motion.
    • Performance Based Game Adaptation
    • Anxiety-Based Game Adaptation

Pramila Rani

  • Signal conditioning and processing
    • Algorithms for artifact-rejection, adaptive thresholding, signal conditioning and feature-extraction for various signals
    • Wavelet transform, Fourier transform and statistical analysis and were extensively used in order to perform signal processing
  • Machine learning for affect recognition
    • A systematic comparison of the strengths and weaknesses of four machine learning methods - K-Nearest Neighbor, Regression Tree, Bayesian Network and Support Vector Machine* was performed

* SVM analysis was done by Mr. Changchun Liu

Pramila Rani

signal processing
Signal Processing
  • Adaptive Thresholding
    • An adaptive or continuously changing threshold value was used to determine whether candidate for peaks qualified to be valid peaks.
    • The need for this arose from the fact that for signals such as PPG (pulseplethysmogram), the average peak amplitude shows a large deviation over a given period of time.

Pramila Rani

adaptive thresholding
Adaptive Thresholding
  • A moving window was used to determine the threshold
  • The peaks in a given window were weighed so that the most recent peaks had higher weight values than the older peaks.
  • Where C = Scaling factor, wi = weight for peak (k-i), and Ak-I is the amplitude of peak (k-i). The values of wi were such that smaller the value of i, greater the value of wi. The values of wi, k, and C were determined by Monte Carlo Simulations.

Pramila Rani

fourier transform
Fourier Transform
  • Powerful signal analysis technique to study the frequency components of the physiological signals
  • Time series waveforms do not capture frequency-related variabilities easily
  • Frequency domain analysis has proven valuable in linking physiological abnormalities and variability to specific frequency bands.
  • Fourier Transform of the interbeat interval (IBI) derived from ECG

Pramila Rani

wavelet transform
Wavelet Transform
  • Signals such as PCG are complex and highly non-stationary
  • FFT analysis has limited analysis capabilities for such signals
  • Wavelets can be a very powerful tool for performing time-frequency analysis of non-stationary signals such as heart sounds.
  • It allows simultaneous localization in time and frequency domain
  • It has inbuilt noise filtering

Pramila Rani

filtering artifact rejection
Filtering & Artifact Rejection
  • Filtering of the signal is required to focus on a narrow band of electrical energy that is of interest
  • It removes noise and artifact such as that commonly found at 50 or 60 Hz (emitted into the recording
  • environment by devices such as florescent lights , computer power supplies)
  • Other elements that need to be filtered out are the artifacts caused by limb motions
  • Band pass, low pass and high pass were employed depending upon the frequency of interest

EMG Signal Filtered in Different Ways

Pramila Rani

machine learning for affect recognition
Machine learning for affect recognition
  • KNN
    • description
  • BNT
    • description
  • RT
    • description
  • SVM
    • description

Pramila Rani

system development
System Development

PhaseI System Set-up

Pramila Rani

system development1
System Development


Task in Progress

Room 1- Experimental Set-up


Wearable Sensors


Pramila Rani

system development2
System Development

Phase II System Set-up for Pong

Pramila Rani

system development3
System Development

Phase II System Set-up for Robot Basketball

Pramila Rani

  • Model building and Verification

Model Verification and Analysis

Phase I Experiments

Data Analysis for Model Building

Phase II Experiments

Models for Affective States

(I will make a better figure)

Pramila Rani

sliding window technique
Sliding Window Technique
  • For New Participants

Pramila Rani

  • Phase I
    • Fifteen participants took part in a 2-month study during which each person completed six sessions (three sessions of playing Pong and three sessions of solving anagrams)
  • Phase II
    • In the verification experiments for Pong nine participants who also took part in Phase I experiments volunteered
    • Fifteen participants took part in the robot-based basketball game. None of them had participated in Phase I and four of them were new.

Pramila Rani

  • Experiment Procedure

Pramila Rani

robot basketball
Robot Basketball
  • Experiment Procedure

Pramila Rani

  • Relationship between physiological signals and affective states
  • Accuracy of Regression Tree based affect recognition
  • Comparison between Regression Tree, KNN, Bayesian Networks and Support Vector Machines
  • Results of Pong game and robot-based basketball game with real-time affective feedback

Pramila Rani


Relationship Between Physiological Signals and Affective States

  • High Correlations found between physiological signals and affective states
  • Extent of correlation was different for different affective states

Pramila Rani


Physiological Signals and Affective States

  • Person Stereotypy
  • The highly correlated physiological indices vary from individual to individual

Pramila Rani


Comparison between Machine Learning Techniques

  • A systematic comparison of four machine learning methods - K-Nearest Neighbor, Regression Tree, Bayesian Network and Support Vector Machine was performed
  • All the four methods performed competitively
  • Support Vector Machine yielded the best classification accuracy
  • Regression Tree gave the next best classification accuracy
  • Regression Tree was the most space and time efficient method.

Pramila Rani


Regression Tree-Based Affect Recognition

Classification accuracy

Distinction wrt anxiety

Pramila Rani


Pong Game with Real-Time Affect Recognition

Increase in Performance

Increase in Challenge

(More results will be added)

Decrease in Anxiety

Pramila Rani

  • BB Game with Real-Time Affect Recognition
    • Anxiety
    • Challenge
    • Performance
    • Satisfaction Index

Pramila Rani


Pramila Rani


Pramila Rani

future work
Future Work

Pramila Rani


Dissertation Related Publications:

  • Rani, P, Sims, J, Brackin, R, and N. Sarkar, “Online Stress Detection using Psychophysiological Signal for Implicit Human-Robot Cooperation,” inRobotica, Vol. 20, No. 6, pp. 673-686, 2002.
  • Rani, P., Sarkar, N., Smith, C., and L. Kirby, “Anxiety Detecting Robotic Systems – Towards Implicit Human-Robot Collaboration,” in Robotica, Vol. 22, No. 1, pp. 85-95, 2004.
  • (Under Review) Rani, P., Sarkar, N., Smith, C., A., Adams, J., A., “Affective Communication for Implicit Human-Machine Interaction”, IEEE Transactions on Systems, Man, and Cybernetics.
  • (Under Review) Rani, P., Sarkar, N., “An Approach to Human-Robot Interaction Using Affective Cues,” IEEE Transactions on Robotics.
  • Rani, P., Sarkar, N., "Operator Engagement Detection and Robot Behavior Adaptation in Human-Robot Interaction", IEEE International Conference on Robotics and Automation, April 2005, Barcelona, Spain.
  • Rani, P., Sarkar, N., Smith, C., "Affect-Sensitive Human-Robot Cooperation –Theory and Experiments", IEEE International Conference on Robotics and Automation, pp: 2382-2387, Taiwan, September 2003.
  • Rani, P., Sarkar, N., "Maintaining Optimal Challenge in Computer Games Through Real-Time Physiological Feedback ", HCI International, July 2005, Las Vegas, USA.
  • Rani, P., Sarkar, N., Smith, “Anxiety Detection for Implicit Human-Robot Collaboration”, IEEE International Conference on Systems, Man & Cybernetics, Washington D.C., pp: 4896-4903, October 2003.
  • Rani, P., Sarkar, N., "Emotion-Sensitive Robots- A New Paradigm for Human-Robot Interaction", IEEE-RAS/RSJ International Conference on Humanoid Robots (Humanoids 2004), November 2004, Los Angeles, USA
  • Adams, J, Rani, P, Sarkar, N, “Mixed Initiative Interaction and Robotic Systems”, Workshop on Supervisory Control of Learning and Adaptive Systems, Nineteenth National Conference on Artificial Intelligence (AAAI-04), San Jose, CA, July, 2004.
  • (Submitted) Liu, C, Rani, P., Sarkar, N., "Comparison of Machine Learning Techniques for Affect Detection in Human Robot Interaction," IEEE/RSJ International Conference on Intelligent Robots and Systems, August 2005, Canada.
  • (Submitted), Rani, P., Sarkar, N., “Making Robots Emotion-Sensitive - Preliminary Experiments and Results,”, ROMAN 2005

Pramila Rani



  • Rani, P., Sarkar, M., Brackin, R., Sarkar, N., "Semi-Autonomous Human-Robot Interaction Using EMG-Based Modified Morse Code", Workshop on Multi-point interaction in Robotics and Virtual Reality, IEEE International Conference on Robotics and Automation, New Orleans, USA, April 2004. (To appear as a chapter in Springer-Verlag Tract on Advanced Robotics)
  • Erol, D, Rani P, Brackin, RL, Sarkar, MS, and Sarkar, N, "Robotic Aid to People with Disability By Means of an Innovative Communication Paradigm", 9th Mechatronics Forum International Conference (Mechatronics 2004) , September 2004, Ankara, Turkey
  • Chai, P, Rani, P, and N. Sarkar, “An innovative high-level human-robot interactions for disabled persons,” IEEE International Conference on Robotics and Automation, New Orleans, USA, April 2004.
  • (Submitted), Rani, P., Sarkar, M., “EMG-Based High Level Human-Robot Interaction System for People with Disability,” ROMAN 2005

Pramila Rani


Pramila Rani


Pramila Rani