slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Introduction Our goal PowerPoint Presentation
Download Presentation
Introduction Our goal

Loading in 2 Seconds...

play fullscreen
1 / 41

Introduction Our goal - PowerPoint PPT Presentation


  • 59 Views
  • Uploaded on

Automated Analysis of Interactional Synchrony using Robust Facial Tracking and Expression Recognition. Xiang Yu 1 , Shaoting Zhang 1 , Yang Yu 1 , Norah Dunbar 2 , Matthew Jensen 2 , Judee K. Burgoon 3 , Dimitris N. Metaxas 1 1 CBIM, Rutgers Univ., NJ

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Introduction Our goal' - jared


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Automated Analysis of Interactional Synchrony using Robust Facial Tracking and Expression Recognition

Xiang Yu1, Shaoting Zhang1, Yang Yu1,Norah Dunbar2, Matthew Jensen2, Judee K. Burgoon3, DimitrisN. Metaxas1

1CBIM, Rutgers Univ., NJ

2Oklahoma University, Ok 3Univ. of Arizona, AZ

introduction our goal
IntroductionOur goal
  • Predict whether an interactantis truthful or deceptive by analyzing interactional synchrony
  • Propose a computational framework to do so
introduction what is interactional synchrony
IntroductionWhat is interactional synchrony?
  • Interpersonal communication is contingent upon some form of mutual adaptation such as:
    • Accommodation
    • Interpersonal coordination
    • Matching
    • Mirroring
    • Compensation
    • Divergence
    • Complementarity
introduction why investigating interactional synchrony
IntroductionWhy investigating interactional synchrony?
  • Practitioners have suggested using interactional synchrony for detecting deception:
    • with terrorists (Turvey, 2008)
    • in FBI interviews (Navarro, 2003)
    • and in police investigations (Kassin et al., 2007)
  • Assumption: interviews with deceivers are less synchronous than interviews with truth tellers.
  • However, few systematic studies of coordination, synchrony or reciprocity have examined the effects of synchrony on deception.
introduction relevant t echniques
IntroductionRelevant techniques
  • Face Tracking

- Active Shape Model (Cootes et al. CVIU 1995)

- Active Appearance Model (Cootes et al. ECCV 1998)

- Constrained Local Model (Cristinacce and Cootes, BMVC 2006)

  • Gesture and Expression recognition

- Head nodding, shaking, HMM(Kapoor and Picard PUI 2001)

- Gesture recognition, CRF (Morency et al. CVPR 2007)

- Image based expression recognition (Pantic et al. PAMI 2000)

- Video based (Black and Yacoob, IJCV1997; Cohen et al. CVIU 2003)

module 1 robust facial tracking
Module 1: Robust Facial Tracking
  • Robust facial tracking is the foundation of interactional synchrony analysis
  • Challenges:
    • Partial occlusion
    • Multiple poses
    • Poor lighting conditions
active shape models
Active Shape Models
  • Active Shape Model

Shape vector:

Shape matrix:

PCA: Covariance matrix:

Shape representation:

active shape models1
Active Shape Models
  • Using KLT tracker to estimate shape locally.
  • Optimize shape , not only similar to , but also follows the shape distribution:
  • Alternately optimize and , until they converge.
limitations
Limitations
  • Partial occlusion:
    • Bayesian inference [Y. Zhou CVPR’03]
    • Pictorial structures [P. Felzenszwalb IJCV’05]
    • Sparse outliers [F. Yang FG’11]
  • Multiple poses:
    • Mixture of Gaussian distribution [T. Cootes IVC’99]
    • Kernel PCA [S. Romdhani BMVC’07]
    • Hierarchical multi-state [Y. Tong PR’07]
face tracking handle occlusions
Face Tracking: Handle Occlusions
  • We explicitly model the large errors as a sparse vector :

sparsity number of

face tracking handle multi pose
Face Tracking: Handle Multi-pose
  • Instead of a low dimensional subspace, we model the shape as a sparse linear combination of training shapes:

...

face tracking handle multi pose1
Face Tracking: Handle Multi-pose
  • Instead of a low dimensional subspace, we model the shape as a sparse linear combination of training shapes:

sparsity number of

sparsity number of

synthetic data
Synthetic data
  • Shape optimization from side pose with outliers. Blue line: detection results. Red line: fitting result.

E-ASM

Sparse shape registration

(Handle gross error)

Our method

(Handle gross error

and multi-pose)

quantitative comparison
Quantitative Comparison
  • The point error by pixel
occlusion by hat
Occlusion by Hat

Extended ASM

Sparse Shape Registration

Our method

occlusion by scarf
Occlusion by Scarf

Extended ASM

Sparse Shape Registration

Our method

multiple pose
Multiple Pose

Sparse shape Registration

Our Method

multiple pose with occlusions
Multiple Pose with Occlusions

Sparse shape Registration

Our Method

face tracking results
Face Tracking Results
  • Tracking result from CMC dataset
module 2 and 3 expression detection and head pose
Module 2 and 3: Expression Detection and Head Pose

Demo of the successful detection of events

(both are smiling and with the same head pose)

module 2 expression recognition
Module 2: Expression Recognition

Use the relative intensity order of facial expressions to learn a ranking model (RankBoost) for recog. and intensity estimation

6 universal facial expressions (i.e., fear, anger, sadness, disgust, happiness, surprise)

Trained ranking model gives a score for the prob. of smiling in real time.

expression recognition 1
Expression Recognition(1)
  • Facial feature representation.
expression recognition 2
Expression Recognition(2)
  • Ordinal pair-wise data organization.
expression recognition 3
Expression Recognition(3)
  • Ranking Model.

- sparse Rankboost (weak classifier)

lost function:

- Adaboost strong classifier

smile detection
Smile Detection

Illustration of synchrony for a smiling event (both the subject and the interviewer are smiling at about the same time instant)

smile synchrony
Smile Synchrony

Left: smiling scores curves for both the subject (in blue) and interviewer (in red). Here we see that there is synchrony.

Right: Snapshots from the actual tracked footage illustrating smiling synchrony.

module 3 head pose module nodding
Module 3: Head Pose Module: Nodding

A 3D head nodding sequence. The green lines show head pitch angle. The black lines show head yaw angle.

Head pose pitch curve along time axis. The pattern of the blue plot is characteristic of head nodding

head pose module shaking
Head Pose Module: Shaking

A 3D head shaking sequence. The green lines show head pitch angle. The black lines show head yaw angle.

head pose yaw curve. The pattern of the red plot is characteristic of a head shake.

head pitch synchrony nodding
Head Pitch Synchrony (nodding)

head pitch curves for the subject (in blue) and the interviewer (in red). The patterns are characteristic of head nodding. Here we see that there is synchrony (both plots show nodding pattern).

head yaw dissynchrony shaking
Head Yaw Dissynchrony (shaking)

head yaw curves for the subject (in blue) and the interviewer (in red). The distinct pattern of the red plot is characteristic of a head shake. This head shaking event is NOT in synchrony.

module 4 synchrony feature extraction cross correlation
Module 4: Synchrony Feature ExtractionCross correlation
  • Generate higher level synchrony feature from Lower level features
  • Correlation based strategy
synchrony feature extraction hit miss rate
Synchrony Feature Extractionhit-miss rate
  • Synchrony definition.
    • If event A is detected for subject X at time T and for subject Y at time T ± w  we say we have synchrony
  • Some attributes can be extracted from detected Synchrony:
    • 1. Smile_Hit_Subject_Rate_S(x): Rate of smiling synchrony (led by subject and followed by interviewer) in segment (x)
    • 2. Smile_Hit_Interviewer_Rate_S(x): Rate of smiling synchrony (led by interviewer and followed by subject)

Subject

Interviewer

Smile

Smile_Hit_Interviewer

Smile_Hit_Subject

Yes

Subject

Interviewer

Smile

Yes

No

Time

No

Time

synchrony feature extraction hit miss rate1
Synchrony Feature Extractionhit-miss rate

Subject

Interviewer

Smile

Yes

No

Time

Smile_Miss_Subject

Smile_Miss_Interviewer

Subject

Interviewer

Smile

Yes

No

Time

  • Some attributes can be extracted from detected Synchrony:
    • 3. Smile_Miss_Subject_Rate_S(x): The rate of no smiling synchrony (led by subject and not followed by interviewer) in segment (x)
    • 4.Smile_Miss_Interviewer_Rate_S(x): The rate of no smiling synchrony (led by subject and not followed by interviewer) in segment (x)
module 5 feature selection and classification
Module 5: Feature Selection and Classification
  • Feature Selection, Genetic Algorithm.
  • Classification

Two-class problem:

Truthful vs. Deceptive

Three-class problem:

Truthful, Sanctioned and Unsanctioned cheating

experimental databases
Experimental Databases
  • Computer-Mediated Communication Dataset (CMC)
  • Face to Face Communication Dataset (FtF)
evaluation of two class classification
Evaluation of Two-class Classification
  • Confusion Matrices
  • Performance
evaluation of three class classification
Evaluation of Three-class Classification
  • Confusion Matrices
  • Performance
conclusions
Conclusions
  • We investigated how the degree of synchrony effects the result of deception detection.
  • Automatic methods provide an important way to evaluate synchrony other than manual coding
  • Some observations:
    • Different lower level features contribute differently to the deception detection.
    • Modalities have subtle influence in detecting deception. (CMC vs. FtF)