my phd project l.
Skip this Video
Loading SlideShow in 5 Seconds..
My PhD Project PowerPoint Presentation
Download Presentation
My PhD Project

Loading in 2 Seconds...

play fullscreen
1 / 22

My PhD Project - PowerPoint PPT Presentation

  • Uploaded on

My PhD Project. Quantifying Visual Cues of Psychological States in Interview Situations (QVCPS). 18/05/2005 Enrica Dente Email: Website: Contents. Problem Hypothesis Project Objectives Project Plan Achievements to Date

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'My PhD Project' - betty_james

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
my phd project

MyPhD Project

Quantifying Visual Cues of Psychological States in Interview Situations (QVCPS)


Enrica Dente



  • Problem
  • Hypothesis
  • Project Objectives
  • Project Plan
  • Achievements to Date
  • Conclusion

Enrica Dente, QVCPS Project

  • Most lie detection approaches can only detect “psychological states”, not “deception” (i.e. are less than 80% accuracy) [Vrij, 2004]
  • Manual observation of human behavior subjective and limited [Burgoon, 2005]
  • Polygraph, 100 years old lie detector problematic but still in use
  • Increased interest in alternative approaches:
    • Automated Face Analysis (i.e. changes in facial expressions)
    • Thermal imaging (i.e. changes in blood flow around eyes)
    • FMRI (i.e. BOLD)
    • Brain fingerprinting (i.e. brain waves)
    • Eye tracking (i.e. irregular eye movements in front of images )
  • Intrusiveness yet to be addressed.

Enrica Dente, QVCPS Project

  • Non skilled liars more likely to experience cognitive load and controlled behavior[Vrij, 2004]
  • Decrease of hand/finger movements when subjects lie yet to be proved (or disproved)
  • Quantification of observations required
  • Focus on tracking of hand/finger movements in interview situations
  • Real life cases can make a difference in results but problematic [Vrij 2005]
  • Verbal behavior required to add context to the tracking

Enrica Dente, QVCPS Project

project objectives
Project Objectives
  • To automate the tracking of finger/hand movements in interview situations
  • Tocompute the frequency of hand and finger movements normalised to the baseline of each individual
  • To build a biomechanical model to aid hand gesture tracking
  • To provide a simple means of accessing and referring to the verbal behaviour, when required
  • Later, to compare the movements of the interviewee with the movements of the interviewer.

Enrica Dente, QVCPS Project

project plan
Project Plan
  • Year 1:
    • Skin Colour Modelling
    • Hand Tracking and Finger Position
    • Coding behaviour markup
    • Background Modelling
    • Graphical User Interface
    • Experimental Protocol
  • Year 2 and Year 3:
    • Kinematic models of upper body, limb and finger motion
    • Introduction of verbal behavior to add context to tracking
    • Train system to quantify different types of hand and finger movements
    • Writing up.

Enrica Dente, QVCPS Project

achievements to date 1 segmentation
Achievements to Date: 1) Segmentation

(a) Skin Colour Modelling

    • Color Predicate[Kieldsen, 1996]
    • Bayesian Posterior Map and Parzen colour space probability density estimates more accurate.

(b) Hand Tracking and Finger Position

    • Connected Component and central moments
    • Centroid correspondence
    • Hands separation based on area and distance heuristics
    • Complex wavelet decomposition to detect changes in finger position.
  • Next Step: background modelling using parametric methods to address robustness.

Figure 1 -Hand Tracking

Figure 2 -Finger Orientations

Enrica Dente, QVCPS Project

a skin colour modelling
(a) Skin Colour Modelling
  • Joint Conditional Density function in hue and sat for hand and non-hands (i.e. Parzen density estimation)

where H and S represent hue and sat, Hn and Sn represent the observed values amongst the NH hands training set pixels, KH is a normalised constant and σ0 species the width of the kernel used in smoothing

  • Bayesian Posterior Map to identify most probable locations of skin in each frame

where we assume that p(H,S | Hands) is approximated by the left hand side of Equation (1) and p(H,S | NonHands) is approximated by the left hand side of Equation (2).



Enrica Dente, QVCPS Project

a skin colour modelling cont

Figure 3 - Likelihood function for hand regions in a), for non-hand regions in b)

(a) Skin Colour Modelling (cont.)
  • Parzen colour space probability density estimates for hand and for non-hand regions
  • There is overlap between the two classes

Enrica Dente, QVCPS Project

b hand tracking and finger position
(b) Hand Tracking and Finger Position


  • Centroid correspondence of each object across frames from binary image created in (a)
  • Connected component analysis to label blobs.
  • Switch from a one-hand to two-hand state by using distance and area heuristics.


  • Complex wavelet decomposition
  • Orientation of local image structure relatively invariant to local phase
  • Filter bank of complex wavelets, each tuned to one of four directions
  • We compare output of four complex masks to make estimate of the presence of a line or an edge in each direction
  • Vector indicating the direction of local image structure estimated by a weighted vector summation operation
  • Multirate scheme (i.e. series of different scales).

Enrica Dente, QVCPS Project

finger position estimate
Finger Position Estimate
  • K/2 of the real and imaginary impulse responses of wavelets at scale 1 for K = 8
  • Orientation field O(l) (m, n) is defined by:

Figure 4 -WaveletKernels

Enrica Dente, QVCPS Project

finger position estimate cont

Figure 5 (a) Hands together (b) Hands moving

Finger Position Estimate (.cont)
  • Histogram of orientations for each hand in each frame
  • Weighting of the product of the orientation
  • Field magnitude and posterior skin tone map to appropriate histogram bin
  • Orientation density functions can be used to detect finger movement!

Enrica Dente, QVCPS Project

achievements to date 2 xml markup for behavior coding
Achievements to Date:2)XML Markup for Behavior Coding

Check Verbal behavior:

Check Non Verbal behavior:

Enrica Dente, QVCPS Project

achievements to date 4 experimental protocol
Achievements to Date:4) Experimental Protocol
  • Stakes low in current lie detection experiments:
    • Interviewees are asked to lie on simulated actions
    • Interviewees less motivated to lie than in real-life situations, performance less effective
    • Vrij’s real-life police interview limited to visual observation of one liar
    • Problem: questions based on prior knowledge of what the interviewees will lie on and when
  • Our experimental protocol aims to raise the stakes of deception detection in interview situations:
    • Solution: design experiments where interviewees are not told in advance what they have to lie on.

Enrica Dente, QVCPS Project

literature review 1 skin colour modelling
Literature Review1) Skin Colour Modelling
  • Non Parametric Methods
    • Create an histogram or lookup table (LUT) common and simple
    • However, storage required, bins may miss pixel values, noise removal and occlusion still an issue
    • Bayesian posterior map computed from “hands” and non “hands” histograms more promising than Colour Predicate
  • Parametric Methods
    • Model skin colour as components of mixtures of Gaussians
    • Provide robustness, no need for storage space and ability to interpolate or generalise the training data
    • However, they require accurate initialization and assume number of components to be known in advance

Enrica Dente, QVCPS Project

literature review 1 skin colour modelling17
Literature Review1) Skin Colour Modelling
  • Parametric Methods (cont.)
    • Foreground and background modelled by a joint probability density function
    • Model parameters (i.e. means and covariance) for the Gaussians estimated from the training data using maximum likelihood or Bayesian inference
    • Updated online based on colour, on position and/or on motion information
    • Skin probability computed from Gaussian Probability Density Functions (PDF's)
    • However, in interview situations update cannot be based on assumption that foreground is moving.

Enrica Dente, QVCPS Project

literature review 2 kinematic 3 d models
Literature Review2) Kinematic 3-D models
  • Need accurate tracking of hand/finger gestures
  • Choice of degrees of freedom crucial
  • Choice of type of model (i.e. stick figure or statistical)
  • Need markers for validation
  • Need simple and consistent movements
  • Does deception occur fast or slow in interviews?
  • How can we track several movements at the same time? Number of cameras?
  • How visible can hand's edges be in interviews? Camera resolution?

Enrica Dente, QVCPS Project

  • Problem and Hypothesis:
    • Most lie detection approaches detect “psychological states”, not “deception”
    • Non skilled liars more likely to experience cognitive load and controlled behavior
    • Quantification of observations required
  • Achievements to Date:
    • Hand Tracking using Bayesian Posterior Map
    • Finger Position using Complex Wavelets
    • XML markup for correlating visual and verbal behavior
    • Experimental Protocol
  • Next Step:
    • Background modelling to address robustness.

Enrica Dente, QVCPS Project


Enrica Dente, QVCPS Project

  • [Vrij, 2004] Vrij Aldert, Psychology, Crime and Law, Challenging Interviewees During Interviews, The Potential Effects on Lie Detection, 2004
  • [Vrij2, 2004] Vrij Aldert, Why Professionals fail to catch liars and how they can improve, Legal and Criminological Psychology, 9, 159-181, 2004
  • [Vrij, 2002] Aldert Vrj, Detecting Lies and Deceit : The Psychology of Lying and the Implications for Professional Practice, John Wiley & Sons, Inc
  • [Watson, 1081] Watson, K. W. Oral and written linguistic indices of deception during employment interviews. (Doctoral dissertation, Louisiana State University, 1981). Dissertations Abstracts International, 42, 06A 2367.
  • [Parker et al., 2000] Parker, A. & Brown, J., Detection of deception: Statement Validity Analysis as a means of determining truthfulness or falsity of rape allegations. Legal and Criminological Psychology, 5, 237-259, 2000

Enrica Dente, QVCPS Project

references cont
References (cont.)
  • [Bradley, 1993] Bradley MT, Cullen MC, Polygraph lie detection on real events in a laboratory setting, 76(3 Pt 1):1051-8, Percept Mot Skills. June 1993
  • [Etcoff, 2000] Nancy L. Etcoff*, Paul Ekman, John J. Magee, Mark G. Frank, NATURE , VOL 405, 11 MAY 2000, Lie detection and language comprehension, 2000 Macmillan Magazines Ltd, Available Online:
  • [Kieldsen, 1996] R. Kjeldsen and J. Kender. Finding skin in color images. In Second International Conference on Automatic Face and Gesture Recognition, 1996.
  • [Burgoon, 2005] J. Burgoon et al. An approach for intent identi-
  • cation by building on deception detection. In Proc. of the 38th Annual Hawaii Int. Conf. on Detection of Deception: Collaboration Systems and Technology, 2005.
  • [Bharath, 2003] A. Bharath and J. Ng. A Steerable Complex Wavelet Construction and Its Application to Image Denoising. IEEE Transactions on Image Processing. June 2003

Enrica Dente, QVCPS Project