Lip feature extraction using red exclusion
Sponsored Links
This presentation is the property of its rightful owner.
1 / 30

Lip Feature Extraction Using Red Exclusion PowerPoint PPT Presentation


  • 100 Views
  • Uploaded on
  • Presentation posted in: General

Lip Feature Extraction Using Red Exclusion. Trent W. Lewis and David M.W. Powers Flinders University of SA VIP2000. Overview. Context Lip Feature Extraction Related Work (greyscale, horizontal edges, red and hue colour spaces) Red Exclusion AVSR: Results and Issues Summary. Context.

Download Presentation

Lip Feature Extraction Using Red Exclusion

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Lip Feature Extraction Using Red Exclusion

Trent W. Lewis and David M.W. Powers

Flinders University of SA

VIP2000


Overview

  • Context

  • Lip Feature Extraction

    • Related Work (greyscale, horizontal edges, red and hue colour spaces)

    • Red Exclusion

  • AVSR: Results and Issues

  • Summary

Lip Feature Extraction Using Red Exlucsion


Context

  • Audio Speech Recognition (ASR)

  • Psycholinguistic Research

  • Audio Visual Speech Recognition (AVSR)

Lip Feature Extraction Using Red Exlucsion


Context - ASR

  • Up to 99% word accuracy

  • However,

    • limited context

    • limited vocabulary

    • trained on individual

    • close microphone, cannot handle noise

Lip Feature Extraction Using Red Exlucsion


Context - Psycholinguistic

  • McGurk Effect

    • A[ba] + V[ga]  [da]

  • Viseme

    • visual phonemes

    • form complementary sets

  • Demonstrates vision can assist the perception of speech  AVSR

Lip Feature Extraction Using Red Exlucsion


Acoustic Features

Visual Features

width

height

oral cavity

Integration

Early

Late

Context - AVSR

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Pixel-BasedModel

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Pixel-BasedModel

    • raw pixels or minimal processing

    • retain linguistically relevant data

    • large amounts of data,  time

    • shift and lighting variant

    • normalisation and PCA

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Pixel-BasedModel

    • reduced input to set of hand-crafted features

    • width, height, average intensity, etc.

    • less features,  time

    • model fitting,  time

    • lose linguistically relevant features

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

Our Approach

  • Pixel-BasedModel

    • feature extraction

    • Steps

      • preprocess to enhance contrast

      • locate mouth edges

      • identify corners, height, and other key features

      • train recogntion engine

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Database

1

2

3

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Preprocessing Techniques

    • Grey-scale

    • Horizontal Edges

    • Red Analysis

    • Hue, Saturation, and Value (HSV)

    • Red Exclusion

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Grey-scale

    • vertical position of mouth

      • minimum row sum

    • threshold minimum row

      • average of min and max of row

    • search for above threshold pixels

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Grey-scale

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Grey-scale

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Horizontal Edges

    • high horizontal edge content

    • 3x3, DY Prewitt operator

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Horizontal Edges

“Found” Corners

Binary Image

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Red Analysis

    • overcome bearded subjects

    • used for face location

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Red Analysis

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • HSV

    • disentangles illumination from colour

    • Illumination > Hue

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • HSV

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Red Exclusion

    • needed extraction method for AVSR

    • similar to Red Analysis

      • face predominantly red

      • variations occur in the blue and green colours

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Red Exclusion

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Corners found using Red Exclusion

Lip Feature Extraction Using Red Exlucsion


Lip Feature Extraction

  • Comparison

Lip Feature Extraction Using Red Exlucsion


AVSR: Results and Issues

  • Application for red exclusion

  • Used in finding lip features

    • Width

    • Height

    • Key pixels

Lip Feature Extraction Using Red Exlucsion


AVSR: Results and Issues

  • Visual Speech Recognition

Lip Feature Extraction Using Red Exlucsion


AVSR: Results and Issues

  • AVSR - Integration

Lip Feature Extraction Using Red Exlucsion


Summary

  • Vision can help ASR

    • AVSR

  • Needed good extraction technique

    • Red Exclusion

  • AVSR is difficult when both signals degraded

Lip Feature Extraction Using Red Exlucsion


Questions?

Trent W. Lewis

BSc (Cognitive Science)

Flinders University

[email protected]

Lip Feature Extraction Using Red Exlucsion


  • Login