Lip feature extraction using red exclusion
Download
1 / 30

Lip Feature Extraction Using Red Exclusion - PowerPoint PPT Presentation


  • 128 Views
  • Uploaded on

Lip Feature Extraction Using Red Exclusion. Trent W. Lewis and David M.W. Powers Flinders University of SA VIP2000. Overview. Context Lip Feature Extraction Related Work (greyscale, horizontal edges, red and hue colour spaces) Red Exclusion AVSR: Results and Issues Summary. Context.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Lip Feature Extraction Using Red Exclusion' - norm


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Lip feature extraction using red exclusion

Lip Feature Extraction Using Red Exclusion

Trent W. Lewis and David M.W. Powers

Flinders University of SA

VIP2000


Overview
Overview

  • Context

  • Lip Feature Extraction

    • Related Work (greyscale, horizontal edges, red and hue colour spaces)

    • Red Exclusion

  • AVSR: Results and Issues

  • Summary

Lip Feature Extraction Using Red Exlucsion


Context
Context

  • Audio Speech Recognition (ASR)

  • Psycholinguistic Research

  • Audio Visual Speech Recognition (AVSR)

Lip Feature Extraction Using Red Exlucsion


Context asr
Context - ASR

  • Up to 99% word accuracy

  • However,

    • limited context

    • limited vocabulary

    • trained on individual

    • close microphone, cannot handle noise

Lip Feature Extraction Using Red Exlucsion


Context psycholinguistic
Context - Psycholinguistic

  • McGurk Effect

    • A[ba] + V[ga]  [da]

  • Viseme

    • visual phonemes

    • form complementary sets

  • Demonstrates vision can assist the perception of speech  AVSR

Lip Feature Extraction Using Red Exlucsion


Context avsr

Acoustic Features

Visual Features

width

height

oral cavity

Integration

Early

Late

Context - AVSR

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction
Lip Feature Extraction

  • Pixel-Based Model

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction1
Lip Feature Extraction

  • Pixel-Based Model

    • raw pixels or minimal processing

    • retain linguistically relevant data

    • large amounts of data,  time

    • shift and lighting variant

    • normalisation and PCA

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction2
Lip Feature Extraction

  • Pixel-Based Model

    • reduced input to set of hand-crafted features

    • width, height, average intensity, etc.

    • less features,  time

    • model fitting,  time

    • lose linguistically relevant features

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction3
Lip Feature Extraction

Our Approach

  • Pixel-Based Model

    • feature extraction

    • Steps

      • preprocess to enhance contrast

      • locate mouth edges

      • identify corners, height, and other key features

      • train recogntion engine

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction4
Lip Feature Extraction

  • Database

1

2

3

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction5
Lip Feature Extraction

  • Preprocessing Techniques

    • Grey-scale

    • Horizontal Edges

    • Red Analysis

    • Hue, Saturation, and Value (HSV)

    • Red Exclusion

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction6
Lip Feature Extraction

  • Grey-scale

    • vertical position of mouth

      • minimum row sum

    • threshold minimum row

      • average of min and max of row

    • search for above threshold pixels

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction7
Lip Feature Extraction

  • Grey-scale

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction8
Lip Feature Extraction

  • Grey-scale

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction9
Lip Feature Extraction

  • Horizontal Edges

    • high horizontal edge content

    • 3x3, DY Prewitt operator

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction10
Lip Feature Extraction

  • Horizontal Edges

“Found” Corners

Binary Image

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction11
Lip Feature Extraction

  • Red Analysis

    • overcome bearded subjects

    • used for face location

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction12
Lip Feature Extraction

  • Red Analysis

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction13
Lip Feature Extraction

  • HSV

    • disentangles illumination from colour

    • Illumination > Hue

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction14
Lip Feature Extraction

  • HSV

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction15
Lip Feature Extraction

  • Red Exclusion

    • needed extraction method for AVSR

    • similar to Red Analysis

      • face predominantly red

      • variations occur in the blue and green colours

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction16
Lip Feature Extraction

  • Red Exclusion

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction17
Lip Feature Extraction

  • Corners found using Red Exclusion

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction18
Lip Feature Extraction

  • Comparison

Lip Feature Extraction Using Red Exlucsion


Avsr results and issues
AVSR: Results and Issues

  • Application for red exclusion

  • Used in finding lip features

    • Width

    • Height

    • Key pixels

Lip Feature Extraction Using Red Exlucsion


Avsr results and issues1
AVSR: Results and Issues

  • Visual Speech Recognition

Lip Feature Extraction Using Red Exlucsion


Avsr results and issues2
AVSR: Results and Issues

  • AVSR - Integration

Lip Feature Extraction Using Red Exlucsion


Summary
Summary

  • Vision can help ASR

    • AVSR

  • Needed good extraction technique

    • Red Exclusion

  • AVSR is difficult when both signals degraded

Lip Feature Extraction Using Red Exlucsion


Questions
Questions?

Trent W. Lewis

BSc (Cognitive Science)

Flinders University

[email protected]

Lip Feature Extraction Using Red Exlucsion


ad