Lip feature extraction using red exclusion
This presentation is the property of its rightful owner.
Sponsored Links
1 / 30

Lip Feature Extraction Using Red Exclusion PowerPoint PPT Presentation


  • 91 Views
  • Uploaded on
  • Presentation posted in: General

Lip Feature Extraction Using Red Exclusion. Trent W. Lewis and David M.W. Powers Flinders University of SA VIP2000. Overview. Context Lip Feature Extraction Related Work (greyscale, horizontal edges, red and hue colour spaces) Red Exclusion AVSR: Results and Issues Summary. Context.

Download Presentation

Lip Feature Extraction Using Red Exclusion

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Lip feature extraction using red exclusion

Lip Feature Extraction Using Red Exclusion

Trent W. Lewis and David M.W. Powers

Flinders University of SA

VIP2000


Overview

Overview

  • Context

  • Lip Feature Extraction

    • Related Work (greyscale, horizontal edges, red and hue colour spaces)

    • Red Exclusion

  • AVSR: Results and Issues

  • Summary

Lip Feature Extraction Using Red Exlucsion


Context

Context

  • Audio Speech Recognition (ASR)

  • Psycholinguistic Research

  • Audio Visual Speech Recognition (AVSR)

Lip Feature Extraction Using Red Exlucsion


Context asr

Context - ASR

  • Up to 99% word accuracy

  • However,

    • limited context

    • limited vocabulary

    • trained on individual

    • close microphone, cannot handle noise

Lip Feature Extraction Using Red Exlucsion


Context psycholinguistic

Context - Psycholinguistic

  • McGurk Effect

    • A[ba] + V[ga]  [da]

  • Viseme

    • visual phonemes

    • form complementary sets

  • Demonstrates vision can assist the perception of speech  AVSR

Lip Feature Extraction Using Red Exlucsion


Context avsr

Acoustic Features

Visual Features

width

height

oral cavity

Integration

Early

Late

Context - AVSR

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction

Lip Feature Extraction

  • Pixel-BasedModel

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction1

Lip Feature Extraction

  • Pixel-BasedModel

    • raw pixels or minimal processing

    • retain linguistically relevant data

    • large amounts of data,  time

    • shift and lighting variant

    • normalisation and PCA

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction2

Lip Feature Extraction

  • Pixel-BasedModel

    • reduced input to set of hand-crafted features

    • width, height, average intensity, etc.

    • less features,  time

    • model fitting,  time

    • lose linguistically relevant features

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction3

Lip Feature Extraction

Our Approach

  • Pixel-BasedModel

    • feature extraction

    • Steps

      • preprocess to enhance contrast

      • locate mouth edges

      • identify corners, height, and other key features

      • train recogntion engine

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction4

Lip Feature Extraction

  • Database

1

2

3

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction5

Lip Feature Extraction

  • Preprocessing Techniques

    • Grey-scale

    • Horizontal Edges

    • Red Analysis

    • Hue, Saturation, and Value (HSV)

    • Red Exclusion

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction6

Lip Feature Extraction

  • Grey-scale

    • vertical position of mouth

      • minimum row sum

    • threshold minimum row

      • average of min and max of row

    • search for above threshold pixels

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction7

Lip Feature Extraction

  • Grey-scale

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction8

Lip Feature Extraction

  • Grey-scale

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction9

Lip Feature Extraction

  • Horizontal Edges

    • high horizontal edge content

    • 3x3, DY Prewitt operator

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction10

Lip Feature Extraction

  • Horizontal Edges

“Found” Corners

Binary Image

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction11

Lip Feature Extraction

  • Red Analysis

    • overcome bearded subjects

    • used for face location

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction12

Lip Feature Extraction

  • Red Analysis

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction13

Lip Feature Extraction

  • HSV

    • disentangles illumination from colour

    • Illumination > Hue

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction14

Lip Feature Extraction

  • HSV

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction15

Lip Feature Extraction

  • Red Exclusion

    • needed extraction method for AVSR

    • similar to Red Analysis

      • face predominantly red

      • variations occur in the blue and green colours

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction16

Lip Feature Extraction

  • Red Exclusion

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction17

Lip Feature Extraction

  • Corners found using Red Exclusion

Lip Feature Extraction Using Red Exlucsion


Lip feature extraction18

Lip Feature Extraction

  • Comparison

Lip Feature Extraction Using Red Exlucsion


Avsr results and issues

AVSR: Results and Issues

  • Application for red exclusion

  • Used in finding lip features

    • Width

    • Height

    • Key pixels

Lip Feature Extraction Using Red Exlucsion


Avsr results and issues1

AVSR: Results and Issues

  • Visual Speech Recognition

Lip Feature Extraction Using Red Exlucsion


Avsr results and issues2

AVSR: Results and Issues

  • AVSR - Integration

Lip Feature Extraction Using Red Exlucsion


Summary

Summary

  • Vision can help ASR

    • AVSR

  • Needed good extraction technique

    • Red Exclusion

  • AVSR is difficult when both signals degraded

Lip Feature Extraction Using Red Exlucsion


Questions

Questions?

Trent W. Lewis

BSc (Cognitive Science)

Flinders University

[email protected]

Lip Feature Extraction Using Red Exlucsion


  • Login