Sign language recognition using webcams
This presentation is the property of its rightful owner.
Sponsored Links
1 / 16

Sign Language Recognition using Webcams PowerPoint PPT Presentation


  • 213 Views
  • Uploaded on
  • Presentation posted in: General

Sign Language Recognition using Webcams. Overview. Average person’s typing speed Composing: ~19 words per minute Transcribing: ~33 words per minute Sign speaker Full sign language: ~200 words per minute Spelling out: estimate: 50 words per minute Up to 3x faster. Purpose and Scope.

Download Presentation

Sign Language Recognition using Webcams

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Sign language recognition using webcams

Byron Hood | version 0.4

Computer Systems Lab Project2007-2008

Sign Language Recognition using Webcams


Overview

Byron Hood | version 0.4

Computer Systems Lab Project2007-2008

Overview

  • Average person’s typing speed

    • Composing: ~19 words per minute

    • Transcribing: ~33 words per minute

  • Sign speaker

    • Full sign language: ~200 words per minute

    • Spelling out: estimate: 50 words per minute

    • Up to 3x faster


Purpose and scope

Byron Hood | version 0.4

Computer Systems Lab Project2007-2008

Purpose and Scope

  • Native signers can input faster

  • Benefits:

    • Hearing & speaking disabled

    • Sign interpreters

  • Just letters & numbers for now

    • Additional complexity too much to handle

    • Would require smaller distinctions


Research

Byron Hood | version 0.4

Computer Systems Lab Project2007-2008

Research

  • Related projects

    • Using mechanical gloves, colored gloves

    • Tracking body parts

    • Neural network-based application

      • Still images: 92% accuracy

      • Motion: less than 50% accuracy

    • Feature vector-based application

      • Also about 90% accuracy on stills

      • No motion tests


More research

Byron Hood | version 0.4

Computer Systems Lab Project2007-2008

More Research

  • Image techniques:

    • Edge detection (Robert’s Cross)‏

    • Line detection (Hough transform)‏

    • Line interpretation methods

      • Chaining groups of lines

      • Macro-scale templates

    • Residual math

  • Memory management


Testing model

Byron Hood | version 0.4

Computer Systems Lab Project2007-2008

Testing Model

  • Human interaction necessary

  • General testing model:

    [email protected]:~/syslab-tech$ \

    > ./main images/hand.png

    [DEBUG] Edge detect time: 29 ms

    Errors:0Warnings:0


Program architecture

Byron Hood | version 0.4

Computer Systems Lab Project2007-2008

Program Architecture

SERVER PROCESS

Webcam capture

Edge detection

Line detection

Interpretation

Attribute matching

IMAGE

FEATURE OUTLINE

LINE LIST

FINGER POSITONS


Edge detection results

Byron Hood | version 0.4

Computer Systems Lab Project2007-2008

Edge Detection Results

  • Results:

    • Outlines the important edges and not much besides

    • Robert’s Cross balances detection of major and minor lines

Original image

(800 x 703)‏

Final image

(800 x 703)‏


Cropping results

Byron Hood | version 0.4

Computer Systems Lab Project2007-2008

Cropping Results

  • Remove useless rows & columns with no features

  • Better contrast

  • Very large optimization

    • Memory savings

    • Area difference means order n2

Original

(800 x 703)‏

Result

(633 x 645)‏


Line detection

Byron Hood | version 0.4

Computer Systems Lab Project2007-2008

Line detection

  • Finished!

    • Recently finished tweaking sensitivities

    • Still a few potential memory issues


Line grouping

Byron Hood | version 0.4

Computer Systems Lab Project2007-2008

Line Grouping

  • Part of line detection

  • Large optimization

    • Iterate over an order of magnitude fewer items

    • Easier to handle, more pronounced trends

Examples of line groups, called “chains”


Line interpretation

Byron Hood | version 0.4

Computer Systems Lab Project2007-2008

Line Interpretation

  • Chaining groups of lines

  • Templates

    • Generation

    • Template-based comparison

  • Line residuals

    • Use point coordinate averages

    • Calculate average offset from average

    • Easy to find height of finger


Sample output

Byron Hood | version 0.4

Computer Systems Lab Project2007-2008

Sample Output

  • After a typical run:

    10days,6:12:19until graduation!!

    [email protected]~/syslab-tech/src$ ./main hand.png

    Edge detection took 0.04 sec

    Image cropping took 0.00 sec

    Line detection took 0.17 sec (detected 1424 lines)‏

    Line chaining took 0.25 sec (detected 130 chains)‏

    Getting orientation took -0.08 src (1 => ORIENTATION_FORWARD)‏

    Getting pinky pos. took 0.00 sec (2 => FINGER_BENT)‏

    Getting ring pos. took 0.00 sec (2 => FINGER_BENT)‏

    Getting middle pos. took 0.01 sec (2 => FINGER_BENT)‏

    Getting index pos. took -0.00 sec (4 => FINGER_TUCKED)‏

    Overall process took 0.47 sec

    [TOTALCOUNT] allocated: 10718901, freed: 10364880; leaked: 354021.


Timing

Byron Hood | version 0.4

Computer Systems Lab Project2007-2008

Timing

  • Timing data from runs:

    • To nearest hundredth of a second

      • Edge detection: 0.04 sec

      • Image cropping: 0.00 sec

      • Line detection: 0.17 sec

      • Line chaining: 0.25 sec

      • orientation: 0.08 sec

      • Pinky finger: 0.00 sec

      • Ring finger: 0.00 sec

      • Middle finger: 0.01 sec

      • Index finger: 0.00 sec

      • Overall process: 0.47 sec

  • A little slow considering goal of real-time


  • The mysterious future

    Byron Hood | version 0.4

    Computer Systems Lab Project2007-2008

    The Mysterious Future

    • Perfect line interpretation

    • Work on memory management

      • Am leaking large quantities (~50K) of memory

      • Aggressive profiling needed

    • Finish camera-computer interaction

      • Device control must be precise, picky


    The end

    Byron Hood | version 0.4

    Computer Systems Lab Project2007-2008

    The End!

    • Code will be available to future years

    • Contact me for a copy:

      [email protected]


  • Login