Asl2txt
This presentation is the property of its rightful owner.
Sponsored Links
1 / 80

ASL2TXT PowerPoint PPT Presentation


  • 133 Views
  • Uploaded on
  • Presentation posted in: General

ASL2TXT. Converting sign language gestures from digital images to text George Corser. Presentation Overview. Concept Foundation : Barkoky & Charkari (2011) Segmentation Thinning My Contribution : Corser ( 2012) Segmentation (similar to Barkoky )

Download Presentation

ASL2TXT

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Asl2txt

ASL2TXT

Converting sign language gestures from digital images to text

George Corser


Presentation overview

Presentation Overview

  • Concept

  • Foundation: Barkoky& Charkari (2011)

    • Segmentation

    • Thinning

  • My Contribution: Corser (2012)

    • Segmentation (similar to Barkoky)

    • CED: Canny Edge Dilation (Minus Errors)

    • Assumption: User trains his own phone


Concept

Concept

  • Deaf and hearing people talking on the phone, each using their natural language

  • Sign-activated commands like voice-activated


Situation drive thru window

Situation: Drive Thru Window

Think:

Stephen Hawking

Deaf person signs order

Phone speaks order

Confirmation on screen


Process flow

Process Flow

  • Requires several conversion processes

  • Many have been accomplished

  • Remaining: ASL2TXT


Goal find an algorithm

Goal: Find an Algorithm

  • Find an image processing algorithm that recognizes ASL alphabet

= A

Web site


Barkoky segmentation thinning

Barkoky: Segmentation & Thinning

Barkoky counts

endpoints to

determine sign

(doesn’t work for ASL)


Barkoky process

Barkoky Process

Segmentation

Thinning

Input: hand segment

Apply thinning

Find endpoints, joints

Calculate lengths

Clean short lengths

Identify gesture by counting endpoints

  • Capture RGB image

  • Rescale

  • Extract using colors

  • Reduce noise

  • Crop at wrist

  • Result: hand segment


1 capture rgb image 2 rescale

1. Capture RGB Image2. Rescale

% ---------- 1. Capture RGB image

a = imread('DSC04926.JPG');

figure('Name','RGB image'),imshow(a);

% ---------- 2. Rescale image to 205x154

a10 = imresize(a, 0.1);

figure('Name','Rescaled image'),imshow(a10);


3 extract hand using colors

3. Extract Hand Using Colors

% ---------- 3. Extract hand using color

abw10 = zeros(205,154,1);

for i=1:205,

for j=1:154,

if a10(i,j,2)<140 && a10(i,j,3)<100,

abw10(i,j,1)=255;

end;

end;

end;

figure('Name','Extracted'),imshow(abw10);

Note: Color threshold code

differs from Barkoky


Colors training set histograms

Colors: Training Set Histograms


Colors training set 2

Colors: Training Set (2)

Red

Green

Blue

Excel


Colors test set histograms

Colors: Test Set Histograms


4 reduce noise

4. Reduce Noise

% ---------- 4. Reduce noise

for i=2:204, for j=1:154,

if abw10(i-1,j,1)==0

if abw10(i+1,j,1)==0,

abw10(i,j,1)=0; end; end;

if abw10(i-1,j,1)==255

if abw10(i+1,j,1)==255,

abw10(i,j,1)=0; end; end;

end; end;

abw10 = imfill(abw10,'holes');


5 identify wrist position

5. Identify Wrist Position

% ---------- 5. Identify wrist position

for i=204:-1:1,

for j=1:154,

if abw10(i,j,1)==255, break; end;

end;

if j ~= 154 && abw10(i+1,j,1)~=255,

wristi=i+1;

wristj=j+1;

break;

end;

end;


Wrist detection

Wrist Detection

  • Algorithm searches bottom-to-top of image

  • Finds a leftmost white pixel above black pixel

  • Sets wrist position SE of found white pixel


Corser segmentation ced

Corser: Segmentation & CED

  • Segmentation (similar to Barkoky)

    • Color threshold technique slightly different

    • American Sign Language (ASL) alphabet, not Persian Sign Language (PSL) numbers

  • Image Comparison: Tried Several Methods

    • Full Threshold (Minus Errors)

    • Diced Segments (Minus Errors)

    • Endpoint Count Difference

    • CED: Canny Edge Dilation


Asl training set

ASL Training Set

Hit-or-miss: 23%

Barkoky: 8%


Asl test set

ASL Test Set

MATLAB


Asl2txt

A


Asl2txt

A


Asl2txt

B


Asl2txt

B


Asl2txt

C


Asl2txt

C


Asl2txt

D


Asl2txt

D


Asl2txt

E


Asl2txt

E


Asl2txt

F


Asl2txt

F


Asl2txt

G


Asl2txt

G


Asl2txt

H


Asl2txt

H


Asl2txt

I


Asl2txt

I


Asl2txt

J


Asl2txt

J


Asl2txt

K


Asl2txt

K


Asl2txt

L


Asl2txt

L


Asl2txt

M


Asl2txt

M


Asl2txt

N


Asl2txt

N


Asl2txt

O


Asl2txt

O


Asl2txt

P


Asl2txt

P


Asl2txt

Q


Asl2txt

Q


Asl2txt

R


Asl2txt

R


Asl2txt

S


Asl2txt

S


Asl2txt

T


Asl2txt

T


Asl2txt

U


Asl2txt

U


Asl2txt

V


Asl2txt

V


Asl2txt

W


Asl2txt

W


Asl2txt

X


Asl2txt

X


Asl2txt

Y


Asl2txt

Y


Asl2txt

Z


Asl2txt

Z


Asl2txt

Z


Hybrid algorithm example

Hybrid Algorithm Example

% ---------- MATLAB Code -------------------

matchtotal = 0;

if abs(x10range - x20range) < 20,

matchtotal = matchtotal + 10;

end;

if abs(y10range - y20range) < 20,

matchtotal = matchtotal + 11;

end;

matchtotal = matchtotal - abs(h10 - h20);

% ----- h10, h20 are vector magnitudes -----


Erosion subtraction

Erosion Subtraction


Canny edge

Canny Edge


Canny edge dilation code

Canny Edge Dilation Code

% ---------- MATLAB Code -------------------

se = strel('disk',5);

a10 = edge(a10,'canny');

a20 = edge(a20,'canny');

a10 = imdilate(a10,se);

a20 = imdilate(a20,se);

% ----- Then calculate matches minus errors


Experimental results

Experimental Results


Disadvantages

Disadvantages

  • Dependent on lighting conditions

  • Fails with flesh-tone backgrounds

  • Requires calibration to a specific user

  • Limited applications: text messaging, activation (“sign” similar to voice activation)

  • ASL numbers (A=10, D=1, O=0, V=2, W=6)

  • Alphabet is tiny portion of full translation: complete translation maybe many years away


Future work

Future Work

  • Barkoky claims flesh tones can be detected, but I have yet to replicate (even Barkoky changed his color detection scheme)

  • Could write letter-by-letter algorithm

  • Could use range camera to compute distance of finger instead of shape of hand

  • Motion analysis or edge count

  • Many possibilities… we’ve only just begun!

Cue: music

http://www.youtube.com/watch?v=__VQX2Xn7tI


The end

The End


  • Login