This presentation is the property of its rightful owner.
1 / 80

# ASL2TXT PowerPoint PPT Presentation

ASL2TXT. Converting sign language gestures from digital images to text George Corser. Presentation Overview. Concept Foundation : Barkoky & Charkari (2011) Segmentation Thinning My Contribution : Corser ( 2012) Segmentation (similar to Barkoky )

ASL2TXT

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

## ASL2TXT

Converting sign language gestures from digital images to text

George Corser

### Presentation Overview

• Concept

• Foundation: Barkoky& Charkari (2011)

• Segmentation

• Thinning

• My Contribution: Corser (2012)

• Segmentation (similar to Barkoky)

• CED: Canny Edge Dilation (Minus Errors)

• Assumption: User trains his own phone

### Concept

• Deaf and hearing people talking on the phone, each using their natural language

• Sign-activated commands like voice-activated

### Situation: Drive Thru Window

Think:

Stephen Hawking

Deaf person signs order

Phone speaks order

Confirmation on screen

### Process Flow

• Requires several conversion processes

• Many have been accomplished

• Remaining: ASL2TXT

### Goal: Find an Algorithm

• Find an image processing algorithm that recognizes ASL alphabet

= A

Web site

### Barkoky: Segmentation & Thinning

Barkoky counts

endpoints to

determine sign

(doesn’t work for ASL)

### Barkoky Process

Segmentation

Thinning

Input: hand segment

Apply thinning

Find endpoints, joints

Calculate lengths

Clean short lengths

Identify gesture by counting endpoints

• Capture RGB image

• Rescale

• Extract using colors

• Reduce noise

• Crop at wrist

• Result: hand segment

### 1. Capture RGB Image2. Rescale

% ---------- 1. Capture RGB image

figure('Name','RGB image'),imshow(a);

% ---------- 2. Rescale image to 205x154

a10 = imresize(a, 0.1);

figure('Name','Rescaled image'),imshow(a10);

### 3. Extract Hand Using Colors

% ---------- 3. Extract hand using color

abw10 = zeros(205,154,1);

for i=1:205,

for j=1:154,

if a10(i,j,2)<140 && a10(i,j,3)<100,

abw10(i,j,1)=255;

end;

end;

end;

figure('Name','Extracted'),imshow(abw10);

Note: Color threshold code

differs from Barkoky

Red

Green

Blue

Excel

### 4. Reduce Noise

% ---------- 4. Reduce noise

for i=2:204, for j=1:154,

if abw10(i-1,j,1)==0

if abw10(i+1,j,1)==0,

abw10(i,j,1)=0; end; end;

if abw10(i-1,j,1)==255

if abw10(i+1,j,1)==255,

abw10(i,j,1)=0; end; end;

end; end;

abw10 = imfill(abw10,'holes');

### 5. Identify Wrist Position

% ---------- 5. Identify wrist position

for i=204:-1:1,

for j=1:154,

if abw10(i,j,1)==255, break; end;

end;

if j ~= 154 && abw10(i+1,j,1)~=255,

wristi=i+1;

wristj=j+1;

break;

end;

end;

### Wrist Detection

• Algorithm searches bottom-to-top of image

• Finds a leftmost white pixel above black pixel

• Sets wrist position SE of found white pixel

### Corser: Segmentation & CED

• Segmentation (similar to Barkoky)

• Color threshold technique slightly different

• American Sign Language (ASL) alphabet, not Persian Sign Language (PSL) numbers

• Image Comparison: Tried Several Methods

• Full Threshold (Minus Errors)

• Diced Segments (Minus Errors)

• Endpoint Count Difference

• CED: Canny Edge Dilation

Hit-or-miss: 23%

Barkoky: 8%

MATLAB

### Hybrid Algorithm Example

% ---------- MATLAB Code -------------------

matchtotal = 0;

if abs(x10range - x20range) < 20,

matchtotal = matchtotal + 10;

end;

if abs(y10range - y20range) < 20,

matchtotal = matchtotal + 11;

end;

matchtotal = matchtotal - abs(h10 - h20);

% ----- h10, h20 are vector magnitudes -----

### Canny Edge Dilation Code

% ---------- MATLAB Code -------------------

se = strel('disk',5);

a10 = edge(a10,'canny');

a20 = edge(a20,'canny');

a10 = imdilate(a10,se);

a20 = imdilate(a20,se);

% ----- Then calculate matches minus errors

### Experimental Results

• Dependent on lighting conditions

• Fails with flesh-tone backgrounds

• Requires calibration to a specific user

• Limited applications: text messaging, activation (“sign” similar to voice activation)

• ASL numbers (A=10, D=1, O=0, V=2, W=6)

• Alphabet is tiny portion of full translation: complete translation maybe many years away

### Future Work

• Barkoky claims flesh tones can be detected, but I have yet to replicate (even Barkoky changed his color detection scheme)

• Could write letter-by-letter algorithm

• Could use range camera to compute distance of finger instead of shape of hand

• Motion analysis or edge count

• Many possibilities… we’ve only just begun!

Cue: music