asl2txt n.
Download
Skip this Video
Download Presentation
ASL2TXT

Loading in 2 Seconds...

play fullscreen
1 / 80

ASL2TXT - PowerPoint PPT Presentation


  • 157 Views
  • Uploaded on

ASL2TXT. Converting sign language gestures from digital images to text George Corser. Presentation Overview. Concept Foundation : Barkoky & Charkari (2011) Segmentation Thinning My Contribution : Corser ( 2012) Segmentation (similar to Barkoky )

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'ASL2TXT' - idana


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
asl2txt

ASL2TXT

Converting sign language gestures from digital images to text

George Corser

presentation overview
Presentation Overview
  • Concept
  • Foundation: Barkoky& Charkari (2011)
    • Segmentation
    • Thinning
  • My Contribution: Corser (2012)
    • Segmentation (similar to Barkoky)
    • CED: Canny Edge Dilation (Minus Errors)
    • Assumption: User trains his own phone
concept
Concept
  • Deaf and hearing people talking on the phone, each using their natural language
  • Sign-activated commands like voice-activated
situation drive thru window
Situation: Drive Thru Window

Think:

Stephen Hawking

Deaf person signs order

Phone speaks order

Confirmation on screen

process flow
Process Flow
  • Requires several conversion processes
  • Many have been accomplished
  • Remaining: ASL2TXT
goal find an algorithm
Goal: Find an Algorithm
  • Find an image processing algorithm that recognizes ASL alphabet

= A

Web site

barkoky segmentation thinning
Barkoky: Segmentation & Thinning

Barkoky counts

endpoints to

determine sign

(doesn’t work for ASL)

barkoky process
Barkoky Process

Segmentation

Thinning

Input: hand segment

Apply thinning

Find endpoints, joints

Calculate lengths

Clean short lengths

Identify gesture by counting endpoints

  • Capture RGB image
  • Rescale
  • Extract using colors
  • Reduce noise
  • Crop at wrist
  • Result: hand segment
1 capture rgb image 2 rescale
1. Capture RGB Image2. Rescale

% ---------- 1. Capture RGB image

a = imread('DSC04926.JPG');

figure('Name','RGB image'),imshow(a);

% ---------- 2. Rescale image to 205x154

a10 = imresize(a, 0.1);

figure('Name','Rescaled image'),imshow(a10);

3 extract hand using colors
3. Extract Hand Using Colors

% ---------- 3. Extract hand using color

abw10 = zeros(205,154,1);

for i=1:205,

for j=1:154,

if a10(i,j,2)<140 && a10(i,j,3)<100,

abw10(i,j,1)=255;

end;

end;

end;

figure('Name','Extracted'),imshow(abw10);

Note: Color threshold code

differs from Barkoky

colors training set 2
Colors: Training Set (2)

Red

Green

Blue

Excel

4 reduce noise
4. Reduce Noise

% ---------- 4. Reduce noise

for i=2:204, for j=1:154,

if abw10(i-1,j,1)==0

if abw10(i+1,j,1)==0,

abw10(i,j,1)=0; end; end;

if abw10(i-1,j,1)==255

if abw10(i+1,j,1)==255,

abw10(i,j,1)=0; end; end;

end; end;

abw10 = imfill(abw10,'holes');

5 identify wrist position
5. Identify Wrist Position

% ---------- 5. Identify wrist position

for i=204:-1:1,

for j=1:154,

if abw10(i,j,1)==255, break; end;

end;

if j ~= 154 && abw10(i+1,j,1)~=255,

wristi=i+1;

wristj=j+1;

break;

end;

end;

wrist detection
Wrist Detection
  • Algorithm searches bottom-to-top of image
  • Finds a leftmost white pixel above black pixel
  • Sets wrist position SE of found white pixel
corser segmentation ced
Corser: Segmentation & CED
  • Segmentation (similar to Barkoky)
    • Color threshold technique slightly different
    • American Sign Language (ASL) alphabet, not Persian Sign Language (PSL) numbers
  • Image Comparison: Tried Several Methods
    • Full Threshold (Minus Errors)
    • Diced Segments (Minus Errors)
    • Endpoint Count Difference
    • CED: Canny Edge Dilation
asl training set
ASL Training Set

Hit-or-miss: 23%

Barkoky: 8%

hybrid algorithm example
Hybrid Algorithm Example

% ---------- MATLAB Code -------------------

matchtotal = 0;

if abs(x10range - x20range) < 20,

matchtotal = matchtotal + 10;

end;

if abs(y10range - y20range) < 20,

matchtotal = matchtotal + 11;

end;

matchtotal = matchtotal - abs(h10 - h20);

% ----- h10, h20 are vector magnitudes -----

canny edge dilation code
Canny Edge Dilation Code

% ---------- MATLAB Code -------------------

se = strel('disk',5);

a10 = edge(a10,'canny');

a20 = edge(a20,'canny');

a10 = imdilate(a10,se);

a20 = imdilate(a20,se);

% ----- Then calculate matches minus errors

disadvantages
Disadvantages
  • Dependent on lighting conditions
  • Fails with flesh-tone backgrounds
  • Requires calibration to a specific user
  • Limited applications: text messaging, activation (“sign” similar to voice activation)
  • ASL numbers (A=10, D=1, O=0, V=2, W=6)
  • Alphabet is tiny portion of full translation: complete translation maybe many years away
future work
Future Work
  • Barkoky claims flesh tones can be detected, but I have yet to replicate (even Barkoky changed his color detection scheme)
  • Could write letter-by-letter algorithm
  • Could use range camera to compute distance of finger instead of shape of hand
  • Motion analysis or edge count
  • Many possibilities… we’ve only just begun!

Cue: music

http://www.youtube.com/watch?v=__VQX2Xn7tI