Sign Language Translation System

1. Sign Language Translation System Group ID: 13-008

2. Introduction

3. There are about 72 million deaf people who use sign language as their first language or mother tongue. • It is estimated that more than 80% of these 72 million live in developing countries, where authorities are rarely familiar with their needs or desires. • Recent figures from the British Deaf Association suggest that on any day up to 250,000 people use some BSL. • Our Concern: is to interpret the BSL, British two-handed sign language alphabet and number gestures (plus others for essential additional keywords)

4. Drawbacks of existing systems: • All systems designed for ASL • Capture only one hand • Mostly accessories like wrist bands and glows are used • Costly equipments like kinect is used • Works under background constraints

5. Scope of the system: • A combination of four components and convinces two approaches • Sign to Text/Voice • Image capturing, Enhancement and gesture recognition • Image segmentation and feature extraction • Database and shot segmentation • Text/Voice to Sign • Convert text to visual sign 3D

6. FUNCTIONS

7. Background SubtractionUsed features • Skin color area. • Motion of the hand. • Edges of the background.

8. Skin color area • HSV color space is being used to detect the skin color.

9. Motion of the hand • Assumption : Hands are the only objects which are moving in the interested area. • Difference of two consecutive frames is used to track the motion.

10. Difference image : foreground objects are eliminated.

11. Edges of the background • Laplacian operator is being used to detect the edges.

12. Steps Briefly • Skin area is extracted • Take the difference of two consecutive frames.

13. Threshold the difference image.

14. Take the difference of skin area and the threshold frame.(a)

15. Take the edges from laplacian operator and dialate the output.

16. And the dialated edges with threshold.(b)

17. And output a and b and then erode it. This gives the final output of the edge of the moving hand.

18. Shot Segmentation Separation between words using speed variation

19. Feature Extraction • Hand region recognition and finger tip detection • Sequence of convex points. • ConvexityDefects defect points • Depth d of the fingers • Calculates mid of the palm • Radius of the palm • Angles between fingers using palm center, convex and defect points

20. Hu moments • Use moments() function in opencv • Returns three types of moments, • Spatial moments, • Central Moments and • Central Normalized Moments • Calculated from Central Moments which are invariant to size, position and orientation • Circularity

21. Edge Histogram Descriptor • Edge: important low level feature • Essential for content based image analysis • 16 segments of the image • Describes 5 edge types • Mpeg7 executable • Homogeneous Texture Descriptor (HTD) • Edge Histogram Descriptor (EHD)

22. Data Classification using SVM(Support vector machines) Becoming more and more popular: for its easiness Data format is an issue The goal of SVM To produce a model (based on the training data) which predicts the target values of the test data

23. Procedure followed: • Transformed data to the format of an SVM package • Conducted simple scaling on the data in the range of [0,1] (The original data maybe too huge or small in range, thus we can rescale them to the proper range so that training and predicting will be faster.) • Used the RBF kernel K(x; y) • Did cross-validation to find the best parameter C and ϒ using grid.py • Used the best parameter C and ϒ to train the whole training set • Testing was done

24. Current situation Trained SVM for 4 letters in the alphabet A,B,C,D Alphabet was initially classified as single hand and both hands in order to increase accuracy. Ex: Letter c uses only one hand A,B,C letters are identified Problems still occur in identifying letter D Future progress : Adding up of calculated EHD features to the feature set

25. Test to Sign Converter

26. Development Process • Design 3D model • Rigging and Skinning 3D model • Mapping IK[1] and FK[2] • Animation Design • Programming to join text and animations

27. 1. Designing 3D models (Open Source )

28. 2. Design 3D models

29. Design 3D models (Open Source )

31. Rigging and Skinning 3D model • The best way to animate a complex mesh object like a character is through the use of Armatures • armature acts like a skeleton: you actually move the bones of the armature and those bones drive the animation of the character mesh • The process of building an armature is called "rigging," and the process of attaching the armature to a mesh is called "skinning."

32. Rigging and Skinning 3D model

34. Test 01 • Create simple animation for few bones • Export into different file format *.fbx *.x *.3cd

35. *.fbx

36. Implementing 3D model Microsoft XNA

37. Microsoft XNA

38. Test 02 3D Max Features -3D Modeling -3D Animation Designing -3D Programming -Python -MaxScript

39. Disadvantages -Have to animate all animation clips in same layer. So many confused because large time track and have to remember start and end frame key number of each animations

40. Test 03 • MonoDeveloper • Open sourse • -Java script • -C# • -Boo • Unity3D • Powerful Game engine • Possibility of building a executable files for multiplatform like Android, iOS, Web, Windows just in few second

41. Macanim Tools Available Rigging and animation technique Generic Legacy Humanoid

42. Final Implementation + Humanoid Rig + Legacy Motion Builder

43. 3. Mapping IK and FK4. Animation Design Very powerful rigging and animation designing software specially designed for human character animation Today widely use in Film industry Motion Builder

44. Final Output – (up-to-now) + Macanim + Legacy Motion Builder

45. Challenges • Character customization features • Give more Facial expression to the 3D character

46. THANK YOU!!! Q&A