1 / 92

Fingerspelling Recognition with Support Vector Machines and Hidden Conditional Random Fields

Fingerspelling Recognition with Support Vector Machines and Hidden Conditional Random Fields. - A comparison with Neural Networks and Hidden Markov Models - César R. de Souza, Ednaldo B. Pizzolato and Mauro dos Santos Anjo

aaralyn
Download Presentation

Fingerspelling Recognition with Support Vector Machines and Hidden Conditional Random Fields

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Fingerspelling Recognition with Support Vector Machines and Hidden Conditional Random Fields - A comparison with Neural Networks and Hidden Markov Models - César R. de Souza, Ednaldo B. Pizzolato and Mauro dos Santos Anjo Universidade Federal de São Carlos (Federal University of São Carlos) IBERAMIA 2012 Cartagena de Índias, Colombia2012

  2. Introduction Context, Motivation, Objectives and the Organization of this Presentation

  3. Multidisciplinary • Computing and Linguistics • Ethnologue lists about 130 sign languages existent in the world • (LEWIS, 2009) ContextMotivationObjectivesAgenda

  4. Two fronts • Social • Aim to improve quality of life for the deaf and increase the social inclusion • Scientific • Investigation of the distinct interaction methods, computational models and their respective challenges ContextMotivationObjectivesAgenda

  5. This paper • Investigate the behavior and applicability of SVMs and HCRFs in the recognition of specific signs from the Brazilian Sign Language • Long term • Walk towards the creation of a full-fledged recognition system for LIBRAS • This work represents a small but important step in achieving this goal ContextMotivationObjectivesAgenda

  6. ContextMotivationObjectivesAgenda Introduction - Brazilian Sign Language Libras Literature Review and Tools Methods Support Vector Machines Conditional Random Fields Experiments Results Conclusion

  7. The BrazilianSignLanguage (LIBRAS) Structuresandthe manual alphabet

  8. Natural language • Not mimics • Not universal • It is not only “a problem of the deaf or a language pathology” • (QUADROS & KARNOPP, 2004) LIBRASDifficultiesGrammar

  9. Highly context-sensitive • Same sign may have distinct meanings • Interpretation is hardeven for humans LIBRASDifficultiesGrammar

  10. Fingerspelling is onlypart of the Grammar • Needed when explicitly spellingthe name of a person or a location • Subset of the full-language recognition problem LIBRASDifficultiesGrammar

  11. Literature • Layerarchitectures are common • Staticgestures x Dynamicgestures • Oneofthebestworkson LIBRAS handlesonlythemovementaspectofthelanguage(Dias et al.)

  12. Fewstudies explore SVMs • Butmany use Neural Networks • No studiesonHCRFsand LIBRAS

  13. Example • Recognitionof a fingerspelled wordusing a two-layeredarchitecture Pato HMM Sequenceclassifier P P P A A A T T T O O Staticgestureclassifier ANN

  14. YANG, SCLAROFF e LEE, 2009 • Multiplelayers, SVMs • Elmezain, 2011 • HCRF, in-airdrawingrecognition

  15. Models and Tools Overview of the chosen techniquesand reasons for their choice

  16. Static Gesture Recognition Neural Networks and Support Vector Machines for the detection of static signs

  17. Find such that… c b a

  18. Biologically inspired • McCulloch & Pitts, Rosenblatt, Rumelhart Neural NetworksSupport Vector MachinesMaximum MarginMultiple Classes

  19. Perceptron • Hyperplanedecision • Linearlyseparableproblems • Learning is a ill-posedproblem • Multiple local minima, ill-conditioning • Layerarchitecture • Universal approximator

  20. Strong theoretical basis • Statistical Learning Theory • Structural Risk Minimization (SRM) Neural NetworksSupport Vector MachinesMaximum MarginMultiple Classes

  21. Riskminimizationthroughmarginmaximization • Capacitycontrolthroughmargincontrol • Sparsesolutionsconsideringonly a fewsupportvectors Neural NetworksSupport Vector MachinesMaximum MarginMultiple Classes • Large-marginclassifiers

  22. Problem: binary-only classifier • How to generalize to multiple classes? Neural NetworksSupport Vector MachinesMaximum MarginMultiple Classes

  23. Problem: binary-only classifier • How to generalize to multiple classes? • Classical approaches • One-against-all • One-against-one • Discriminant functions • Drawbacks • Only works when equiprobable • Evaluation of c(c-1)/2 machines • Non-guaranteed optimum results With 27 static gestures, this would result in 351 SVM evaluations each time a new classification is required!

  24. Problem: binary-only classifier • How to generalize to multiple classes? • Directed Acyclic Graphs • Generalization of Decision Trees,allowing for non-directed cycles • Require at maximum c-1 evaluations Neural NetworksSupport Vector MachinesMaximum MarginMultiple Classes So, for 27 static gestures, only 26 SVM evaluations are required. Only 7.4% of the original effort 

  25. Elimination proccess • One class eliminated at a time Candidates A x D A B C D A B C D A lost D lost B x D A x C B C D A B C B lost C lost D lost A lost C x D B x C A x B C D B C A B B lost C lost C lost D lost A lost B lost D C B A

  26. However, no matter the model • We’ll have (extreme) noise due pose transitions • How can we cope with that?

  27. Dynamic Gesture Recognition Hidden Markov Models, Conditional Random Field and Hidden Conditional Random Fields for dynamic gesture recognition.

  28. Find such that givenextremelynoisysequencesoflabels, estimatethewordbeingsigned. hello blyrei hi hil bye hmeylrlwo

  29. Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs)

  30. Hidden Markov Models • Joint probability model of a observation sequence and its relationship with time Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs) A B

  31. Hidden Markov Models • Marginalizing over y, we achieve the observation sequence likelihood • Which can be used for classificationusing either the ML or MAP criteria Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs)

  32. Word 1 Word 2 ... Word n Onemodel for eachword

  33. Hidden Markov Models have found great applications in speech recognition • However, a fundamental paradigmshift recently occurred in this field Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs)

  34. Probability distributions governing speech signals could not be modeled accurately, turning “Bayes decision theory inapplicable under those circumstances” • (Juang & Rabiner, 2005) Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs)

  35. Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs)

  36. Conditional Random Fields • Generalization of the Markov models Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs) • Discriminative Models • Model without incorporating • Designates a family of MRFs • Each new observation originates a new MRF

  37. Generative Sequence Graphs Naïve Bayes HMM Directional models Discriminative Logistic Regression Linear-chain CRF CRF Infographbasedonthe tutorial by Sutton, C., McCallum, A., 2007

  38. Conditional Random Fields • Generalization of the Markov models Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs) PotentialFunctions Parameter vector whichcanbeoptimizedusinggradientmethods Potential Cliques Characteristicfunction vector Partitionfunction

  39. Conditional Random Fields • Generalization of the Markov models Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs) • How do we initialize those models? • Reaching HCRFs from a HMM

  40. a11 a12 a13 b11 b12 a21 a22 a23 b21 b22 a31 a32 a33 b31 b32

  41. i=1j=1 i=1j=2 i=1j=3 i=2j=1 i=2j=2 i=2j=3 i=3j=1 i=3j=2 i=3j=1 i=1o=1 i=1o=2 i=2o=1 i=2o=2 i=3o=1 i=3o=2 a11 a12 a13 a21 a22 a23 a31 a32 a33 b11 b12 b21 b22 b31 b32

  42. Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs) • Drawback • Assumes both and are known

  43. Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs)

  44. Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs)

  45. Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs) • Hidden Conditional Random Fields • Generalization of the hidden Markov classifiers Parameter space

  46. Hidden Markov Models (HMMs)Conditional Random Fields (CRFs)Hidden Conditional Random Fields (HCRFs) • Hidden Conditional Random Fields • Generalization of the hidden Markov classifiers • Sequence classification • Model without explicitly modeling • Do not require to be known • The sequence of states is now hidden

More Related