1 / 21

ON-LINE HANDWRITING RECOGNITION OF MEETING NOTES USING PEN PROSODY

ON-LINE HANDWRITING RECOGNITION OF MEETING NOTES USING PEN PROSODY. D evelop a novel way to improve on-line character recognition performance of handwritten notes using hidden Markov models with multi-space probability distributions. Off-line Handwriting Recognition.

kiele
Download Presentation

ON-LINE HANDWRITING RECOGNITION OF MEETING NOTES USING PEN PROSODY

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. ON-LINE HANDWRITING RECOGNITION OF MEETING NOTES USING PEN PROSODY Develop a novel way to improve on-line character recognition performance of handwritten notes using hidden Markov models with multi-space probability distributions.

  2. Off-line Handwriting Recognition

  3. On-line Handwriting Recognition

  4. Off-line/On-line Handwriting Recognition

  5. Handwriting Recognition • Machine Learning • Neural Networks • Connectionist Computing • Natural Language Processing • Signature Verification • Postal-Address Interpretation • Bank-Check Processing

  6. Ami Corpus Augmented Multi-Party Interaction http://corpus.amiproject.org/ “A 15-member multi-disciplinary consortium dedicated to the research and development of technology that will help groups interact better.”

  7. Pen Data • Digital pens and whiteboards are used by participants to make hand written notes. • Time stamped x-y co-ordinates of pen strokes are captured. • The pen strokes are stored as xml files for subsequent processing. • Prosody-like features such as stroke direction, duration and pen speed are captured.

  8. Pen Data <Stroke> <StrokeIdx>0</StrokeIdx> <ID>0</ID> <StartDate>2004-12-13T10:47:45.0000000-00:00</StartDate> <StartMs>0</StartMs> <Duration>506</Duration> <Color>-16776961</Color> <Width>1</Width> <Points> <X>132.5</X> <Y>27</Y> </Points> . . . . <Points> <X>146.875</X> <Y>39.25</Y> </Points> <Forces>6E:72:72:72:68:4C:00:02:3C:5A:6E:74:70:6E:60:5A:52:5C:74:74:66:00:</Forces> </Stroke>

  9. Pen Data • The XML files are converted into JPEG images (offline representation). • A DIV-X movie of these files is also produced (online representation). • A transcription of written data is also generated using an optical character recognition technique based on Liwicki and Bunke(2005).

  10. Classification Each written character is represented by a vector sequence of observations O defined as: so then the probability of a word given a particular sequence can be computed using Bayes rule

  11. Markov Models • Direct estimation of the joint conditional probability is not practicable. • Parametric model is required

  12. Markov Model

  13. Hidden Markov Models

  14. Hidden Markov Models • Replace with the much simpler problem of estimating the Markov model parameters

  15. Hidden Markov Models • HMM’s are categorized into discrete and continuous. • Handwriting can contain both. • Multi-space distribution HMM (MSD-HMM), which is an extended version of HMM, will be employed to address the problem.

  16. The Hidden Markov Model Toolkit “Portable toolkit for building and manipulating hidden Markov models.” • Speech recognition • Speech synthesis • DNA sequencing • Character recognition

  17. HTK Tools • Training • Testing • Analysis • Continuous density mixture Gaussians • Discrete distributions • Does not support MSD’s!!!

  18. Next Steps • Learning to use the HTK. • Research into using multi space probability distributions in HMM’s. • Adapting HTK to utilise multi space probability distributions. • Testing and comparing accuracy with normal HMM (both discrete and continuous). • Write up paper and prepare final presentation.

  19. Questions?

  20. Sources • AMI Project Website - amiproject.org • A tutorial on Hidden Markov Models and selected applications in speech recognition -IEEE White Paper by L R Rabiner (1989) • HMM-Based On-Line Recognition of Handwritten Whiteboard Notes – M Liwicki, H Bunke • HTK Handbook • HTK Website - htk.eng.cam.ac.uk • Multi-Space Probability Distribution HMM - IEICE White Paper by K Tokuda, T Masuko, N Miyazaki and T Kobayashi (2000)

More Related