Paper & Pencil Interface 1999 년 10 월 20 일 KAIST 전산학과 인공지능 연구실 김 진 형 Jkim@cs.kaist.ac.kr

1. Paper & Pencil Interface 1999년 10월 20일 KAIST 전산학과 인공지능 연구실 김 진 형 Jkim@cs.kaist.ac.kr http://ai.kaist.ac.kr/~jkim

2. 목 차 • Paper & Pen Interface • Pen Computers : Dream and Realization • Pen Computing 요소 기술 • Pen Computing을 위한 인식기술 • KAIST Approach • Killer Applications • Innovative Applications • KAIST Pen-based Arithmetic Tutor • Conclusion

3. "Paper and Pencil" Interface • 가장 자연스러운 Interface • (Probably) Between Computer and Humans • VDT, Keyboard, Mouse combination is limited • VDT syndrome : eyestrain, headaches, backaches, stiff necks, sore wrists • Mouse+keyboard vs Pen • Equation input • Drawing input • Large set Character input • 한자, 한글(?)

4. Pen Computer • Pen is the major input device • optional keyboard • various size and shape • Aims vertical market • Not a general purpose device • Some are already on the market • Not a big success yet • No killer applications yet

5. Pen Computer의 일반적인 사양 • 작고 가벼워서 들고 다닐 수 있다. • Mobile computing device, PDA • (전자)펜으로 글씨와 그림 작성, 메뉴 선택 • 무선 통신에 의한 정보 교환 • Handy internet terminal로 • MultiMedia 처리 • 글씨, 도형, 화상, 음성 • 값이 싸다.

6. Long History of Research • Dreams • DynaBook - Alan Kay • Knowledge Navigator - Apple Co. • Tablet - U. of Illinois Undergrads • Projects • Pattern Information Processing (Japan) • Electronic Paper (ESPRIT) • IBM, Sony, etc.

7. Samsung Penmaster 386

8. Samsung Export Model 1994(?)

12. PDA models

13. NeoPoint • Read and send email • Access Internet • Popular PDA programs • Contact • Schedule • To do • Sync data with PC • Wake-up-call

14. E-book : Rocket Book • Download book contents • Touch-Screen based interaction • Make note, annotation, book mark, under lining • No handwriting yet • Too expensive • No other usage • Need Internet Access

15. E-book : everybook

16. 전자부품연구원 ebook • 인터넷에 접속 정보검색 • 문서작성(?) • 8.2인치 터치 스크린 • 무게: 1㎏ • 배터리 사용시간 : 8시간 • PDA 4배크기 칼라 화면 • 저장 : 반도체 메모리 • 중소기업과 협동작업

17. PenComputer 요소 기술 • Hardware 기술 • Flat Panel Display • Pen/Digitizer • MicroProcessor • Battery and Power management • Storage Devices • Packaging • Software 기술 • Wireless Communications • Operating System • Handwriting recognition • Utilities and Accessories • Applications

18. Pen Computing을 위한 인식 문제 • 인식 대상 (online) • Menu 선택 • Characters (한글, 영문자, 한자, 숫자, 특수문자) • Drawings, Gestures • 영문 인식 시스템 시장에 출현 • Limited Capability • Printed style Only : C+ • Cursive Style : C- • 적극적인 활용의 장애 요인 • Pattern Recognition 학계의 활발한 연구 주제

19. 필기 문자 • A Sequence of some writing units • Temporally ordered • (mostly) left-to-right

20. 필기 문자 인식 • Source of Difficulty • Static Variability - personal style • Dynamic Variability - shape deviation • Stroke connection - coarticulation effect • Problems to solve Free-Writing • Variability Modeling • simple model for high flexibility • Resolve coarticulation • segmentation problem

21. Handwritten Roman Styles

22. Non-Roman Character Recognition • 동양언어권에서의 펜 인식 요구가 강함 • 일본의 PDA products • 상자안에 쓴 KANA • 또박 또박 쓴 한자 • 상자안에 쓴 영문자 • 중국인들의 노력 • 한국에서의 노력 • 한글 실용화 수준에 도달 • KAIST에서 수년간 연구

23. Printed All alphabet separated One Syllable in a box Cursive Ligature within Syllable One Syllable in a box Handwritten Hangul Styles Cursive Ligature within Syllable Syllables may Overlab Spatially Cursive Ligature over Syllables

24. KAIST Online 필기문자 인식 연구 • 한글 인식기(2 개의 국내 특허) • 무제한 필기 형태, 백지위에 연속 필기 가능 • 인식률 95+% • 무제한 연속 필기 영문자 단어(US Patent) • Boxed, Run-on, unconstrained cursive word • 약 88% 의 인식률 • 한자 인식기 • Gesture Recognizer • 응용 연구 • 도형편집기 • Arithmetic Tutoring System 등

25. 필기 문자 인식을 위한 KAIST의 HMM 기반 방법론 • Variability is modeled with HMM • Alphabet as character model • Stroke connection as ligature model • as a separate entity • Viewing handwritten word as an alternating sequence of character model and ligature model • Network of HMM • knowledge of language utilized • Hierarchy of Networks • component - character - word - sentence

26. Initial Node Final Node BongNet : Hangul Syllable Model Consonant Ligature Vowel Ligature Consonant

27. Initial State Final State Circular Network for English Words A z ligature

28. Recognition Problem • A Path corresponds to a Hangul Syllable / English word • Complete sequence of states and arcs from initial node to final node • Recognition • Finding the maximal probability path for given input chain code sequence • Yields optimal segmentation and character label, simultaneously

29. Cursive Hangul Recognition

30. Hangul Recognition (Segmentation Result)

31. Unconstained English Word Recognition

32. Unconstrained English Word Recognition (Segmentation Result)

33. Network Approach의 장점 • Segmentation and character labels obtained simultaneously • No external segmentation needed • Segmentation is obtained from global view point • Hierarchy of HMM can be constructed • Handwritten sentences with word model and inter-word gap model • Smooth Integration with postprocessing / language models • Framework for unified recognizer of multiple languages

34. Framework of Integrating Knowledge Advantages of Network Approach • Various knowledge can be easily integrated • Augmented Transition Network • data-driven and knowledge-driven approach combined • Language model checking • bigram, trigram, dictionary lookup • Structural constraint checking • within or intercharacter constraints • Complementary to Markov Assumption • Enables Pairwise discrimination

35. Intermixed Use of Languages English words and Chinese characters embedded in Hangul text

36. Inter-Character Ligature Hangul Digit HMM ? HMM ? HMM ? English HMM 멇 HMM 멊 HMM 멯 HMM 멲 HMM 멳 HMM 몒 Inter-Character Ligature for English Unified Network for Multiple Languages

37. Recognition of Multiple Language Text

38. Potential Killer Applications • PenSpreadsheet • Pen-based 한글(한자) Email System • Freehand Editor (교정부호 이용) • Free Memo - Fax machine • Pen-based Address Book(방명록) • Signature Verifier • Freehand Equation Input/Editor, Freehand Calculator • 외국어 훈련기 • 게임(바둑, 체스, Crossword Puzzle) • 작곡, 악보 편집

39. Pen computer : Note Taker

40. Pen computer : Fax / Phone

41. Pen computer : Draw and Spell

42. Pen Computer : Family Message Center

43. Pen Computer : Inventory Watch

44. Pen Computer : Data Collector

45. replace insert exchange join paragraphs insert space insert text Pen-Based Editing • 교정부호(Gesture)를 사용한 자연스로운 교정

46. Pen Computer : Portable Map

47. 지휘관의 전자수첩

48. FreeDraw

49. Pen Computer : Archtect's SketchPad

50. Free Hand Spreadsheet