Pen-CentricShorthand Interfaces Charles C. Tappert Seidenberg School of CSIS, Pace University
Themes of Presentation • Online Handwriting Recognition and Pen Computing Tutorial • Historical Research – undertaken for the Palm-Xerox Patent Infringement Lawsuit • Recent Research - Enhanced Pen-Centric Shorthand Interfaces can have benefits • DPS dissertation could extend M.S. thesis
Enhanced Pen-Centric Shorthand Interfaces • Can use word/phrase shorthand to speed text input • Can provide critical infrastructure for many pen-centric applications • Can enhance natural pen-centric interactions for many applications • Will have greatest impact on the utility of applications running on small mobile devices
Part 1: Online (Pen-Centric)Handwriting Recognition • Written Languages and Handwriting Properties • The Fundamental Property of Writing • Handwriting Recognition Difficulties • Online(Pen-Centric) Handwriting Recognition • Onlinemore accurate than Offline Recognition • Online Info Can Complicate Recognition Process • Design Tradeoffs / Design Decisions • Computer Problems in English
Written Language and Handwriting Properties • Alphabet • Letters, digits, punctuation, special symbols • Writing is a time sequence of strokes • Stroke – writing from pen down to pen up • Usually complete one character before beginning the next • Spatial order – e.g., in English left to right
Fundamental Property of Writing • Differences between different characters are more significant than differences between different drawings of the same character • This makes handwritten communication possible • Can there be exceptions – say, different characters written identically?
Fundamental Property of Writingin English • Property holds within subalphabets of uppercase, lowercase, and digits, but not across them • “I”, “l”, and “1” written with single vertical stroke • “O” and “0” written similarly with an oval
Handwriting Recognition Difficulties • Shape, size, and slant variation • Similarly shaped characters – U and V • Careless writing • in the extreme, almost illegible writing • Resolving difficult ambiguities requires sophisticatedrecognition algorithms, syntax/semantics
Online(Pen-Centric) Handwriting Recognition • Electronic tablets invented in late 1950s • Digitizer and display in separate surfaces • Pen Computers arrived in 1980s • Combined digitizer and display • Brought input and output into one surface • Immediate feedback via electronic ink • Created paper-like interface
Tablet Digitizers – Dynamic Information • Pen down – indication of inking • X-Y coordinates as function of time • Sampling rate: 100 points per second • Resolution: 200 points per inch
Early Pen-Centric Interface • Different surfaces for input and output • Rand system, about 1959
Pen Computers • IBM vision • Paper-like interface, 1992 • Microsoft Tablet PC • Launched, 2001
Pen-Centric PDAs • Early Palm Pilot • Palm Tungsten T3 and Sony Clié TH55
Online(Pen-Centric) Handwriting Recognition • Machine recognizes the writing as the user writes • Digitizer equipment captures the dynamic information of the writing • Stroke number,order,direction,speed • A stroke is the writing from pen down to pen up
Online(Pen-Centric) more accurate than Offline (Static) Recognition • Can use both dynamic and static information • Can often distinguish between similarly shaped characters • E.g., 5 versus S where the 5 is usually written with two strokes and the S with one stroke
Online Information Can Complicate Recognition Process • Large number of possible variations • E can be written with one, two, three, or four strokes, and with various stroke orders and directions • A four-stroke E has 384 variations (4! stroke orders x 24 stroke directions)
Online Information Can Complicate Recognition Process • Other variations
Online Information Can Complicate Recognition Process • Segmentation ambiguities • character-within-character problem • lowercase d might be recognized as a cl if drawn with two strokes that are somewhat separated from one another
Design Tradeoffs/Decisions • No constraints on the user • Machine recognizes user's normal writing • User severely constrained • Must write in particular style such as handprint • Must write strokes in particular order, direction, and graphical specification
English Writing Styles • Handprint • Uppercase – about 2 strokes per letter • Lowercase- about 1 stroke per letter • Cursive Script • Usually less than 1 stroke per letter • Delayed crossing and dotting strokes
Computer Problems in English • Constrained Handprint • Printing one symbol per box – form filling • Printing on lines – symbols can touch or overlap • Unconstrained Handprint • No lines and symbols can touch or overlap • Cursive Script • Mixed Printing and Cursive
Part 2Shorthand in Pen-Centric PDAs • Famous Uses of Shorthand • Historical Shorthand Alphabets • Pen-Centric Shorthand Alphabets • Pen-Centric Word/Phrase Shorthand • Allegro/Chatroom Shorthand System • M.S. thesis that could be extended into a DPS dissertation
Background • Famous writings throughout history were effectively written in a style of shorthand • Cicero’s orations • Martin Luther’s sermons • Shakespeare’s and George Bernard Shaw’s plays • Samuel Pepys’ diary • Sir Isaac Newton’s notebooks
Historical Shorthand Alphabets • We first review the history of shorthand systems prior to pen computing • Shorthand is “a method of writing rapidly by substituting characters, abbreviations, or symbols for letters, words, or phrases” • Shorthand can be traced back to the Greeks in 400 B.C.
Historical Shorthand Alphabets • We focus on shorthand alphabets that might be appropriate for PDAs • We review two types of shorthand • Geometric shorthand • Small number of basic shapes • Shapes reused in multiple orientations • Non-geometric shorthand shorthand
Historical Shorthand Alphabets • Ancient Greeks – 400 BC • Tironian Alphabet – 63 BC • John Willis’s Stenography – 1602 • Gabelsberger Alphabet – 1834 • Moon Alphabet – 1845
Stenography Alphabet, 1602 • Basic Shapes and Orientations
Other Historical Shorthand Systems • Phonetic alphabets • Pitman (1837), was popular in UK • Gregg (1888), was popular in USA • Systems for the blind • Braille(1821)
Pen-Centric Shorthand Alphabets • Some of the earliest were for CAD/CAM • symbols represent graphical items and commands • Others developed for text input on small consumer devices like PDAs that have limited computing power • We review geometric and non-geometric shorthands appropriate for small devices
Pen-Centric Shorthand Alphabets • Historical alphabets presented above could be used for machine recognition • symbols drawn with a single stroke (except “K” in Tironian and “+” in Stenography) • In addition to shape and orientation, online systems can use stroke direction to differentiate among symbols
Pen-Centric Shorthand Alphabets • Geometric Pen-Centric Shorthands • Organek – 1991 • Allen – filed 1991, patent 1993 • Goldberg (Xerox) – filed 1993, patent 1997 • Non-Geometric Pen-Centric Shorthands • Graffiti (Palm Computing) – 1995 • Allegro (Papyrus) – 1995
Organic Alphabet, 1991 Basic Shapes and Orientations One shape in 4 orientations. This gives 8 directions that together with 3 lengths provide 24 symbols. A second wheel provides additional symbols.
Goldberg patent, filed 1993 5 Basic shapes 4 Orientations 2 Stroked Directions 40 Possible Symbols • Designed for Speed of Input and Maximum Symbol Separation
Shorthand Alphabet Design • How would you design a shorthand alphabet? • What would be the design criteria?
Design of Graffiti Alphabetfor the Palm Pilot • Small alphabet • Uppercase, digits, special symbols • One stroke per symbol to avoid segmentation difficulty • Separate writing areas for letters and digits to avoid same-shape confusions
Graffiti Mimics Keyboard Input • Character by character input • Mode shifts for • Uppercase • Special characters • Eyes can focus on application’s insertion point rather than on input area
Graffiti Alphabet Design • What was the additional design criterion?
Graffiti Alphabet Design • Designed for ease of learning • 20 letters exactly match the Roman alphabet • 6 remaining ones match partially