Advanced 3D Handwriting Analysis System for Image Enhancement and Character Recognition

1. 3D Handwriting AnalysisA. Razdan, J. Femiani, J. RowePartnership for Research in Spatial Modeling (PRISM) Dr. Anshuman Razdan Director (razdan@asu.edu)

2. Parsing the OCR Problem • Preprocessing and Image enhancement • Pen Stroke Creation • Character recognition • Word recognition

3. Image Enhancement • Preprocessing includes enhancing and refining the raw image. • Identifying and extracting blurred, stained, faded, bled through, or transferred characters, etc. • New PRISM method specifically identifies and analyzes linear structures (line strokes). • This technique works in both 3D (CT, MRI) and 2D (images) domains.

4. Image Refinement • 1D and 2D function models based on the 3 observed shape characteristics have been developed, and enhanced images are derived from their second derivatives. • A two-stage algorithm is developed to extract line and net patterns. Line and net patterns are first enhanced and then extracted by applying threshold value. • Line and net patterns in a noisy environment exist in many imaging technologies • Examples: Roads and rivers in satellite photos, curves in finger prints, blood vessels in CT angiography

5. Enhancement & Thresholding Original image Enhanced image Line extraction by thresholding

6. Spanish Manuscript Example

7. Why 3D Analysis?

8. Flat Land: A Romance of Many Dimensions • You have to view the problem in at least one dimension higher than the data to get a sense of it(Flatland: A Romance of Many Dimensions: by Edwin A. Abbott, A Square, circa. 1884) Observer in 2D Land KING of 1D Land woman You are in 3D looking down at 2D space High Priest

9. An Example

10. Now I See Now I Don’t

11. Flat Land Conclusion • 1D (line) embed in 2D space (paper surface) • 2D (images) embed in 3D space (like this room) • 3D (objects) embedded in 4D or 5D space …. • Given this argument, using 3D space for understanding 2D images makes sense….

12. 3D Pen Traces

13. 3D Pen Trace Recreation • Concept of raising or embedding 2D image in 3D space a.k.a Flat Land. • Understanding ink flow and information embedded in the pen strokes • Theory of Volume Modeling and Iso-surface Extraction

14. Chain Codes or Pen Traces • For any character matching/recognition algorithm to work efficiently it needs to unravel the stroking of the pen. • This means figuring out the chain code. Since it is not available in 2D bitmap we do it using 3D.

15. Pen Stroking • Pressure is applied to via the pen and is different in upstrokes and down strokes and also angle of writing. • There is flow of ink from the pen to the paper. Crossovers result in darker images

16. How 2D is raised to 3D • A transfer function is applied which converts intensity at each pixel into a height function and also a density function • Results in Volumetric data same as CT or MRI H(i,j) = F(x,y, I(x,y)) D(i,j,k) = I(x,y) Vol Func(x,y,H(i,j)) = D(I(x,y)) 2D Image Transformed into 3D

17. Marching Cubes • Marching cubes is used for making 3D surfaces from volumetric data such as MRI, CAT scan, etc.

18. MC: Thresholding • Explanation of how Marching Cubes uses predefined triangulations for each cube to form a whole mesh.

19. Volume Blurring • Start with Volume Function (V) on raw image (left image) • Apply Marching Cubes on V (middle image) • Create V’ = GnV (Blurring filter applied n times and then MC to create right image). Gn is the secret sauce.

20. Modern Writing

21. Demo of Current Implementation

22. Curve Shape Measures and Matching for Character Recognition

23. The Problem • Given two curves X1 and X2, one can ask two distinct questions: • Curve matching i.e. • Is X1 = X2 ? • Or one a subset of the other curve • Or how similar are the two curves? • Curve alignment i.e. • What is the rotation and translation required to align one curve with the other?

24. Curve Matching Applied to Chars (Demo)

25. Conclusions • Novel method to unravel strokes, characters and letterforms in complex handwritten documents. • Segments by Region/Row irrespective of scale, orientation, or position. • Geometry based curve matching technique for character recognition (dictionary generation, text recognition, and translation) • Language independence • Doesn’t need expensive scanning equipment (we paid $24.99). • Can be combined with existing technologies. • Provisional Patent filed in April 2003. Full patent filing spring 2004.

26. Partial Match

27. Best Match

28. Weaknesses • Requires continuous tone original source (can not address single bit image i.e. FAX). • Can be computationally expensive for certain applications such as forgery but the technology is built to take advantage of parallelization.

29. Opportunities • Extend concept of volumes to other applications • Forensics (Offline comparisons) • Biometrics (Online authentication – wacom demo) • Forgery detection • Number extraction from noisy background (Currencies) • Opportunities for derivative patents

30. Gaps • Need to combine power of Stroke extraction and curve matching with traditional HMM and other statistical methods or commercial engines. • Man power/expertise required • AI/Statistics/traditional char recognition expert to create powerful hybrid engine • Language specific expert/paleographer • Requires productization and field testing.

31. Threats • Competition by 2D solutions and existing technologies. • Lack of awareness of the capabilities of 3D analytical tools in OCR world. • Geometry solution in a world seeped in statistical methods. • Establishing validity of the 2D - 3D conversion algorithm

32. Discussion and Q/A

33. Appendix

34. Two labs on campus 0ne moving to bigger space in BY – downtown Tempe. Additional 8000 sq ft slated for a new project (Decision Theatre) in downtown Tempe. 24 proc SGI, 20+ workstations (Unix, PC and Linux) Four 3D Laser scanners for inanimate objects 3D face scanner (recent acquisition) 2 Rapid Prototyping machines PRISM Infrastructure

35. Image Refinement • Biomedical Examples: White matter in brain MRI scans, cell spindle fibers, membranes in laser confocal microscopic data. Fungus membrane Brain MRI Scan Mouse egg

36. Image Refinement • Blood Vessel • 3 characteristics (Chaudhuri et al) • Piecewise linear segments • Cross section as a Gaussian function • Relatively constant width

37. 2D Line Model Blood Vessel (x,y)

38. 2D Case: 2nd Derivatives C: constant, N: noise

39. Enhancement • Maximal eigenvalue as an enhanced image Enhanced Image

40. Results Crest lines extraction A synthetic image Matched filters Our method

41. Applications of Curve Matching

42. Distance Between Two Functions Case 1: f and g continuous over [0,1] Case 2: f over [0,1] and g over [0,d], d <= 1 Penalty function

43. Curve Shape Measures • Shape Measures or Properties • Curvature (planar) • Torsion (space curves) • Total or absolute Curvature (space) • Classical Differential geometry says if the curvatures are identical then so are the curves subject to position and rotation

44. Curve Matching • Remember • Writing in terms of curvatures • What about partial match? • Or the general case

45. Three Matching Mesaures