Two-dimensional Context-Free Grammars : Mathematical Formulae Recognition

1. Two-dimensional Context-Free Grammars: Mathematical Formulae Recognition Daniel Průša, Václav Hlaváč Center for Machine Perception Faculty of Electrical Engineering Czech Technical University, Prague

2. Presentation Overview • Formulae recognition, problem formulation • Known methods • General idea of structural recognition • Two-dimensional context-free grammars • Extension of the grammars • Recognition tool, pilot implementation • Results, future plans

3. Motivation for this work • To test a theoretical construct on a practical pilot problem with explicit structure mathematical formulae • The group of Schlesinger, Savchynskyy from Kiev works on music score recognition. We cooperate in a joint research project.

4. Math. formulae, off-line or on-line Formulae recognition can be divided into two groups by the type of input: • Off-line recognition – a formula is depicted in a raster image. • On-line recognition – a formula represented by a sequence of pen strokes (growing importance due to tablet PCs).

5. Math. formulae recognition, usage • Off-line recognition – conversion of scanned printed mathematical texts into an electronic form. • On-line recognition – connected to pen-based computing technologies (electronic tablets). There are many papers on formulae recognition, but only a few commercial products (e.g., xMathJournal by xThink)

6. Usual architecture Two independent layers: • Symbol detection and recognition. • Structural analysis. image, sequence of strokes symbol recognition symbols (+ coordinates and font size) error corrections (optional) structural analysis derivation tree

7. Symbol recognition methods • Image segmentation + OCR tool. • Image segmentation and character recognition performed simultaneously (e.g., by Hidden Markov Models). • It is very difficult to recover from errors made in segmentation phase. • Semantic not taken into account.

8. Structural analysis methods Grammar based • geometric grammars • graph grammars • Non-grammar based • minimum spanning tree • hard-coded rules

9. Our approach to structural recognition Based on general structural constructions by M.I. Schlesinger, V. Hlaváč in Ten Lectures on Statistical and Syntactic Pattern Recognition (Kluwer Academic Publishers, 2002) • Do not separate segmentation and parsing, perform them simultaneously. • Suitable for recognition of objects with rich structure. • Already successfully applied to music scores and electric circuits diagrams.

10. Structural Recognition – General Idea Assumptions: input image, set of derivation rules Recognition: • Algorithm starts with regions labeled by terminals • - squares corresponding to one symbol, • - regions detected by an external tool. • Bigger regions labeled by non-terminals are derived by applying the rules, each derivation is assigned by a penalty. • Result: region matching the whole picture with the smallest penalty. N Region N is derived by a rule from regions A, B, C, D B A D C

11. Structural Recognition Applied on Formulaeusing 2D Context-free Grammars • Uniform shapes of regions considered – rectangles • 2D grammar for mathematical formulae designed. • Terminals detection - detect all possible occurrences of elementary symbols using an OCR tool, evaluate the occurrences by a penalty (computed by the OCR tool). fraction line, minus sign symbol 5

12. Structural Recognition Applied on Formulaeusing 2D Context-free Grammars Parsing – let the structural analysis decide what is the best segmentation and interpretation of the elementary symbols, i.e. find derivation tree covering the whole image, evaluated by the smallest penalty. - 5 2

13. Two-dimensional Context-free Grammars … set of terminals … set of non-terminals … initial non-terminal … set of productions Three basic types of productions in P: Generalized form of productions:

14. Interpretation of Productions Ggenerates pictures that can be named by the initial non-terminalS

15. Theoretical Results on 2D CF Languages L(2CFG) ... class of languages that can be generated by a 2D CF grammar • L(2CFG) includes 1D context-free languages • L(2CFG) and L(2FSA) are not comparable • There is no analogy to the Chomsky normal form of productions • Basic form of productions is weaker than general one • Emptiness problem is not decidable • Languages in L(2CFG) can be recognized in polynomial time Observation: natural generalization, but the properties of L(2CFG) differ to the properties of the class of 1D context-free languages.

16. Recognition in Polynomial Time 2D CF grammars with productions in the basic form: Generated languages can be recognized in time picture size (M.I. Schlesinger) Algorithm can be generalized on all languages in L(2CFG) Maximal number of rows on the right-hand side of a production. Maximal number of columns on the right-hand side of a production. • degree of the polynomial depends on size of the productions

17. Extension of 2D CF Grammars 2D context-free grammar are not power enough to express complex structure of mathematical formulae. We need a formalism allowing to easily work with relative positions and sizes of symbols, e.g. to express relationships like “a symbol is superscript of another symbol”, etc. 1 3 5 2 + 5 3 6 4

18. Extension of 2D CF Grammars • Regions are still rectangles. • Each derived region is assigned by a feature point (logical center). The feature point a derived region is determined by the applied production. 1 5 3

19. Extension of 2D CF Grammars • Usage of productions is not limited on directly neighboring (touching) rectangles. • Productions can specify a rectangular area where some specific point of a rectangle has to be contained. • Position and sizes can be given relative to one of the rectangles. • Restrictions on relative sizes of rectangles are also possible. 3+2 5

20. Penalty Computation Based on summing partial penalties determined by the following criterions: • Used production. • Relative sizes and positions of regions the production is applied on (original regions). • Number of black pixels in the new region that are not in the original regions. • Penalty of the original regions.

21. Implementation of the Recognition Tool • Off-line recognition. • Implemented in Java. • Trained and tuned for hand-written formulae. • Black and white images (but can be extended on gray-scale images). • The following constructs are supported: • variables, numbers, parenthesis, • common unary and binary operators, power to operator, • fractions, square root, subscripts, superscripts, • sum, integral. • Can deal with noise, ambiguities, touching or split symbols, etc. and also with misplaced symbols.

22. Tool Architecture OCR tool terminals detection 2D grammar parsing

23. Terminals Detection Ideally, all regions should be scanned for an elementary symbol presence, but this consumes much time, two smarter strategies implemented: • Scanning rectangular windows of some predefined sizes (not all sizes). • Detection based on connectivity components. Limitations of the method: overlaping symbols’ bounding boxes, symbols that intersect Used OCR tool: A simple method implemented - feature vector extracted from image, k-nearest neighbor classifierused to classify the vector. Trained for all supported elementary symbols.

24. Remarks on Terminals Detection • Symbols that do not have size limited by a constant are not treated as terminal symbols (e.g., fraction line, square root). • In addition, square root cannot be separated from an image by a rectangle (it surrounds its argument). • Solution: Treat these cases as symbols composed of several terminal symbols, extend grammar by related productions.

25. Parsing Algorithm • Bottom up approach, as described in the general structural recognition. • Complexity – depends on the number of terminals detected during the first phase; in general, can be exponential, but it is substantially reduced by production restristions and usage of suitable data structures • Data structures for orthogonal range queries (searching points that are located in a rectangle) used to speed up the algorithm.

26. Future Plans • Focus on printed formulae • Collect sufficiently large set of annotated printed formulae • Apply learning methods: learn etalons of elementary symbols and productions parameters