A Performance Characterization Algorithm for Symbol Localization

1. A Performance Characterization Algorithmfor Symbol Localization Mathieu Delalandre1,2, Jean-Yves Ramel2, Ernest Valveny1 and Muhammad Muzzamil Luqman1,2 1 CVC, Barcelona city, Spain 2 LI Laboratory, Tours city, France LaBRI - Partnerships Meeting Bordeaux, France Thursday 14th of October 2010

2. A Performance Characterization Algorithmfor Symbol Localization tub door skin door sofa r1 r2 r3 Symbol localization systems (recognition and spotting) Performance characterization tub door skin door sofa Labels Ranks labels learning database QBE Learning Spotting/Recognition System document database r1 r2 r3 Recognition Region Of Interest Spotting Groundtruth Matching localization results with groundtruth Characterization measures truthresults Query By Example (QBE) rank Performance characterization To make the correspondence in term of localization To compute characterization measures (recall, precision, recognition rates, etc.)

3. A Performance Characterization Algorithmfor Symbol Localization Performance evaluation of image segmentation [Zhang’1996] Performance evaluation of object localization [Delalandre2009] Single : an object in groundtruth matches only with one detected object. Split : two objects in groundtruth match with one detected object. Merge : an object in groundtruth matches with two detected objects. • Global discrepancy methods • Number of missed segmented pixels • Position of missed segmented pixels • Local discrepancy methods • Number of region in the image • Features values of regions truthresults False alarm : a detected object doesn't match with any object in groundtruth. Miss : an object in groundtruth doesn't match with any detected object. results groundtruth Performance evaluation : image segmentation vs. object localization groundtruth

4. A Performance Characterization Algorithmfor Symbol Localization Performance evaluation of object localization [Delalandre2009] False alarm : a detected object doesn't match with any object in groundtruth. Miss : an object in groundtruth doesn't match with any detected object. Single : an object in groundtruth matches only with one detected object. Split : two objects in groundtruth match with one detected object. Merge : an object in groundtruth matches with two detected objects. truthresults Layout analysis [Antonacopoulos1999] Symbol spotting [Rusinol2009] Text/graphics separation [Liu1997] results groundtruth results groundtruth groundtruth groundtruth groundtruth results results char and text boxes isothetic polygons Convex hulls

5. A Performance Characterization Algorithmfor Symbol Localization Groundtruth, gravity centers, contours Open problem with object localization Result points detection rate Highest probabilities in a “part of” segmentation problem, how to make the difference between segmentation errors of background with segmentation errors of objects Lowest probabilities probability error p2 p1 p3 Ways to solve ... 1. “naive” : To use thresholds to “reject” some segmentation results (bad ...) 2. ideal : To define directed knowledge based approaches to model localization/segmentation algorithms (hard ...) 3. intermediate (proposed) : To use “fuzzy-based” approach, to characterize the characterization results according to confidence rate i.e. this is a positive matching between groundtruth and system’s results with a confidence rate of .

6. A Performance Characterization Algorithmfor Symbol Localization Groundtruth, gravity centers, contours Result points detection rate detection rate Groundtruth Results Highest probabilities Lowest probabilities probability error probability error Localization comparison Probability scores p2 p1 p3 Matching algorithm

7. A Performance Characterization Algorithmfor Symbol Localization Groundtruth, gravity centers, contours Result points detection rate detection rate Groundtruth Results Highest probabilities Lowest probabilities probability error probability error Localization comparison Probability scores p2 p1 Groundtruth, gravity center, contours c L g Result point i r p3 Intersection point Matching algorithm lgi Intersection line lgr

8. A Performance Characterization Algorithmfor Symbol Localization g2 Groundtruth, gravity centers, contours s2 g1 r s1 Result points detection rate detection rate s3 Groundtruth Results Highest probabilities Lowest probabilities g3 probability error probability error Localization comparison s r gi Probability scores p2 p1 Groundtruth points g2 g2 Result point s2 p3 Matching algorithm s2 g1 g1 s1 r r s3 s3 g3 g3 null probabilities, = equidistant case highest probabilities, = nearest points maximum probability, = equality case

9. A Performance Characterization Algorithmfor Symbol Localization Groundtruth, gravity centers, contours How to compute the probability between a groundtruth point giand the result point r, considering the neighboring groundtruth point gj we define - pi the probability r  gi, regarding gj - si is the scaling factor between gi and r - sj is the scaling factor between gj and r Result points detection rate detection rate Groundtruth Results Highest probabilities Lowest probabilities r gi gj probability error probability error pi Localization comparison r gi gj si sj Probability scores p2 p1 r r r p3 Matching algorithm gi gj gi gj gi gj

10. A Performance Characterization Algorithmfor Symbol Localization Groundtruth, gravity centers, contours Result points detection rate detection rate Groundtruth Results Thus, our probability function must respect the following properties Highest probabilities Lowest probabilities probability error probability error Localization comparison Several mathematics functions could be used (affine, exponential, trigonometric, etc.) we choose a Gaussian based function as it is good model of random distribution Probability scores p2 p1 p3 Matching algorithm 1 Probability score function Gaussian function x 0 1 0 1 2 3 4

11. A Performance Characterization Algorithmfor Symbol Localization Groundtruth, gravity centers, contours Result points detection rate detection rate Groundtruth Results We extend the computation of probability to a neighboring composed of n groundtruth points like this we define - is the set of groundtruth points - si is the scaling factor between gi and r - are the scaling factors between and r - is the probability r  gi, regarding Highest probabilities Lowest probabilities probability error probability error Localization comparison Probability scores p2 p1 g2 s2 p3 Matching algorithm g1 r s1 s3 g3

12. A Performance Characterization Algorithmfor Symbol Localization Groundtruth, gravity centers, contours Result points detection rate detection rate Groundtruth Results Highest probabilities Lowest probabilities probability error probability error Localization comparison g1 g2 … gn dg1=2 dgn=0 Probability scores p2 dr1=1 p1 r1 r2 … rq Groundtruth points Result points p3 Matching algorithm multiple (Tm) single (Ts) 1 detection rates alarm (Tf) ε 0 0 score error 1

13. A Performance Characterization Algorithmfor Symbol Localization floorplans false alarm (Tf) Qureshi’2008 multiple (Tm) max max detection rates single (Ts) floorplans 1 score error diagrams false alarm (Tf) diagrams multiple (Tm) max max detection rates single (Ts) 1 score error

14. A Performance Characterization Algorithmfor Symbol Localization detection rate detection rate Groundtruth 2 Results 2 Groundtruth 1 Results 1 probability error probability error Characterization Characterization Each result is context dependent, how to compare them ?

15. A Performance Characterization Algorithmfor Symbol Localization We compute the difference between a result and self-matching of his groundtruth (g), to make the new results test-independent. detection rate detection rate Groundtruth Results Groundtruth Results 1 probability error probability error 1(ε) 1(1) 1 2(1) 2(ε) i(ε) g Characterization Characterization single detections Ts si 0 1 0 score error (x) ε 0 0 score error (ε) 1 Transform function + q number of results (q) n i 0 1 0 global score i detection rate probability error

16. A Performance Characterization Algorithmfor Symbol Localization Qureshi’2008 electrical diagrams i(1) = 0.529 i(1) = 0.496 floorplans i(ε) floorplans electrical diagrams floorplans 1,00 score error (ε) diagrams i(ε) score error (ε)

17. Conclusion and perspectives • Conclusion • A new fuzzy way to evaluate object localization distribution of matching cases regarding a “confidence rate” • Experimentation with a real system electrical and architectural drawings, 200 test images, 3821 symbols • Perspectives • Extending experiments several systems, to add noise, scalability, real datasets