An Approximate Nearest Neighbor Retrieval Scheme for Computationally Intensive Distance Measures

1. An Approximate Nearest Neighbor Retrieval Scheme forComputationally Intensive Distance Measures Pratyush Bhatt MS by Research(CVIT)

2. Nearest Neighbor Retrieval • Representation of an object • Fixed Length • Variable Length • (Dis)similarity Function (Distance Measure) • Neighborhood • Nearest neighbor retrieval problem can now be formalized as retrieving objects similar to a given object, where similarity is in accordance to a given similarity function

3. Need of NN retrieval schemes • If the search space is small, Sequential search is applied for accurate results. • With increase in memory, volume of data stored online has increased. • Sequential search is time consuming • Need to index data for fast retrieval • Birth of NN search algorithms

4. Computationally Expensive Distance Measures • X1 more similar to X2 than X3 visually • L1 distance between X1 and X2 is more than X1 and X3

5. Time complexity is super-linear to length of input. • Edit Distance: O(d2) Chamfer Distance

6. Nearest Neighbor Classification • Computationally expensive distance functions • Nearest neighbor classifiers often impractical for real applications. • Takes over 20 minutes to classify a single object on a modern PC using an optimized C++ implementation. • Larger the available training data, better will be the accuracy but at the cost of high computation time.

7. Motivation(Approx. NN) • Why to find similarity with all samples when decision is based on top K ? • How to find top K without finding similarity with all the samples ? • Solution: Compute K1 > K Approx. NN and find best K in that list. • In order to compute K1-NN, use fewer explicit matches.

8. Problem Statement • Improve nearest neighbor retrieval and classification performance in spaces with computationally expensive distance measures. • Generate an expansible approximate nearest neighbor list for a given query. • Dealing with points in non-metric space.

9. Metric Space

10. Metric Space Tree based (KD-tree, R-Tree) Hashing Based

11. KD Tree for Metric Space

12. Non-Metric Space

13. FingerExtension Wrist Rotation

15. Applications • Classification based on K-NN • K is determined empirically • Identification • Stop when similarity is above a fixed threshold • Retrieval Applications • Optimizing network usage in peer-to-peer computer networks. • Content-based retrieval systems • Concept of similarity is abstract • Need to generate expansible list • Learn from user feedbacks

16. Challenges • Accuracy depends on the similarity function used to compute Approx. NN • List can not be pre-computed, should be generated on-the-fly • Should be incremental to support scalability • Non-metric space • Prohibit application of triangular inequality

17. Manifold Theory • Given set of high dimensional points, where • Find embedding set in a low dimensional space • Preserves local similarity • Assumption: Data is well distributed

18. Run MDS on similarity matrix to get • Embedding of new sample is given by mean of column of squared matrix requires computation of similarity of new sample to points

19. Related Work • FastMap , Random Reference Objects, Random Line Projections, VP-Trees • Finds embedding of the query by computing only a few exact distances • Assumption: Triangular Inequality • BoostMap: Used AdaBoost to combine many simple 1-D embedding Lipschitz Embedding FastMap

20. Problem Formulation • Goal : Compute approx. NN of a query point q, from a set S of N points in accordance to similarity function, F. • Solution : • Split the data into a multi-level hierarchy • Exploit local similarity property to direct the search from top to bottom

21. Hierarchical Local Maps(HLM)

22. HOW TO FIND SIMILARITY OF QUERY WITH THESE SAMPLES WITHOUT EXPLICITLY CALCULATING IT ? 3. GET CHILDREN OF NEAREST NEIGHBOR 2. IDENTIFY NEAREST NEIGHBORS 1. FIND SIMILARITY OF QUERY WITH POINTS AT TOPMOST LEVEL 4. PROJECT POINTS ON THE MANIFOLD 6. FIND NEAREST NEIGHBOURS OF QUERY IN THIS METRIC SPACE 7. IDENTIFY THESE POINTS IN THE TREE 5. PROJECT QUERY ON THE MANIFOLD 8. TRAVERSE DOWN IN SAME FASHION

23. Results • Unipen Handwriting Database • 15953 digit examples • Divided into training and testing set with 2:1 ratio • Distance Measure : DTW

24. Number of DTW Computations for K nearest neighbor retrieval

25. Number of DTW Computations for K nearest neighbor retrieval

26. Number of DTW Computations for K nearest neighbor retrieval

27. Classification Accuracy Classification Accuracy on UNIPEN Dataset using exact and approximate k-NN • Average No. of DTW computations done by HLM : 160 • Expected No. of DTW computations done in brute-force : 5315

28. Biometric ( Special Case) • High Inter-Class Variation • Low Intra-Class Variation • Low variation in inter-class distances • Indexing for identification • How to apply HLM in such cases ?? • Local similarity structure becomes degenerate destroying any manifold structure

29. Biometric Data

30. High dimensional data Relative Contrast :

31. Iris • Feature Vector Length 1000 • If same class- < 100 bit differ • Else equal probability of each bit to match or not match • On average 500 bit differ • Imposter Scores would be around 0.5

32. Biometric Data • Such Distribution is bad for Indexing

33. Softness/Hardness • Measure of overlap between genuine and imposter classes • Soft Biometric (Face, Body Silhouette) • Poor Classification Accuracy • Better indexing • Correlates to multi dimensional point • Hard Biometric (Iris, Fingerprint) • Good Classification Accuracy • Poor indexing • Correlates to high dimensional point • Need a balance between two

34. Dataset • CASIA Iris Image Database V3.0 • 855 images corresponding to 285 users in training and testing set • 3 samples per eye • Simlarity Function • Hamming Distance • Euclidean Distance (Softer Metric) • Average gray value of a block resulted in 160D feature vector • Penetration Rate • Percentage of data on which we ran biometric matcher. • False Reject Rate • Percentage of identification instances in which false rejection occurs

35. HLM on CASIA Iris Dataset

36. Synthetic Dataset • Class center sampled from 1D gaussian (0,1) • Generate d-dimensional by sampling d times • Points of same class sampled from gaussian with mean as class centers and varying variance. • Total Number of classes: 500 • Points in same class • Training : 10 • Testing : 5

37. Indexing performance varying number of dimensions

38. Indexing performance varying within class to between class variance ratio

39. Contributions • A representation scheme for objects in a dataset that allows for fast retrieval of approximate nearest neighbors in non-euclidean space. • Search mechanism combined with filter and refine approach is proposed that minimizes the number of exact distance computations for computationally expensive distance measure. • Study performance of our scheme on biometric data and study the parameters affecting its performance.

40. Conclusion and Future Work • Local Similarity Property is well exploited by HLM • Incremental and Scalable • Softer biometric in filtering step combined with hard biometric in refine step would drastically reduce computation time • Optimal construction of HLM • Defining a measure for similarity function that allows hierarchical representation. • Learn a function to find degree of indexibility • Extract parameters from data distribution and similarity function

41. Thank You Related Publication: • Pratyush Bhatt and Anoop Namboodiri “Hierarchical Local Maps for Robust Approximate Nearest Neighbour Computation” Proceedings of the 7th International Conference on Advances in Pattern Recognition (ICAPR 2009), Feb. 4-6, 2009, Kolkatta, India.