Expert Object Recognition in Video

1. Expert Object Recognition in Video Matt McEuen

2. Ventral Visual Pathway EOR Pathway From Draper, Baek, Boody - 2002 The EOR Pathway • Early vision (feature extraction) • Categorization • Exemplar matching

3. Feature Extraction Clustering Exemplar Matching Expert Object Recognition

4. Early Vision: Edge Detection • Gabor filters • Three filter sizes • Four orientations • Even and odd

5. Rectified energy 0° ... Normalized Sum 135° ... Filter output 90° 45° ... Early Vision: Edge Detection

6. Early Vision: Line Detection • Non-accidental structural properties • collinearity • parallelism • symmetry • Hough transform

7. Categorization • Allows a unique subspace for each category • K-Means

8. Alternating optimization: optimize C optimize cluster membership repeat Categorization

9. Exemplar Matching • Principal Component Analysis (PCA) • Based on covariance • Visual memory reconstruction

10. PCA • Calculate covariance matrix of the samples • Get the eigenvectors of the covariance matrix • Choose which eigenvectors to keep • Transform the data with the resulting matrix From Moeslund, 2001

11. 64 x 64 Image 5 Element Vector Exemplar Matching

12. VENUS • Biologically inspired • Habituation • Low-level features

13. Knowledge and Hierarchical Learning Architecture Knowledge Based on Object Interactions Increasing Semantic Knowledge Real World Knowledge Knowledge Based on Object Activities Knowledge Based on Identified Objects and Context Knowledge Based on Low Level Features

14. Benefits of EOR for video • General-purpose • Segmentation: Attention window • Associative memory in VENUS

15. Problems with EOR for video Why learning?

16. Problems with EOR for video • Hard training / testing distinction • Lots of processing • The parameter k

17. Data

18. (2) (1) Segmentation (3—training only) Data (87,79)

19. Ventral Visual Pathway VEOR EOR Pathway VEOR architecture

20. Foreground segmentation Ventral Visual Pathway VEOR EOR Pathway VEOR architecture

21. Foreground segmentation Object tracking Ventral Visual Pathway VEOR EOR Pathway VEOR architecture

22. Foreground segmentation Object tracking EOR subsystem Ventral Visual Pathway VEOR EOR Pathway VEOR architecture

23. Foreground segmentation Object tracking EOR subsystem Membership subsystem Ventral Visual Pathway VEOR EOR Pathway VEOR architecture

24. Foreground segmentation

25. Ok Too small Y Associate with existing object New object Overlap? Object tracking N

26. Extract patch, resize Masking Global PCA Feature extraction

27. Clustering: goals • Automatically determine k • Facilitate learning • ... efficiently

28. Clustering: solutions • Reuse of cluster centroids

29. Clustering: solutions • Reuse of cluster centroids • Cluster growing and splitting

30. Clustering: solutions • Reuse of cluster centroids • Cluster growing and splitting • Incremental clustering

31. Clustering: Fuzzy K-means

32. Done (remember centroids) Optimize cluster membership Optimize cluster centroids First time: 4 random cenroids nth time, n>1: saved centroids Yes FKM all clusters small enough? Yes No change < threshold? No Create additional centroid Clustering

33. Exemplar matching • PCA • Dirty flags • Output: best match & distance

34. Membership subsystem • Associative memory • Multi-class [0,1] hypothesis • One exemplar match per image • One membership hypothesis per tracked object

35. Three kinds of exemplar matching • Matching to a training image • Matching to a different learned object • Matching to the same learned object

36. Exemplar match: training image • Certainty of 1.0 in one class • Small distance == strong match

37. Exemplar match: different object • Contribution comes from object • Doesn't matter which image

38. Exemplar match: same object • No “new” evidence • Rebalance existing evidence • Recontribute match's contribution

39. Learning • Images of familiar classes, familiar views • New views of familiar classes • New classes

40. Results – Pet Faces Configuration 1 2 3 4 5 6 Accuracy 67.0% 80.8% 84.6% 81.5% 81.7% 83.8% Clustering algorithm HKM HKM FKM FKM FKM FKM Fuzziness n/a n/a 1.1 1.1 1.1 1.3 PCA on raw images? Yes Yes Yes No Yes Yes Hough transform? Yes No No No No No Incremental clustering? No No Yes Yes No Yes

41. Results – Vehicle Profiles

42. Results – Video Clips Global dims. Local dims. Accuracy 10 10 64.4% 50 10 82.2% 50 50 82.2%

43. Results – Video Clips Global dims. Local dims. Accuracy w/o Trucks 10 10 64.4% 50 10 82.2% 100% 50 50 82.2% 96.6%

44. Questions?

45. >> n = [3 5 6; 3 3 2; 5 1 2] n = 3 5 6 3 3 2 5 1 2 >> cov(n) ans = 1.3333 -2.0000 -1.3333 -2.0000 4.0000 4.0000 -1.3333 4.0000 5.3333 >> Covariance

46. Non-eigenvector: Eigenvector: A square matrix of n dimensions has n eigenvectors. Eigenvectors are orthogonal to one another. Eigenvectors

47. Accuracy Subspace dimensions From Draper, Baek, Boody, 2004 Draper's Results