Sparse Coding and Its Extensions for Visual Recognition

1. Sparse Coding and Its Extensions for Visual Recognition Kai Yu Media Analytics Department NEC Labs America, Cupertino, CA

2. Visual Recognition is HOT in Computer Vision 80 Million Tiny Images Caltech 101 ImageNet PASCAL VOC

3. The pipeline of machine visual perception Low-level sensing Pre-processing Feature extract. Feature selection Inference: prediction, recognition Most Efforts in Machine Learning • Most critical for accuracy • Account for most of the computation • Most time-consuming in development cycle • Often hand-craft in practice

4. Computer vision features SIFT Spin image HoG RIFT GLOH Slide Credit: Andrew Ng

5. Learning everything from data Low-level sensing Pre-processing Feature extract. Feature selection Inference: prediction, recognition Machine Learning Machine Learning

6. BoW + SPM Kernel Bag-of-visual-words representation (BoW) based on vector quantization (VQ) Spatial pyramid matching (SPM) kernel • Combining multiple features, this method had been the state-of-the-art on Caltech-101, PASCAL, 15 Scene Categories, … Figure credit: Fei-Fei Li, Svetlana Lazebnik

7. Winning Method in PASCAL VOC before 2009 VQ Coding, Histogram, SPM Multiple Feature Sampling Methods MultipleVisual Descriptors Nonlinear SVM

8. Convolution Neural Networks Conv. Filtering Pooling Conv. Filtering Pooling • The architectures of some successful methods are not so much different from CNNs

9. BoW+SPM: the same architecture Pooling Local Gradients VQ Coding Nonlinear SVM Average Pooling (obtain histogram) e.g, SIFT, HOG • Observations: • Nonlinear SVM is not scalable • VQ coding may be too coarse • Average pooling is not optimal • Why not learn the whole thing?

10. Develop better methods Better Coding Better Pooling Better Coding Scalable Linear Classifier Better Pooling

11. Sparse Coding Sparse coding (Olshausen & Field,1996). Originally developed to explain early visual processing in the brain (edge detection). Training: given a set of random patches x, learning a dictionary of bases [Φ1, Φ2, …] Coding: for data vector x, solve LASSO to find the sparse coefficient vector a

12. Sparse Coding Example Natural Images Learned bases (f1 , …, f64): “Edges” » 0.8 * + 0.3 * + 0.5 * Test example x»0.8 * f36+ 0.3 * f42+ 0.5 * f63 [a1, …, a64] = [0, 0, …, 0,0.8, 0, …, 0, 0.3, 0, …, 0, 0.5, 0] (feature representation) Compact & easily interpretable Slide credit: Andrew Ng

13. Self-taught Learning [Raina, Lee, Battle, Packer & Ng, ICML 07] Motorcycles Not motorcycles Testing: What is this? Testing: What is this? … Unlabeled images Slide credit: Andrew Ng

14. ClassificationResult on Caltech 101 9K images, 101 classes 64% SIFT VQ + Nonlinear SVM 50% PixelSparse Coding + Linear SVM

15. Sparse Coding on SIFT [Yang, Yu, Gong & Huang, CVPR09] Local Gradients Pooling Sparse Coding Scalable Linear Classifier Max Pooling e.g, SIFT, HOG

16. Sparse Coding on SIFT [Yang, Yu, Gong & Huang, CVPR09] Caltech-101 64% SIFT VQ + Nonlinear SVM 73% SIFTSparse Coding + Linear SVM

17. What we have learned? Local Gradients Pooling Sparse Coding Scalable Linear Classifier Max Pooling e.g, SIFT, HOG • Sparse coding is a useful stuff (why?) • Hierarchical architecture is needed

18. MNIST Experiments Error: 4.54% Error: 3.75% Error: 2.64% • When SC achieves the best classification accuracy, the learned bases are like digits – each basis has a clear local class association.

19. Distribution of coefficient (SIFT, Caltech101) Neighbor bases tend to get nonzero coefficients

20. Interpretation 1 • Discover subspaces • Each basis is a “direction” • Sparsity: each datum is a linear combination of only several bases. • Related to topic model • Interpretation 2 • Geometry of data manifold • Each basis an “anchor point” • Sparsity is induced by locality: each datum is a linear combination of neighbor anchors.

21. A Function Approximation View to Coding • Setting: f(x) is a nonlinear feature extraction function on image patches x • Coding: nonlinear mapping • x  a • typically, a is high-dim & sparse • Nonlinear Learning: • f(x) = <w, a> A coding scheme is good if it helps learning f(x)

22. A Function Approximation View to Coding – The General Formulation Function Approx. Error An unsupervised learning objective ≤

23. Local Coordinate Coding (LCC) Yu, Zhang & Gong, NIPS 09 Wang, Yang, Yu, Lv, Huang CVPR 10 • Dictionary Learning: k-means (or hierarchical k-means) • Coding for x, to obtain its sparse representation a • Step 1 – ensure locality: find the K nearest bases • Step 2 – ensure low coding error:

24. Super-Vector Coding (SVC) Zhou, Yu, Zhang, and Huang, ECCV 10 • Dictionary Learning: k-means (or hierarchical k-means) • Coding for x, to obtain its sparse representation a • Step 1 – find the nearest basis of x, obtain its VQ coding • e.g. [0, 0, 1, 0, …] • Step 2 – form super vector coding: • e.g. [0, 0, 1, 0, …, 0, 0, (x-m3），0，…] Zero-order Local tangent

25. Function Approximation based on LCC Yu, Zhang, Gong, NIPS 10 locally linear data points bases

26. Function Approximation based on SVC Zhou, Yu, Zhang, and Huang, ECCV 10 Piecewise local linear (first-order) Local tangent data points cluster centers

27. PASCAL VOC Challenge 2009 Best of Other Teams Ours Difference Classes No.1 for 18 of 20 categories We used only HOG feature on gray images

28. ImageNet Challenge 2010 1.4 million images, 1000 classes, top5 hitrate ~40% VQ + Intersection Kernel 64%~73% Various Coding Methods + Linear SVM 50% Classification accuracy

29. Hierarchical sparse coding Yu, Lin, & Lafferty, CVPR 11 Learning from unlabeled data Conv. Filtering Pooling Conv. Filtering Pooling

30. A two-layer sparse coding formulation

31. MNIST Results -- classification  HSC vs. CNN: HSC provide even better performance than CNN  more amazingly, HSC learns features in unsupervised manner!

32. MNIST results -- effect of hierarchical learning Comparing the Fisher score of HSC and SC  Discriminative power: is significantly improved by HSC although HSC is unsupervised coding

33. MNIST results -- learned codebook One dimension in the second layer: invariance to translation, rotation, and deformation

34. Caltech101 results -- classification  Learned descriptor: performs slightly better than SIFT + SC

35. Conclusion and Future Work • “function approximation” view to derive novel sparse coding methods. • Locality – one way to achieve sparsity and it’s really useful. But we need deeper understanding of the feature learning methods • Interesting directions • Hierarchical coding – Deep Learning (many papers now!) • Faster methods for sparse coding (e.g. from LeCun’s group) • Learning features from a richer structure of data, e.g., video (learning invariance to out plane rotation)

36. References • Learning Image Representations from Pixel Level via Hierarchical Sparse Coding, • Kai Yu, Yuanqing Lin, John Lafferty. CVPR 2011 • Large-scale Image Classification: Fast Feature Extraction and SVM Training, • Yuanqing Lin, FengjunLv, Liangliang Cao, Shenghuo Zhu, Ming Yang, TimotheeCour, Thomas Huang, Kai Yu • in CVPR 2011 • ECCV 2010 Tutorial, Kai Yu, Andrew Ng (with links to some source codes) • Deep Coding Networks, • Yuanqing Lin, Tong Zhang, Shenghuo Zhu, Kai Yu. In NIPS 2010. • Image Classification using Super-Vector Coding of Local Image Descriptors, • Xi Zhou, Kai Yu, Tong Zhang, and Thomas Huang. In ECCV 2010. • Efficient Highly Over-Complete Sparse Coding using a Mixture Model, • Jianchao Yang, Kai Yu, and Thomas Huang. In ECCV 2010. • Improved Local Coordinate Coding using Local Tangents, • Kai Yu and Tong Zhang. In ICML 2010. • Supervised translation-invariant sparse coding, • Jianchao Yang, Kai Yu, and Thomas Huang, In CVPR 2010 • Learning locality-constrained linear coding for image classification, • Jingjun Wang, Jianchao Yang, Kai Yu, FengjunLv, Thomas Huang. In CVPR 2010. • Nonlinear learning using local coordinate coding, • Kai Yu, Tong Zhang, and Yihong Gong. In NIPS 2009. • Linear spatial pyramid matching using sparse coding for image classification, • Jianchao Yang, Kai Yu, Yihong Gong, and Thomas Huang. In CVPR 2009.