Presented by: Mingyuan Zhou Duke University, ECE September 18, 2009

1. Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng ICML 2009 Presented by: Mingyuan Zhou Duke University, ECE September 18, 2009

2. Outline • Motivations • Contributions • Backgrounds • Algorithms • Experiment results • Deep Vs Shallow • Conclusions

3. Motivations • To Learn hierarchical models which simultaneously represent multiple levels, e.g., pixel intensities, edges, object parts, objects, and beyond can be represented by layers from low to high. • Combining top-down and bottom-up processing of an image. • Limitations of deep belief networks (DBNs) • Scaling DBNs to realistic-size images remains challenging: images are high-dimentional and objects can appear at arbitrary locations in images.

4. Contributions • Convolutional RBM: feature detectors are shared among all locations in an image. • Probabilistic max-pooling: in a probabilistic sound way allowing higher-layer units to cover larger areas of the input. • The first translation invariant hierarchical generative model supporting both top-down and bottom-up probabilistic inference and sales to realistic image sizes.

5. Backgrounds: Restricted Boltzmann Machine (RBM) • Giving the visible layer, the hidden units are conditionally independent, and vise versa. • Efficient block Gibbs sampling can be performed by alternately sampling each layer’s units. • Computing the exact gradient of the log-likelihood is intractable, so the contrastive divergence approximation is commonly used. (binary v) (real-value v)

6. Backgrounds: Deep belief network (DBN) • In a DBN, two adjacent layers have a full set of connections between them, but no two units in the same layer are connected. • A DBN can be formed by stacking RBMs. • An efficient algorithm for training DBNs (Hinton et al., 2006): greedily training each layer, from lowest to highest, as an RBM using the previous layer's activations as inputs.

7. Algorithms: Convolutional RBM (CRBM)

8. Algorithms: Probabilistic max-pooling

9. Algorithms: Probabilistic max-pooling • Each unit in a pooling layercomputes the maximum activation of the units in asmall region of the detection layer. • Shrinking the representationwith max-pooling allows higher-layer representationsto be invariant to small translations of theinput and reduces the computational burden. • Max-pooling was intended only for feed-forward architectures. A generative model of images which supports both top-down and bottom-up inference is of interest.

10. Algorithms: Sparsity regulations • Only a tiny fractionof the units should be active in relation to a givenstimulus. • Regularizing the objective function toencourageeach of the hidden units to have a mean activationclose to some small constant .

11. Algorithms: Convolutional DBN (CDBN) • CDBN consists of several max-pooling-CRBMs stacked on top of one another. • Once a given layer istrained, its weights are frozen, and its activations areused as input to the next layer.

12. Hierarchical probabilistic inference

13. Experimental Results: natural images

14. Experimental Results: image classification

15. Experimental Results: unsupervised learning of object parts

16. Experimental Results: Hierarchical probabilistic inference

17. Deep Vs Shallow From Jason Weston’s slides: DEEPLEARNING VIASEMI-SUPERVISED EMBEDDING, ICML 2009 WORKSHOP ON LEARNING FEATURE HIERARCHIES . From Francis Bach’s slides:Convex sparse methods for feature hierarchies, ICML 2009WORKSHOP ON LEARNING FEATURE HIERARCHIES

18. Conclusions Convolutional deep belief network: • A scalable generative model for learning hierarchical representations from unlabeled images. • Performing well in a variety of visual recognition tasks.