Advanced Object Recognition and Action Classification using Deep Neural Networks

3. Object ClassesMost recent work is at the object level

4. We want in addition: 1. Individual Recognition

5. 2. Object parts and sub-partsCalled: Full Interpretation Window Mirror Window Door knob Headlight Back wheel Bumper Front wheel Headlight

6. 3. Action recognition (which 2 are different?)

7. 4. Agents Interactions 3 1 2 4 5 6

8. Class Non-class

9. Class Non-class

10. Is this an airplane?

11. Unsupervised Training Data

12. Features and Classifiers In DNN -- the net produces features of the top layer Previous work explored a broad range of features

13. Features used in the past: Generic Features Simple (wavelets) Complex (Geons)

14. Marr-Nishihara

15. Marr Net 2017 rotated versions of the object in the image

16. Past Class-specific Features: Common Fragments

17. Optima Features: Mutual Information I(C,F) Class: 1 1 0 1 0 1 0 0 Feature: 1 0 0 1 1 1 0 0 I(F,C) = H(C) – H(C|F)

18. Star model Detected fragments ‘vote’ for the center location Find location with maximal vote In variations, a popular state-of-the art scheme

19. Hierarchies of sub-fragments(a ‘deep net’) Detect the part itself by simpler sub-parts Repeat at multiple levels, to obtain a hierarchy of parts and sub-parts

20. Example Hierarchies

21. Classification by Features Hierarchy c x2 X1 X3 X4 X5 p(c,X,F) = p(c)Πp(xi|xi-) p(Fi|xi)

22. Global optimum can be found by max-sum message passing (two-pass computation) c x2 X1 X3 X4 X5 X = argmax [p(c,X,F) = p(c)Πp(xi|xi-) p(Fi|xi) ]

23. Context: Parts and ObjectsResults of two-pass computation

24. Current use of probabilistic graph models

25. HoG Descriptor Dallal, N & Triggs, B. Histograms of Oriented Gradients for Human Detection SIFT is similar, different details, multi-scale

26. SVM – linear separation in feature space

27. Optimal Separation SVM Perceptron The Nature of Statistical Learning Theory, 1995 Rosenblatt, Principles of Neurodynamics 1962. Find a separating plane such that the closest points are as far as possible

28. +1 The Margin -1 0 Separating line: w ∙ x + b = 0 Far line: w ∙ x + b = +1 Their distance: w ∙ ∆x = +1 Separation: |∆x| = 1/|w| Margin: 2/|w|

29. Using patches with HoG descriptors and classification by SVM Person model: HoG

30. DPM: Adding Parts

31. Bicycle model: root, parts, spatial map Person model

32. Deep Learning

34. ImageNet

37. AlexNet

38. On the history deep learning

39. A Neural Network Model A network of ‘neurons’ with multiple layers Repeating structure, linear, non-linear Automatic learning of weights between units

40. The McCulloch–Pitts neuron (1943) Relu

41. Perceptron learning yj = f(xj)

42. Back-propagation 1986

43. LeNet 1998 Essentially the same as the current generation

44. MNIST data set

45. Hinton Trends in Cognitive Science 2007 The goal: unsupervised Restricted Boltzmann Machines Combining generative model and inference CNN are feed-forward and massively supervised

46. Basic structure of deep nets. Not detailed here, but make sure you know the layers structure and repeating 3-layer arrangement