Heterogeneous convolutional neural networks for visual recognition

1. Heterogeneous convolutional neural networks for visual recognition Xiangyang Li, Luis Herranz, Shuqiang Jiang Key Laboratory of Intelligent Information Processing Institute of Computing Technology, CAS

2. Heterogeneous convolutional neural networks for visual recognition • Deep convolutional neural networks (CNNs) have shown impressive performance for image recognition. • A number of works have focused on how to improve the performance of CNNs: • train and test CNNs with multiple scales [1]. • increase the depth and width of CNNs [2][3]. • ensemble: averaging the results obtained from multiple instances of the same network [1][2][3]. Sermanet at al. Overfeat: integrated recognition, localization and detection using convolutional networks. In ICLR 2014. Simonyan et al. Very deep convolutional networks for large-scale image recognition, In ICLR 2015. Szegedy et al. Going deeper with convolutions, In CVPR 2015.

3. Heterogeneous convolutional neural networks for visual recognition • Features extracted from different architectures have different characteristics: • different number of layers and filter designs (related with different receptive fields). • grandmother cells & distributed codes [1]. D VD Agrawal et al. Analyzing the performance of multilayer neural networks for object recognition. In ECCV 2014.

4. Heterogeneous convolutional neural networks for visual recognition • Here we study combining heterogeneous convolutional neural networks (HCNNs). • Higher level features resulting from the combination of features from heterogeneous CNNs lead to richer and more discriminative feature representations. • The combination network strategy is flexible and it can benefit from large datasets and fine tuning to specific tasks.

5. Heterogeneous convolutional neural networks for visual recognition • Heterogeneous convolutional neural networks： • pre-training base networks • pre-training the combination network • fine tuning

6. Heterogeneous convolutional neural networks for visual recognition • Base networks:

7. Heterogeneous convolutional neural networks for visual recognition • Combination network: • (a) Two fully-connected layers • (b) Three fully-connected layers • (c) Three fully-connected layers with one additional top-layer removed from the base networks (a) (b) (c)

8. Heterogeneous convolutional neural networks for visual recognition • Training the network: • First, the base networks are pre-trained with a suitable large scale dataset. • The second step is training the combination network, which typically uses back propagation over a large scale dataset. • The third step is fine tuning the network.

9. Heterogeneous convolutional neural networks for visual recognition • Experiments: • We conduct our experiments on the Caffe [1] framework. • D denotes Alexnet and VD denotes VGG. We train the models on ImageNet. • In addition to the network D, we trained from scratch another network D’ with the same architecture for comparing with homogeneous architectures. • For test, we only used the central crop of size 227x227 of the image previously resized to 256x256. Jia et al. Caffe: Convolutional architecture for fast feature embedding. In ACM Multimedia 2014.

10. Heterogeneous convolutional neural networks for visual recognition Table 1. Comparison of different architectures for the combination network. • Combination network architecture:

11. Heterogeneous convolutional neural networks for visual recognition • Understanding the combination mechanism: we back propagate the optimal output for each class

12. Heterogeneous convolutional neural networks for visual recognition • Understanding the combination mechanism: we back propagate the optimal output for each class

13. Heterogeneous convolutional neural networks for visual recognition • Visualizing features obtained from different models: (a) fc7 of D (b) fc7 of VD

14. Heterogeneous convolutional neural networks for visual recognition • Visualizing features obtained from different models: (b) fc7 of VD (c) the joint network

15. Heterogeneous convolutional neural networks for visual recognition Table 2. Accuracy (%) for object recognition. • Object recognition performance:

16. Heterogeneous convolutional neural networks for visual recognition • Scene recognition performance: Table 3. Accuracy (%) for scene recognition.

17. Heterogeneous convolutional neural networks for visual recognition • Impact of fine tuning:

18. Heterogeneous convolutional neural networks for visual recognition • Impact of combination architecture and fine tuning: [D,VD]+2CL [Dp,VDp]+2CL

19. Heterogeneous convolutional neural networks for visual recognition • Conclusion: • we studied the unusual case of combining deep networks with heterogeneous architectures, and proposed HCNNs which combine them by concatenating high-layer features and stacking another (combination) network. • The proposed method achieves significant performance on many challenging benchmarks. • The combination network strategy is flexible and it can benefit from large datasets and fine tuning to specific tasks.

20. Thank you！