Transfer Learning of Object Classes: From Cartoons to Photographs

1. NIPS Workshop Inductive Transfer: 10 Years Later Geremy Heitz Gal Elidan Daphne Koller December 9th, 2005 Transfer Learning of Object Classes: From Cartoons to Photographs

2. Localization vs. Recognition Traditional question: “Is there an object of type X in this image?” Airplane? NO Human? YES Dog? YES Our question: “Where in this image is the object of type X?” MAN The man is walking the dog DOG

3. Outline • Landmark-based shape model • Localization as inference • Transfer learning from cartoon drawings • Results

4. Shape Model • Set of landmarks • Piecewise-linear contour between neighbors • Features of individual landmarks • Features of pairs of landmarks tail nose

5. Outline • Landmark-based shape model • Localization as inference • Transfer learning from cartoon drawings • Results

6. “Registering” the Model to an Image ? ? • Requires assigning each landmark to a pixel location

7. Localization Lnose Ltail Lcockpit Lunder Are local cues enough? Markov Random Field “Correct” pixel is often not the best match! • Potentials = Functions of local and global features Need to jointly consider all cues (features) Registration = Most Likely Assignment • Inference using max-product

8. Outline • Landmark-based shape model • Localization as inference • Transfer learning from cartoon drawings • Results

9. Learning Challenge ? ? ? ? ? no confusing background outline (shape) is easily recovered using snake Hand Label Hidden Variables Bootstrap from simple instanceswhere outlining is easy = cartoons / drawings Costly, and time-consuming Where to start? Local optima problem

10. Learning from Cartoon Drawings Shape Learning + Shape and Appearance Learning Registration

11. Phase I: Learning from Cartoons Final Shape Model Registration Pyramid • Extract high resolution contour using snake • Create shape-based model from training contours • Pairwise merging of models • Selection of landmarks

12. Phase II: Learning from Images Natural Image Model • Correspond initial model to training images • Select best correspondences as training instances • Learn final shape- and appearance-based model Training Set Selection Cartoon Phase Model Transfer high score low score

13. Outline • Landmark-based shape model • Localization as inference • Transfer learning from cartoon drawings • Results

14. Localization Results 0.84 0.75 0.84 0.72 0.18 sampletrainingcartoons sampleregistration 0.81 0.81 0.66 0.77 0.40

15. Transfer of Object Shape transfer 0.6 0.5 Benefit of shape transfer no transfer 0.4 Average overlap 0.3 0.2 0.1 0 0 2 4 6 8 10 # images in phase II Transfer of shape speeds up learning

16. Learning Appearance shape + appearance 0.64 0.62 0.6 No Appearance 0.58 Shape template Average overlap 0.56 0.54 0.52 0.5 0.48 0.46 0 2 4 6 8 10 FG/BG Appearance # images in phase II

17. Training Instance Selection PICKED AUTO PICKED HAND 0.7 0.65 AUTO 0.6 0.55 Average overlap 0.5 0.45 0.4 0.35 0.3 0 2 4 6 8 10 12 # images in phase II

18. Summary and Future Work • Flexible probabilistic shape model • Effective registration to images • Transfer • Shape from cartoons • Appearance from real images • Develop a better appearance model • Investigate self-training issues • Transfer from one class to another

19. Thanks!

20. Cartoon vs. Hand Segmentation Human Inter-Observer Hand Constructed 0.9 0.7 Learned from Drawings 0.5 Mean Overlap Score 0.3 cartoon handsegmented 0.1 0 1 2 3 4 5 Number of Training Instances Learning shape from cartoons is competitive with hand segmentation!

21. Landmark Features • Shape Template • Patch Appearance (Foreground/Background) • Location

22. Prediction 1 0.8 0.6 True positive rate 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 False positive rate • Comparable to constellation w/ 5 instances (Fei Fei et. Al) • Leading (discriminative) methods require many instances