1 / 17

Nonparametric Scene Parsing via Label Transfer

Nonparametric Scene Parsing via Label Transfer. Author: Ce Liu Jenny Yuen Antonio Torralba Group 3 Presenter: Hongsheng Yang. The task of object recognition and scene parsing. window. tree. sky. road. Output. Input. field. car. building. unlabeled.

summer
Download Presentation

Nonparametric Scene Parsing via Label Transfer

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Nonparametric Scene Parsing via Label Transfer Author: Ce Liu Jenny Yuen Antonio Torralba Group 3 Presenter: Hongsheng Yang Adapted from Ce Liu's CVPR2009 slides

  2. The task of object recognition and scene parsing window tree sky road Output Input field car building unlabeled Adapted from Ce Liu's CVPR2009 slides

  3. Training based object recognition and scene parsing • Sliding window method - Train a classifier for a fixed-size window (e.g., car vs. non-car) - Try all possible scales and locations, run the classifier - Merge multiple detections • Texton method - Extract pixel-wise high-dimensional feature vectors - Train a multi-class classifier - Spatial regularity: neighboring pixels should agree J. Shottonet al. Textonboost: Joint appearance, shape and context modeling for multi-class object recognition and segmentation. ECCV, 2006 Adapted from Ce Liu's CVPR2009 slides

  4. Label Transfer - Intuition • I’ve seen and recognized a few similar pictures before. • If I could correspond each pixels in the query image to the pixels in the previous seen images, • then I could infer how the new query image looks like based on the database images. Adapted from Ce Liu's CVPR2009 slides

  5. Label Transfer - Pipeline > Given a query image • Find another annotated image with similar scene • Find dense correspondences between these two images • Warp the annotation according to the correspondences > Two key components: • A large, annotated database • Good correspondences for label transfer Query from Database window tree sky road field car Warped annotation User annotation building unlabeled Adapted from Ce Liu's CVPR2009 slides

  6. Large image databases • A subset of LabelMe database (outdoor scenes) • 2688 in total, 2488 for training, 200 for test • 33 object categories + “unlabeled”, including street, beach, mountains, fields, buildings, etc. B. Russell et al. LabelMe: a database and web-based tool for image annotation. IJCV 2008. Adapted from Ce Liu's CVPR2009 slides

  7. A good correspondence approach • SIFT flow - analogous to - Optical flow • Scene level - Image level • SIFT Flow – dense SIFT, spatial regularization Adapted from Ce Liu's CVPR2009 slides Optical flow

  8. Input Support Optical flow Warping of optical flow SIFT flow Dense SIFT image (RGB = first 3 components of 128D SIFT) SIFT flow Warping of SIFTflow Adapted from Ce Liu's CVPR2009 slides

  9. Objective energy function is similar to that of optical flow: Data term (reconstruction) Small displacement bias Smoothness term • MRF - p, q: grid coordinate, w: flow vector, u, v: x- and y-components, s1, s2: SIFT descriptors C. Liu et al. SIFT Flow: Dense Correspondence across Scenes and its Applications. TPAMI 2011 Adapted from Ce Liu's CVPR2009 slides

  10. Design of Nonparametric Scene Parsing System • Scene retrieval: retrieve a set of nearest neighbors in the database for a given query image. (One image is not good enough, using GIST as matching score) • Compute the SIFT flow from the query to each nearest neighbor, and use the achieved minimum energy to re-rank the nearest neighbors. Further select the top M re-ranked retrievals to create the voting candidate set. Adapted from Ce Liu's CVPR2009 slides

  11. Warped Annotations SIFT Query Candidate set Annotation SIFT SIFT flow Adapted from Ce Liu's CVPR2009 slides

  12. Another multi-labeling MRF to integrate the result of candidate annotated images, including per-pixel likelihood, spatial prior, neighborhood spatial consistency Warped Anotation Parsing Ground truth SIFT Query Annotation SIFT SIFT flow Candidate set Warped Annotations Adapted from Ce Liu's CVPR2009 slides

  13. Scene parsing results (1) Annotation of best match Warped best match to query Parsing result of label transfer Query Best match Ground truth

  14. Scene parsing results (2) Annotation of best match Warped best match to query Query Parsing result Ground truth Best match

  15. Pixel-wise performance Our system optimized parameters Per-pixel rate 74.75% Pixel-wise frequency count of each class Stuff Small, discrete objects

  16. The relative importance of different components of the parsing system

  17. Conclusion • Label transfer provides a novel data-driven way to understand scene. • A few future work are conducted from this line: e.g. Superparsing • Need a better robust correspondence approach: e.g. scale rotation invariant dense descriptor? complexity? -> one up-to-date work: Deformable Spatial Pyramid Matching for Fast Dense Correspondences Problem, Jaechul Kim, Ce Liu, FeiSha and Kristen Grauman, CVPR 2013 Adapted from Ce Liu's CVPR2009 slides

More Related