object graphs for context aware visual category discovery
Download
Skip this Video
Download Presentation
Object-Graphs for Context-Aware Visual Category Discovery

Loading in 2 Seconds...

play fullscreen
1 / 37

Object-Graphs for Context-Aware Visual Category Discovery - PowerPoint PPT Presentation


  • 91 Views
  • Uploaded on

Object-Graphs for Context-Aware Visual Category Discovery. Cheng-Ming Chiang Advisor: Sheng- Jyh Wang 2012/7/9. Outline. Introduction Related Work Approach Results Conclusion and Future Work Reference. Introduction. How to discover unfamiliar objects in unlabeled images?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Object-Graphs for Context-Aware Visual Category Discovery' - king


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
object graphs for context aware visual category discovery

Object-Graphs for Context-Aware Visual Category Discovery

Cheng-Ming Chiang

Advisor: Sheng-Jyh Wang

2012/7/9

Reference:

L. Yong Jae and K. Grauman, "Object-Graphs for Context-Aware Visual Category Discovery," PAMI, 2012.

outline
Outline
  • Introduction
  • Related Work
  • Approach
  • Results
  • Conclusion and Future Work
  • Reference
introduction
Introduction
  • How to discover unfamiliar objects in unlabeled images?
    • Unsupervised visual category discovery
  • Existing unsupervised techniques usually use appearance alone to detect visual themes, but it may suffer from
    • Occluded objects
    • Large intra-category variations
    • Low-resolution data
introduction1
Introduction
  • A new idea: How could visual discovery benefit from familiar objects?
    • Model the interaction between a set of detected known categories and the unknown to-be-discovered categories
  • Object-level context cues + Appearance descriptors
    • Introduce a novel object-graph descriptor to encode the 2D and 3D spatial layout
related work
Related Work
  • State-of-the-art discovery method, appearance alone
    • B. C. Russell, W. T. Freeman, A. A. Efros, J. Sivic, and A. Zisserman, "Using Multiple Segmentations to Discover Objects and their Extent in Image Collections," CVPR, 2006

Reference:

B. C. Russell, W. T. Freeman, A. A. Efros, J. Sivic, and A. Zisserman,

"Using Multiple Segmentations to Discover Objects and their Extent in Image Collections," CVPR, 2006

related work1
Related Work
  • Is it possible to learn visual object classes and their segmentations simply from looking at images?
    • Challenges:
      • How to recognize visually similar objects?
      • How to segment them from their background?
  • In fact, both object recognition and image segmentation can be thought of as parts of one large grouping problem
    • Projecting groups onto a particular image gives segmentation
    • Projecting groups onto the image index gives recognition

Reference:

B. C. Russell, W. T. Freeman, A. A. Efros, J. Sivic, and A. Zisserman,

"Using Multiple Segmentations to Discover Objects and their Extent in Image Collections," CVPR, 2006

related work2
Related Work
  • AlgorithmGiven a large, unlabeled collection of images
    • For each image in the collection, compute multiple candidate segmentations
    • For each segment in each segmentation, compute a histogram of “visual words”
    • Perform topic discovery on the set of all segments in the image collection (using Latent Dirichlet Allocation)
    • For each discovered topic, sort all segments by how well they are explained by this topic

Reference:

B. C. Russell, W. T. Freeman, A. A. Efros, J. Sivic, and A. Zisserman,

"Using Multiple Segmentations to Discover Objects and their Extent in Image Collections," CVPR, 2006

related work3
Related Work
  • Generating multiple segmentations
    • Produce sufficient segmentations to have a high chance of obtaining “good” segments that will contain potential objects
  • Obtaining visual words
    • SIFT descriptors for each image and quantized into 2000 visual words
    • Each image segmentis represented by a histogram of visual words contained within the segment

Reference:

B. C. Russell, W. T. Freeman, A. A. Efros, J. Sivic, and A. Zisserman,

"Using Multiple Segmentations to Discover Objects and their Extent in Image Collections," CVPR, 2006

related work4
Related Work
  • The topic discovery models
    • To analyze the collection of segments and discover ‘topics’
  • Sorting the soup of segments
    • Find good segments within each topic

Reference:

B. C. Russell, W. T. Freeman, A. A. Efros, J. Sivic, and A. Zisserman,

"Using Multiple Segmentations to Discover Objects and their Extent in Image Collections," CVPR, 2006

related work5
Related Work

Reference:

B. C. Russell, W. T. Freeman, A. A. Efros, J. Sivic, and A. Zisserman,

"Using Multiple Segmentations to Discover Objects and their Extent in Image Collections," CVPR, 2006

outline approach
Outline- Approach
  • Identifying unknown objects
  • Object graphs: modeling the topology of category predictions
  • Three-dimensional object graphs
  • Category discovery amid familiar objects
approach
Approach
  • Goal
    • Discover categories in unlabeled image collections using appearance and object-level semantic context cues
    • Generate multiple segmentation for each image and classify each region as known or unknown
    • Model the unknown regions’ surrounding contextual information in terms of object-graph
    • Group the unknown regions based on their appearance similarity and relationship to the surrounding known regions
identifying unknown objects
Identifying Unknown Objects
  • Predict which regions are likely instances of the previously learned categories
    • Learn classifiers for N categories,
    • Generate multiple segmentations per image
    • Given the region s, calculate posterior for each class
      • The “known ” object will have only one peak value among all

⇒ Lowerentropy

      • The “unknown” object will have multiple peak values among the posteriors

⇒ Higher entropy

identifying unknown objects1
Identifying Unknown Objects
  • Select a cutoff threshold equal to the midpoint in the entropy rang
    • Lighter/darker color indicate higher/lower entropy
object graphs modeling the topology of category predictions
Object Graphs: Modeling the Topology of Category Predictions
  • Model the unknown regions’ surrounding contextual information in the form of graph representation
    • Regions with similar surrounding context would have similar graphs
  • Generate superpixels for each image, except for the unknown region

Roughly 50 superpixels for each image

object graphs modeling the topology of category predictions1
Object Graphs: Modeling the Topology of Category Predictions
  • From stage 1, we have the posteriors foreach segment
  • Then, map the per-region posteriors to per-pixel posteriors
  • Calculate posteriors for each superpixel regions
object graphs modeling the topology of category predictions2
Object Graphs: Modeling the Topology of Category Predictions
  • For each unknown segment s, we compute a series of histograms using the posterior computed within its neighboring superpixels
    • Each histogram records the posteriors within ’s spatially nearest segments for each of two orientations, above and below the segment
object graphs modeling the topology of category predictions3
Object Graphs: Modeling the Topology of Category Predictions
  • Concatenate the component histograms for to produce the final object-graph descriptor
    • Use R=20 in the example

An dimensional vector

object graphs modeling the topology of category predictions4
Object Graphs: Modeling the Topology of Category Predictions

Similar object graphs for the unknown regions

three dimensional object graphs
Three-Dimensional Object Graphs
  • Is 2D object-graph a reliable descriptor?
    • Relationship between a car and the road
  • Introduce a 3D variant of the object graph
    • Use a depth information to estimate the proximity and relative orientations of surrounding familiar objects
    • Use regions rather than superpixels for 3D object-graph nodes
  • Employ the method of Hoiem et al. to estimate depth
    • D. Hoiem, A.N. Stein, A.A. Efros, and M. Hebert, “Recovering Occlusion Boundaries from a Single Image,” ICCV, 2007.
three dimensional object graphs2
Three-Dimensional Object Graphs

More robust to camera pose variations

category discovery amid familiar objects
Category Discovery amid Familiar Objects
  • Combine object-level context with region-based appearance to form groups from unknown regions
    • Object-level context: 2D or 3D object graph descriptors
    • Appearance descriptor :
      • Texton Histograms(TH) Edge filters + Gaussian filter + Laplacian-of-Gaussian filter
      • Color Histograms(CH)

Lab color space

      • Pyramid HOG(pHOG)

Three pyramid level with eight bins

category discovery amid familiar objects1
Category Discovery amid Familiar Objects
  • Similarity measure
  • Compute the affinities between all pairs of unknown regions to generate an affinity matrix
  • Use the spectral clustering method to group the regions

,where denote a kernel function for two histogram inputs:

algorithm summarization
Algorithm Summarization
  • Offline training
  • Unlabeled novel images as the input
algorithm summarization1
Algorithm Summarization
  • Generate multiple segmentations
  • Compute the posteriors and classify each segment as either known or unknown
algorithm summarization2
Algorithm Summarization
  • Generate superpixel regions and compute the posteriors
algorithm summarization3
Algorithm Summarization
  • Build an object-graph descriptor for each unknown region
algorithm summarization4
Algorithm Summarization
  • Compute affinities between all pairs of unknown regions
  • Cluster using those affinities to group the objects
outline result
Outline- Result
  • Unsupervised discovery accuracy
  • Comparison to the state of the art
  • Discovered categories: qualitative results
unsupervised discovery accuracy
Unsupervised Discovery Accuracy
  • Appearance + object graph V.S. appearance alone

Different known objects

# of unknowns increase, the accuracy of object-graph decreases

unsupervised discovery accuracy1
Unsupervised Discovery Accuracy
  • Greater improvement for high appearance variance
comparison to the state of the art
Comparison to the State of the Art
  • Compare to "Using Multiple Segmentations to Discover Objects and their Extent in Image Collections"
    • Use a bag-of-features representation with SIFT features
reference
Reference

[1] L. Yong Jae and K. Grauman, "Object-Graphs for Context- Aware Visual Category Discovery," PAMI2012.

[2] D. Hoiem, A. N. Stein, A. A. Efros, and M. Hebert, "Recovering Occlusion Boundaries from a Single Image," ICCV 2007

[3] B. C. Russell, W. T. Freeman, A. A. Efros, J. Sivic, and A. Zisserman, "Using Multiple Segmentations to Discover Objects and their Extent in Image Collections," CVPR 2006

[4] http://nlp.stanford.edu/IRbook/html/htmledition/

evaluation-of-clustering-1.html

ad