Manifold Learning in the Wild
This presentation is the property of its rightful owner.
Sponsored Links
1 / 36

Richard G. Baraniuk Chinmay Hegde PowerPoint PPT Presentation


  • 50 Views
  • Uploaded on
  • Presentation posted in: General

Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan Rice University. Richard G. Baraniuk Chinmay Hegde. Sensor Data Deluge. Internet Scale Databases. Tremendous size of corpus of available data

Download Presentation

Richard G. Baraniuk Chinmay Hegde

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Richard g baraniuk chinmay hegde

Manifold Learning in the Wild

A New Manifold Modeling and Learning Framework for Image EnsemblesAswin C. Sankaranarayanan

Rice University

Richard G. BaraniukChinmayHegde


Sensor data deluge

Sensor Data Deluge


Internet scale databases

Internet Scale Databases

  • Tremendous sizeof corpus of available data

    • Google Image Search of “Notre Dame Cathedral” yields 3m results 3Tbof data


Concise models

Concise Models

  • Efficient processing / compression requires concise representation

  • Our interest in this talk: Collections of images


Concise models1

Concise Models

  • Our interest in this talk:

    Collections of image

    parameterized by

q\inQ

  • translations of an object

    • q: x-offset and y-offset

  • wedgelets

    • q: orientation and offset

  • rotations of a 3D object

    • q: pitch, roll, yaw


Concise models2

Concise Models

  • Our interest in this talk:

    Collections of image

    parameterized by

q\inQ

  • translations of an object

    • q: x-offset and y-offset

  • wedgelets

    • q: orientation and offset

  • rotations of a 3D object

    • q: pitch, roll, yaw

  • Image articulation manifold


Image articulation manifold

Image Articulation Manifold

  • N-pixel images:

  • K-dimensional articulation space

  • Thenis a K-dimensional manifoldin the ambient space

  • Very concise model

    • Can be learnt using

      Non-linear dim. reduction

articulation parameter space


Ex manifold learning

Ex: Manifold Learning

LLE

ISOMAP

LE

HE

Diff. Geo…

  • K=1rotation


Ex manifold learning1

Ex: Manifold Learning

  • K=2rotation and scale


Smooth iams

Smooth IAMs

  • N-pixel images:

  • Local isometryimage distance parameter space distance

  • Linear tangent spacesare close approximationlocally

  • Low dimensional

    articulation space

articulation parameter space


Smooth iams1

Smooth IAMs

  • N-pixel images:

  • Local isometryimage distance parameter space distance

  • Linear tangent spacesare close approximationlocally

  • Low dimensional

    articulation space

articulation parameter space


Smooth iams2

Smooth IAMs

  • N-pixel images:

  • Local isometryimage distance parameter space distance

  • Linear tangent spacesare close approximationlocally

  • Lowdimensional

    articulation space

articulation parameter space


Theory practice disconnect isometry

Theory/Practice Disconnect Isometry

  • Ex:translation manifold

    all blue images are equidistant from the red image

  • Local isometry

    • satisfied only when sampling is dense


Theory practice disconnect nuisance articulations

Theory/Practice DisconnectNuisance articulations

  • Unsupervised data, invariably, has additional undesired articulations

    • Illumination

    • Background clutter, occlusions, …

  • Image ensemble is no longer low-dimensional


Image representations

Image representations

  • Conventional representation for an image

    • A vector of pixels

    • Inadequate!

pixel image


Image representations1

Image representations

  • Replace vector of pixels with an abstract bagof features

    • Ex: SIFT (Scale Invariant Feature Transform) selects keypoint locations in an image and computes keypoint descriptorsfor each keypoint

    • Very popular in many many vision problems


Image representations2

Image representations

  • Replace vector of pixels with an abstract bagof features

    • Ex: SIFT (Scale Invariant Feature Transform) selects keypoint locations in an image and computes keypoint descriptorsfor each keypoint

    • Keypoint descriptors are local; it is very easy to make them robust to nuisance imaging parameters


Loss of geometrical info

Loss of Geometrical Info

  • Bag of features representations hide potentially useful image geometry

Image space

Keypoint space

  • Goal: make salient image geometrical info more explicit for exploitation


Key idea

Key idea

  • Keypoint space can be endowed with a rich low-dimensional structure in many situations


Key idea1

Key idea

  • Keypoint space can be endowed with a rich low-dimensional structure in many situations

  • Mechanism: define kernels ,between keypoint locations, keypoint descriptors


Keypoint kernel

Keypoint Kernel

  • Keypoint space can be endowed with a rich low-dimensional structure in many situations

  • Mechanism: define kernels ,between keypoint locations, keypoint descriptors

  • Joint keypoint kernel between two images

    is given by


Many possible kernels

Many Possible Kernels

  • Euclidean kernel

  • Gaussian kernel

  • Polynomial kernel

  • Pyramid match kernel [Grauman et al. ’07]

  • Many others


Keypoint kernel1

Keypoint Kernel

  • Joint keypoint kernel between two images

    is given by

  • Using Euclidean/Gaussian (E/G) combination yields


From kernel to metric

From Kernel to Metric

Lemma: The E/G keypoint kernel is a Mercer kernel

  • enables algorithms such as SVM

    Lemma: The E/G keypoint kernel induces a metricon the space of images

  • alternative to conventional L2 distance between images

  • keypoint metric robust to nuisance imaging parameters, occlusion, clutter, etc.


Keypoint geometry

Keypoint Geometry

Theorem:Under the metric induced by the kernel

certain ensembles of articulating images formsmooth, isometric manifolds

  • Keypointrepresentation compact, efficient, and …

  • Robust to illumination variations, non-stationary backgrounds, clutter, occlusions


Keypoint geometry1

Keypoint Geometry

Theorem: Under the metric induced by the kernel

certain ensembles of articulating images formsmooth, isometric manifolds

  • In contrast: conventional approach to image fusion via image articulation manifolds (IAMs) fraught with non-differentiability (due to sharp image edges)

    • not smooth

    • not isometric


Application manifold learning

Application: Manifold Learning

  • 2D Translation


Application manifold learning1

Application: Manifold Learning

  • 2D Translation

  • IAM KAM


Manifold learning in the wild

Manifold Learning in the Wild

  • Rice University’s Duncan Hall Lobby

    • 158 images

    • 360° panorama using handheld camera

    • Varying brightness, clutter


Manifold learning in the wild1

Manifold Learning in the Wild

  • Duncan Hall Lobby

  • Ground truth using state of the art structure-from-motion software

Ground truth

IAM

KAM


Manifold learning in the wild2

Manifold Learning in the Wild

  • Rice University’s Brochstein Pavilion

    • 400 outdoor images of a building

    • occlusions, movement in foreground, varying background


Manifold learning in the wild3

Manifold Learning in the Wild

  • Brochstein Pavilion

    • 400 outdoor images of a building

    • occlusions, movement in foreground, background

IAM

KAM


Internet scale imagery

Internet scale imagery

  • Notre-dame cathedral

    • 738 images

    • Collected from Flickr

    • Large variations in illumination (night/day/saturations), clutter (people, decorations),camera parameters (focal length, fov, …)

    • Non-uniform sampling of the space


Organization

Organization

  • k-nearest neighbors


Organization1

Organization

  • “geodesics’

“zoom-out”

“Walk-closer”

3D rotation


Summary

Summary

  • Challenges for manifold learning in the wild are both theoretical and practical

  • Need for novel image representations

    • Sparse features

      • Robustness to outliers, nuisance articulations, etc.

      • Learning in the wild: unsupervised imagery

  • Promise lies in fast methods that exploit only neighborhood properties

    • No complex optimization required


  • Login