1 / 80

Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj

Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan Rice University. Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj. Sensor Data Deluge. Concise Models.

salim
Download Presentation

Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image EnsemblesAswin C. Sankaranarayanan Rice University Richard G. BaraniukChinmayHegdeSriramNagaraj

  2. Sensor Data Deluge

  3. Concise Models • Efficient processing / compression requires concise representation • Our interest in this talk: Collections of images

  4. Concise Models • Our interest in this talk: Collections of image parameterized by q\inQ • translations of an object • q: x-offset and y-offset • wedgelets • q: orientation and offset • rotations of a 3D object • q: pitch, roll, yaw

  5. Concise Models • Our interest in this talk: Collections of image parameterized by q\inQ • translations of an object • q: x-offset and y-offset • wedgelets • q: orientation and offset • rotations of a 3D object • q: pitch, roll, yaw • Image articulation manifold

  6. Image Articulation Manifold • N-pixel images: • K-dimensional articulation space • Thenis a K-dimensional manifoldin the ambient space • Very concise model articulation parameter space

  7. Smooth IAMs • N-pixel images: • Local isometryimage distance parameter space distance • Linear tangent spacesare close approximationlocally • Low dimensional articulation space articulation parameter space

  8. Smooth IAMs • N-pixel images: • Local isometryimage distance parameter space distance • Linear tangent spacesare close approximationlocally • Low dimensional articulation space articulation parameter space

  9. Smooth IAMs • N-pixel images: • Local isometryimage distance parameter space distance • Linear tangent spacesare close approximationlocally • Lowdimensional articulation space articulation parameter space

  10. Ex: Manifold Learning LLE ISOMAP LE HE Diff. Geo… • K=1rotation

  11. Ex: Manifold Learning • K=2rotation and scale

  12. Theory/Practice Disconnect Smoothness • Practical image manifolds are not smooth! • If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes] Tangent approximations ?

  13. Theory/Practice Disconnect Smoothness • Practical image manifolds are not smooth! • If images have sharp edges, then manifold is everywhere non-differentiable [Donohoand Grimes] Tangent approximations ?

  14. Failure of Tangent Plane Approx. • Ex: cross-fading when synthesizing / interpolating images that should lie on manifold Input Image Input Image Linear path Geodesic

  15. Theory/Practice Disconnect Isometry • Ex: translation manifold all blue images are equidistant from the red image • Local isometry • satisfied only when sampling is dense

  16. Theory/Practice DisconnectNuisance articulations • Unsupervised data, invariably, has additional undesired articulations • Illumination • Background clutter, occlusions, … • Image ensemble is no longer low-dimensional

  17. Image representations • Conventional representation for an image • A vector of pixels • Inadequate! pixel image

  18. Image representations • Conventional representation for an image • A vector of pixels • Inadequate! • Remainder of the talk • TWOnovel image representations that alleviate the theoretical/practical challenges in manifold learning on image ensembles

  19. Transport operators for image manifolds

  20. The concept ofTransport operators Beyond point cloud model for image manifolds Example

  21. Example: Translation • 2D Translation manifold • Set of all transport operators = • Beyond a point cloud model • Action of the articulation is more accurate and meaningful

  22. Optical Flow • Generalizing this idea: Pixel correspondances • Idea: OF between two images is a natural and accurate transport operator OF from I1 to I2 I1 and I2 (Figures from Ce Liu’s optical flow page)

  23. Optical Flow Transport • Consider a reference imageand a K-dimensional articulation • Collect optical flows fromto all images reachable by aK-dimensional articulation IAM Articulations

  24. Optical Flow Transport • Consider a reference imageand a K-dimensional articulation • Collect optical flows fromto all images reachable by aK-dimensional articulation IAM Articulations

  25. Optical Flow Transport OFM at • Consider a reference imageand a K-dimensional articulation • Collect optical flows fromto all images reachable by aK-dimensional articulation • Collection of OFs is a smooth, K-dimensional manifold(even if IAM is not smooth) for large class of articulations IAM Articulations

  26. OFM is Smooth (Rotation) Pixel intensityat 3 points Intensity I(θ) Flow (nearly linear) Op. flow v(θ) Articulation θ in [⁰]

  27. Main results OFM at • Local model at each • Each point on the OFM defines a transport operator • Each transport operator maps to one of its neighbors • For a large class of articulations, OFMs are smooth and locally isometric • Traditional manifold processing techniques work on OFMs IAM Articulations

  28. Linking it all together OFM at Nonlinear dim. reduction IAM The non-differentiability does not dissappear --- it is embedded in the mapping from OFM to the IAM. However, this is a known map Articulations

  29. The Story So Far… OFM at Tangent space at IAM IAM Articulations Articulations

  30. Input Image Input Image IAM Linear path Geodesic OFM

  31. OFM Manifold Learning Data 196 images of two bears moving linearlyand independently Task Find low-dimensional embedding IAM OFM

  32. OFM ML + Parameter Estimation Data 196 images of a cup moving on a plane Task 1 Find low-dimensional embedding Task 2 Parameter estimation for new images(tracing an “R”) OFM IAM

  33. Karcher Mean • Point on the manifold such that the sum of geodesic distances to every other point is minimized • Important concept in nonlinear data modeling, compression, shape analysis [Srivastava et al] 10 images from an IAM ground truth KM linear KM OFM KM

  34. Sparse keypoint-based image representation

  35. Image representations • Conventional representation for an image • A vector of pixels • Inadequate! pixel image

  36. Image representations • Replace vector of pixels with an abstract bagof features • Ex: SIFT (Scale Invariant Feature Transform) selects keypoint locations in an image and computes keypoint descriptorsfor each keypoint

  37. Image representations • Replace vector of pixels with an abstract bagof features • Ex: SIFT (Scale Invariant Feature Transform) selects keypoint locations in an image and computes keypoint descriptorsfor each keypoint • Feature descriptors are local; it is very easy to make them robust to nuisance imaging parameters

  38. Loss of Geometrical Info • Bag of features representations hide potentially useful image geometry Image space Keypoint space • Goal: make salient image geometrical info more explicit for exploitation

  39. Key idea • Keypoint space can be endowed with a rich low-dimensional structure in many situations • Mechanism: define kernels ,between keypoint locations, keypoint descriptors

  40. Keypoint Kernel • Keypoint space can be endowed with a rich low-dimensional structure in many situations • Mechanism: define kernels ,between keypoint locations, keypoint descriptors • Joint keypoint kernel between two images is given by

  41. Keypoint Geometry Theorem: Under the metric induced by the kernel certain ensembles of articulating images formsmooth, isometric manifolds • In contrast: conventional approach to image fusion via image articulation manifolds (IAMs) fraught with non-differentiability (due to sharp image edges) • not smooth • not isometric

  42. Application: Manifold Learning • 2D Translation

  43. Application: Manifold Learning • 2D Translation • IAM KAM

  44. Manifold Learning in the Wild • Rice University’s Duncan Hall Lobby • 158 images • 360° panorama using handheld camera • Varying brightness, clutter

  45. Manifold Learning in the Wild • Duncan Hall Lobby • Ground truth using state of the art structure-from-motion software Ground truth IAM KAM

  46. Internet scale imagery • Notre-dame cathedral • 738 images • Collected from Flickr • Large variations in illumination (night/day/saturations), clutter (people, decorations),camera parameters (focal length, fov, …) • Non-uniform sampling of the space

  47. Organization • k-nearest neighbors

  48. Organization • “geodesics’ “zoom-out” “Walk-closer” 3D rotation

  49. Summary • Need for novel image representations • Transport operators • Enables differentiability and meaningful transport • Sparse features • Robustness to outliers, nuisance articulations, etc. • Learning in the wild: unsupervised imagery • True power of manifold signal processing lies in fast algorithms that mainly use neighbor relationships • What are compelling applications where such methods can achieve state of the art performance ?

More Related