1 / 50

Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj

Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan Rice University. Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj. Sensor Data Deluge. Concise Models. large wavelet coefficients (blue = 0). pixels.

amal
Download Presentation

Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Go With The Flow A New Manifold Modeling and Learning Framework for Image EnsemblesAswin C. Sankaranarayanan Rice University Richard G. BaraniukChinmayHegdeSriramNagaraj

  2. Sensor Data Deluge

  3. Concise Models largewaveletcoefficients (blue = 0) pixels • Efficient processing / compression requires concise representation • Sparsity of an individual image

  4. Concise Models • Efficient processing / compression requires concise representation • Our interest in this talk: Collections of images

  5. Concise Models • Our interest in this talk: Collections of image parameterized by q\inQ • translations of an object • q: x-offset and y-offset • wedgelets • q: orientation and offset • rotations of a 3D object • q: pitch, roll, yaw

  6. Concise Models • Our interest in this talk: Collections of image parameterized by q\inQ • translations of an object • q: x-offset and y-offset • wedgelets • q: orientation and offset • rotations of a 3D object • q: pitch, roll, yaw • Image articulation manifold

  7. Image Articulation Manifold • N-pixel images: • K-dimensional articulation space • Thenis a K-dimensional manifoldin the ambient space • Very concise model articulation parameter space

  8. Smooth IAMs • N-pixel images: • Local isometry: image distance parameter space distance • Linear tangent spacesare close approximationlocally articulation parameter space

  9. Smooth IAMs • N-pixel images: • Local isometry: image distance parameter space distance • Linear tangent spacesare close approximationlocally articulation parameter space

  10. Ex: Manifold Learning LLE ISOMAP LE HE Diff. Geo… • K=1rotation

  11. Ex: Manifold Learning • K=2rotation and scale

  12. Theory/Practice Disconnect: Smoothness • Practical image manifolds are not smooth! • If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes] Tangent approximations ? Isometry ?

  13. Theory/Practice Disconnect: Smoothness • Practical image manifolds are not smooth! • If images have sharp edges, then manifold is everywhere non-differentiable [Donohoand Grimes] Tangent approximations ? Isometry ?

  14. Failure of Tangent Plane Approx. • Ex: cross-fading when synthesizing / interpolating images that should lie on manifold Input Image Input Image Linear path Geodesic

  15. Failure of Local Isometry • Ex: translation manifold all blue imagesare equidistantfrom the red image • Local isometry • satisfied only when sampling is dense

  16. Tools for manifold processing Geodesics, exponential maps, log-maps, Riemannian metrics, Karcher means, … Smooth diff manifold Algebraic manifolds Data manifolds LLE, kNN graphs Point cloud model

  17. The concept ofTransport operators Beyond point cloud model for image manifolds Example

  18. Example: Translation • 2D Translation manifold barring boundary related issues • Set of all transport operators = • Beyond a point cloud model • Action of the articulation is more accurate and meaningful

  19. Optical Flow • Generalizing this idea: Pixel correspondances • Idea: OF between two images is a natural and accurate transport operator OF from I1 to I2 I1 and I2 (Figures from Ce Liu’s optical flow page)

  20. Optical Flow Transport • Consider a reference imageand a K-dimensional articulation • Collect optical flows fromto all images reachable by aK-dimensional articulation IAM Articulations

  21. Optical Flow Transport OFM at • Consider a reference imageand a K-dimensional articulation • Collect optical flows fromto all images reachable by aK-dimensional articulation • Theorem: Collection of OFs is a smooth, K-dimensional manifold(even if IAM is not smooth) IAM Articulations

  22. OFM is Smooth (Rotation) Pixel intensityat 3 points Intensity I(θ) Flow (nearly linear) Op. flow v(θ) Articulation θ in [⁰]

  23. Main results OFM at • Local model at each • Each point on the OFM defines a transport operator • Each transport operator maps to one of its neighbors • For a large class of articulations, OFMs are smooth and locally isometric • Traditional manifold processing techniques work on OFMs IAM Articulations

  24. Linking it all together OFM at Nonlinear dim. reduction IAM The non-differentiablity does not dissappear --- it is embedded in the mapping from OFM to the IAM. However, this is a known map Articulations

  25. The Story So Far… OFM at Tangent space at IAM IAM Articulations Articulations

  26. Input Image Input Image IAM Linear path Geodesic OFM

  27. OFM Synthesis

  28. Manifold Learning 2D rotations ISOMAP embedding error for OFM and IAM Reference image

  29. Manifold Learning 2D rotations Embedding of OFM Reference image

  30. OFM Manifold Learning Data 196 images of two bears moving linearlyand independently Task Find low-dimensional embedding IAM OFM

  31. OFM ML + Parameter Estimation Data 196 images of a cup moving on a plane Task 1 Find low-dimensional embedding Task 2 Parameter estimation for new images(tracing an “R”) OFM IAM

  32. Karcher Mean • Point on the manifold such that the sum of geodesic distances to every other point is minimized • Important concept in nonlinear data modeling, compression, shape analysis [Srivastava et al] 10 images from an IAM ground truth KM linear KM OFM KM

  33. Manifold Charting • Goal: build a generative model for an entire IAM/OFM based on a small number of base images • Ex: cube rotating about axis. All cube images can be representing using 4 reference images + OFMs • Many applications • selection of target templates for classification • “next-view” selection for adaptive sensing applications

  34. Summary • IAMs a useful concise model for many image processing problems involving image collections and multiple sensors/viewpoints • But practical IAMs are non-differentiable • IAM-based algorithms have not lived up to their promise • Optical flow manifolds (OFMs) • smooth even when IAM is not • OFM ~ nonlinear tangent space • support accurate image synthesis, learning, charting, … • Barely discussed here: OF enables the safe extension of differential geometry concepts • Log/Exp maps, Karcher mean, parallel transport, …

  35. Open Questions • Our treatment is specific to image manifolds under brightness constancy • What are the natural transport operators for other data manifolds? dsp.rice.edu

  36. Related Work • Analytic transport operators • transport operator has group structure [Xiao and Rao07][Culpepper and Olshausen09] [Miller and Younes01] [Tuzel et al 08] • non-linear analytics [Dollar et al 06] • spatio-temporal manifolds [Li and Chellappa10] • shape manifolds [Klassen et al 04] • Analytic approach limited to a small class of standard image transformations (ex: affine transformations, Lie groups) • In contrast, OFM approach works reliably with real-world image samples (point clouds) and broader class of transformations

  37. Limitations • Brightness constancy • Optical flow is no longer meaningful • Occlusion • Undefined pixel flow in theory, arbitrary flow estimates in practice • Heuristics to deal with it • Changing backgrounds etc. • Transport operator assumption too strict • Sparse correspondences ?

  38. Open Questions Theorem: random measurements stably embed aK-dim manifoldwhp[B, Wakin, FOCM’08] Q: Is there an analogousresult for OFMs?

  39. Image Articulation Manifold Tangent space at • Linear tangent space at is K-dimensional • provides a mechanism to transport along manifold • problem: since manifold is non-differentiable, tangent approximation is poor • Our goal: replace tangent spacewith new transport operator that respects the nonlinearity of the imaging process IAM Articulations

  40. OFM Implementation details Reference Image

  41. Pairwise distances and embedding

  42. Flow Embedding

  43. Occlusion • Detect occlusion using forward-backward flow reasoning • Remove occluded pixel computations • Heuristic --- formal occlusion handling is hard Occluded

  44. History of Optical Flow • Dark ages (<1985) • special cases solved • LBC an under-determined set of linear equations • Horn and Schunk (1985) • Regularization term: smoothness prior on the flow • Brox et al (2005) • shows that linearization of brightness constancy (BC) isa bad assumption • develops optimization framework to handle BC directly • Brox et al (2010), Black et al (2010), Liu et al (2010) • practical systems with reliable code

More Related