1 / 54

Interest Points

10/20/11. Interest Points. Computational Photography Derek Hoiem, University of Illinois. Galatea of the Spheres Salvador Dali. Today’s class. Review of recent materials Proj 3 favorites Interest points. Review of “The Digital Canvas”. Cutting Growing Blending Capturing Measuring

hisa
Download Presentation

Interest Points

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. 10/20/11 Interest Points Computational Photography Derek Hoiem, University of Illinois Galatea of the Spheres Salvador Dali

  2. Today’s class • Review of recent materials • Proj 3 favorites • Interest points

  3. Review of “The Digital Canvas” • Cutting • Growing • Blending • Capturing • Measuring • Reconstructing

  4. Cutting: Intelligent Scissors and Graph Cuts • You can treat the image as a graph • Nodes = pixels, edges connect neighboring pixels Intelligent Scissors: Good boundaries are a short (high gradient) path through the graph

  5. Cutting: Intelligent Scissors and Graph Cuts • You can treat the image as a graph • Nodes = pixels, edges connect neighboring pixels Source (Label 0) Cost to assign to 0 Cost to split nodes Cost to assign to 1 Sink (Label 1) Graph cut: Good boundaries are a cheap cut, where some pixels want to be foreground, and some to be background

  6. Growing: Texture synthesis and hole-filling • Transfer based on simple matching is a powerful idea • Hallucinate new pixels from their surroundings Efros & Leung, Criminisi et al.: To fill in pixels, take the center pixel from a patch that matches known surrounding values. Work from the outside in, prioritizing pixels with strong gradients.

  7. Pasting: Compositing and Blending • Gradient-domain fusion: Treat the pixels as variables to solve Adair’s Poisson Blending Result Tim’s Poisson Blending Result Poisson editing: make blended gradient as close as possible to source by solving a system of linear equations.

  8. Pasting: Compositing and Blending laplacian level 4 laplacian level 2 • Laplacian blending: blend low-frequency slowly, high frequency quickly • Blend with alpha mask values ranging from 0 to 1 laplacian level 0

  9. Image warping • Map between 2D coordinates using linear projection

  10. Image morphing = warp + blend • Warp coordinates with triangular mesh • Transfer pixels using affine transforms that map between original and intermediate triangles, then blend • Use mapping from source to target

  11. Fun with faces • Use corresponding points to average, manipulate, and extrapolate • PCA: compactly represent a large number of variables with a few coefficients by projecting onto directions of maximum variation Average face Make more feminine

  12. Pinhole camera model Vertical vanishing point (at infinity) Vanishing line Vanishing point Vanishing point • Linear projection from 3D to 2D • Be familiar with projection matrix (focal length, optical center, etc.) • Parallel lines in 3D intersect at a vanishing point in the image . Optical Center (u0, v0) . . f Z Y v . Camera Center (tx, ty, tz) u

  13. Single-view metrology and focus 5 4 3 2 1 • Can measure relative object heights using vanishing point tricks • Aperture size and focal length control amount of exposure needed, depth of field, field of view

  14. Single-view 3D Reconstruction • Technically impossible to go from 2D to 3D, but we can do it with simplifying models • Need some interaction or recognition algorithms • Uses basic VP tricks and projective geometry

  15. Project 3: Class favorites • Greg M. • Honorable mentions • Jihua • Tim • Sujay Jihua Sujay Timothy Kadich

  16. Next section of topics • Correspondence • How do we find matching patches in two images? • How can we automatically align two images of the same scene? • How do we find images with similar content? • How do we tell if two pictures are of the same person’s face? • How can we detect objects from a particular category? • Applications • Photo stitching • Object recognition • 3D Reconstruction

  17. How can we align two pictures? • What about global matching?

  18. How can we align two pictures? • Global matching? • But what if • Not just translation change, but rotation and scale? • Only small pieces of the pictures match?

  19. Today: Keypoint Matching 1. Find a set of distinctive key- points 2. Define a region around each keypoint A1 B3 3. Extract and normalize the region content A2 A3 B2 B1 4. Compute a local descriptor from the normalized region 5. Match local descriptors K. Grauman, B. Leibe

  20. Question • Why not just take every patch in the original image and find best match in second image?

  21. Goals for Keypoints Detect points that are repeatable and distinctive

  22. Key trade-offs B3 A1 A2 A3 B2 B1 Localization More Repeatable More Points Robust to occlusion Works with less texture Robust detection Precise localization Description More Selective More Robust Minimize wrong matches Deal with expected variations Maximize correct matches

  23. Which patches are easier to match? ?

  24. Choosing interest points • If you wanted to meet a friend would you say • “Let’s meet on campus.” • “Let’s meet on Green street.” • “Let’s meet at Green and Wright.” • Or if you were in a secluded area: • “Let’s meet in the Plains of Akbar.” • “Let’s meet on the side of Mt. Doom.” • “Let’s meet on top of Mt. Doom.”

  25. Choosing interest points • Corners • “Let’s meet at Green and Wright.” • Peaks/Valleys • “Let’s meet on top of Mt. Doom.”

  26. Many Existing Detectors Available Hessian & Harris [Beaudet ‘78], [Harris ‘88] Laplacian, DoG[Lindeberg ‘98], [Lowe 1999] Harris-/Hessian-Laplace [Mikolajczyk & Schmid ‘01] Harris-/Hessian-Affine [Mikolajczyk & Schmid ‘04] EBR and IBR [Tuytelaars & Van Gool ‘04] MSER[Matas ‘02] Salient Regions [Kadir & Brady ‘01] Others… K. Grauman, B. Leibe

  27. Harris Detector [Harris88] Second moment matrix Intuition: Search for local neighborhoods where the image content has two main directions (eigenvectors). K. Grauman, B. Leibe

  28. Ix Iy Harris Detector [Harris88] • Second moment matrix 1. Image derivatives Iy2 IxIy Ix2 2. Square of derivatives g(IxIy) g(Ix2) g(Iy2) 3. Gaussian filter g(sI) 4. Cornerness function – both eigenvalues are strong g(IxIy) 5. Non-maxima suppression har

  29. Matlab code for Harris Detector function [ptx, pty] = detectKeypoints(im, alpha, N) % get harris function gfil= fspecial('gaussian', [7 7], 1); % smoothing filter imblur = imfilter(im, gfil); % smooth image [Ix, Iy] = gradient(imblur); % compute gradient Ixx = imfilter(Ix.*Ix, gfil); % compute smoothed x-gradient sq Iyy = imfilter(Iy.*Iy, gfil); % compute smoothed y-gradient sq Ixy= imfilter(Ix.*Iy, gfil); har= Ixx.*Iyy - Ixy.*Ixy - alpha*(Ixx+Iyy).^2; % cornerness % get local maxima within 7x7 window maxv = ordfilt2(har, 49, ones(7)); % sorts values in each window maxv2 = ordfilt2(har, 48, ones(7)); ind = find(maxv==har & maxv~=maxv2); % get top N points [sv, sind] = sort(har(ind), 'descend'); sind = ind(sind); [pty, ptx] = ind2sub(size(im), sind(1:min(N, numel(sind))));

  30. Harris Detector – Responses [Harris88] Effect: A very precise corner detector.

  31. Harris Detector – Responses [Harris88]

  32. So far: can localize in x-y, but not scale

  33. Automatic Scale Selection How to find corresponding patch sizes? K. Grauman, B. Leibe

  34. Automatic Scale Selection • Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

  35. Automatic Scale Selection • Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

  36. Automatic Scale Selection • Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

  37. Automatic Scale Selection • Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

  38. Automatic Scale Selection • Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

  39. Automatic Scale Selection • Function responses for increasing scale (scale signature) K. Grauman, B. Leibe

  40. What Is A Useful Signature Function? • Difference of Gaussian = “blob” detector K. Grauman, B. Leibe

  41. Difference-of-Gaussian (DoG) = - K. Grauman, B. Leibe

  42. DoG – Efficient Computation • Computation in Gaussian scale pyramid Sampling withstep s4=2 s s s s Original image K. Grauman, B. Leibe

  43. Results: Lowe’s DoG K. Grauman, B. Leibe

  44. Orientation Normalization p 2 0 • Compute orientation histogram • Select dominant orientation • Normalize: rotate to fixed orientation [Lowe, SIFT, 1999] T. Tuytelaars, B. Leibe

  45. Available at a web site near you… • For most local feature detectors, executables are available online: • http://robots.ox.ac.uk/~vgg/research/affine • http://www.cs.ubc.ca/~lowe/keypoints/ • http://www.vision.ee.ethz.ch/~surf K. Grauman, B. Leibe

  46. How do we describe the keypoint?

  47. Local Descriptors • The ideal descriptor should be • Robust • Distinctive • Compact • Efficient • Most available descriptors focus on edge/gradient information • Capture texture information • Color rarely used K. Grauman, B. Leibe

  48. Local Descriptors: SIFT Descriptor • Histogram of oriented gradients • Captures important texture information • Robust to small translations / affine deformations [Lowe, ICCV 1999] K. Grauman, B. Leibe

  49. Details of Lowe’s SIFT algorithm • Run DoG detector • Find maxima in location/scale space • Remove edge points • Find all major orientations • Bin orientations into 36 bin histogram • Weight by gradient magnitude • Weight by distance to center (Gaussian-weighted mean) • Return orientations within 0.8 of peak • Use parabola for better orientation fit • For each (x,y,scale,orientation), create descriptor: • Sample 16x16 gradient mag. and rel. orientation • Bin 4x4 samples into 4x4 histograms • Threshold values to max of 0.2, divide by L2 norm • Final descriptor: 4x4x8 normalized histograms Lowe IJCV 2004

  50. Matching SIFT Descriptors • Nearest neighbor (Euclidean distance) • Threshold ratio of nearest to 2nd nearest descriptor Lowe IJCV 2004

More Related