1 / 62

Image alignment

This resource provides an in-depth look at various image alignment techniques, focusing on feature-based approaches for matching and fitting transformations between images. It covers the theoretical aspects of affine transformations, homographies, and robust methods such as RANSAC, emphasizing challenges like occlusion and intensity changes. Practical applications, including panorama stitching and searching the night sky, showcase the real-world relevance of these techniques. Ideal for students and professionals in computer vision and image processing, this material combines theoretical insights with practical implementations.

lula
Download Presentation

Image alignment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Image alignment Image from http://graphics.cs.cmu.edu/courses/15-463/2010_fall/

  2. A look into the past http://blog.flickr.net/en/2010/01/27/a-look-into-the-past/

  3. A look into the past • Leningrad during the blockade http://komen-dant.livejournal.com/345684.html

  4. Bing streetside images http://www.bing.com/community/blogs/maps/archive/2010/01/12/new-bing-maps-application-streetside-photos.aspx

  5. Image alignment: Applications Panorama stitching

  6. Image alignment: Applications Recognitionof objectinstances

  7. Image alignment: Challenges Small degree of overlap Intensity changes Occlusion,clutter

  8. Feature-based alignment • Search sets of feature matches that agree in terms of: • Local appearance • Geometric configuration ?

  9. Feature-based alignment: Overview • Alignment as fitting • Affine transformations • Homographies • Robust alignment • Descriptor-based feature matching • RANSAC • Large-scale alignment • Inverted indexing • Vocabulary trees • Application: searching the night sky

  10. Alignment as fitting • Previous lectures: fitting a model to features in one image M Find model M that minimizes xi

  11. xi xi ' T Alignment as fitting • Previous lectures: fitting a model to features in one image • Alignment: fitting a model to a transformation between pairs of features (matches) in two images M Find model M that minimizes xi Find transformation Tthat minimizes

  12. 2D transformation models • Similarity(translation, scale, rotation) • Affine • Projective(homography)

  13. Let’s start with affine transformations • Simple fitting procedure (linear least squares) • Approximates viewpoint changes for roughly planar objects and roughly orthographic cameras • Can be used to initialize fitting for more complex models

  14. Fitting an affine transformation • Assume we know the correspondences, how do we get the transformation? Want to find M, t to minimize

  15. Fitting an affine transformation • Assume we know the correspondences, how do we get the transformation?

  16. Fitting an affine transformation • Linear system with six unknowns • Each match gives us two linearly independent equations: need at least three to solve for the transformation parameters

  17. Fitting a plane projective transformation • Homography: plane projective transformation (transformation taking a quad to another arbitrary quad)

  18. Homography • The transformation between two views of a planar surface • The transformation between images from two cameras that share the same center

  19. Application: Panorama stitching Source: Hartley & Zisserman

  20. Fitting a homography • Recall: homogeneous coordinates Converting fromhomogeneousimage coordinates Converting tohomogeneousimage coordinates

  21. Fitting a homography • Recall: homogeneous coordinates • Equation for homography: Converting from homogeneousimage coordinates Converting to homogeneousimage coordinates

  22. Fitting a homography • Equation for homography: 3 equations, only 2 linearly independent

  23. Direct linear transform • H has 8 degrees of freedom (9 parameters, but scale is arbitrary) • One match gives us two linearly independent equations • Homogeneous least squares: find h minimizing ||Ah||2 • Eigenvector of ATA corresponding to smallest eigenvalue • Four matches needed for a minimal solution

  24. Robust feature-based alignment • So far, we’ve assumed that we are given a set of “ground-truth” correspondences between the two images we want to align • What if we don’t know the correspondences?

  25. Robust feature-based alignment • So far, we’ve assumed that we are given a set of “ground-truth” correspondences between the two images we want to align • What if we don’t know the correspondences? ?

  26. Robust feature-based alignment

  27. Robust feature-based alignment • Extract features

  28. Robust feature-based alignment • Extract features • Compute putative matches

  29. Robust feature-based alignment • Extract features • Compute putative matches • Loop: • Hypothesize transformation T

  30. Robust feature-based alignment • Extract features • Compute putative matches • Loop: • Hypothesize transformation T • Verify transformation (search for other matches consistent with T)

  31. Robust feature-based alignment • Extract features • Compute putative matches • Loop: • Hypothesize transformation T • Verify transformation (search for other matches consistent with T)

  32. Generating putative correspondences ?

  33. ( ) ( ) Generating putative correspondences • Need to compare feature descriptors of local patches surrounding interest points ? ? = featuredescriptor featuredescriptor

  34. Feature descriptors • Recall: feature detection and description

  35. Feature descriptors • Simplest descriptor: vector of raw intensity values • How to compare two such vectors? • Sum of squared differences (SSD) • Not invariant to intensity change • Normalized correlation • Invariant to affine intensity change

  36. Disadvantage of intensity vectors as descriptors • Small deformations can affect the matching score a lot

  37. Review: Alignment • 2D alignment models • Feature-based alignment outline • Descriptor matching

  38. Feature descriptors: SIFT • Descriptor computation: • Divide patch into 4x4 sub-patches • Compute histogram of gradient orientations (8 reference angles) inside each sub-patch • Resulting descriptor: 4x4x8 = 128 dimensions David G. Lowe. "Distinctive image features from scale-invariant keypoints.”IJCV 60 (2), pp. 91-110, 2004.

  39. Feature descriptors: SIFT • Descriptor computation: • Divide patch into 4x4 sub-patches • Compute histogram of gradient orientations (8 reference angles) inside each sub-patch • Resulting descriptor: 4x4x8 = 128 dimensions • Advantage over raw vectors of pixel values • Gradients less sensitive to illumination change • Pooling of gradients over the sub-patches achieves robustness to small shifts, but still preserves some spatial information David G. Lowe. "Distinctive image features from scale-invariant keypoints.”IJCV 60 (2), pp. 91-110, 2004.

  40. Feature matching • Generating putative matches: for each patch in one image, find a short list of patches in the other image that could match it based solely on appearance ?

  41. Problem: Ambiguous putative matches Source: Y. Furukawa

  42. Rejection of unreliable matches • How can we tell which putative matches are more reliable? • Heuristic: compare distance of nearest neighbor to that of second nearest neighbor • Ratio of closest distance to second-closest distance will be highfor features that are not distinctive • Threshold of 0.8 provides good separation David G. Lowe. "Distinctive image features from scale-invariant keypoints.”IJCV 60 (2), pp. 91-110, 2004.

  43. RANSAC • The set of putative matches contains a very high percentage of outliers • RANSAC loop: • Randomly select a seed group of matches • Compute transformation from seed group • Find inliers to this transformation • If the number of inliers is sufficiently large, re-compute least-squares estimate of transformation on all of the inliers • Keep the transformation with the largest number of inliers

  44. RANSAC example: Translation Putative matches

  45. RANSAC example: Translation Select one match, count inliers

  46. RANSAC example: Translation Select one match, count inliers

  47. RANSAC example: Translation Select translation with the most inliers

  48. Scalability: Alignment to large databases • What if we need to align a test image with thousands or millions of images in a model database? • Efficient putative match generation • Approximate descriptor similarity search, inverted indices Test image ? Model database

  49. Large-scale visual search Reranking/ Geometric verification Inverted indexing Figure from: Kristen Grauman and Bastian Leibe, Visual Object Recognition, Synthesis Lectures on Artificial Intelligence and Machine Learning, April 2011, Vol. 5, No. 2, Pages 1-181

  50. Example indexing technique: Vocabulary trees Test image Vocabulary tree with inverted index Database D. Nistér and H. Stewénius, Scalable Recognition with a Vocabulary Tree, CVPR 2006

More Related