1 / 79

Image Stitching

Image Stitching. Image Stitching. Align the images over each other camera pan ↔ translation on cylinder Blend the images together. Project 2 – image stitching. Take pictures on a tripod (or handheld) Warp images to spherical coordinates Extract features

edita
Download Presentation

Image Stitching

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Image Stitching

  2. Image Stitching • Align the images over each other • camera pan ↔ translation on cylinder • Blend the images together Image Stitching

  3. Project 2 – image stitching • Take pictures on a tripod (or handheld) • Warp images to spherical coordinates • Extract features • Align neighboring pairs using RANSAC • Write out list of neighboring translations • Correct for drift • Read in warped images and blend them • Crop the result and import into a viewer Image Stitching

  4. Matching features What do we do about the “bad” matches? Image Stitching

  5. RAndom SAmple Consensus Select one match, count inliers Image Stitching

  6. RAndom SAmple Consensus Select one match, count inliers Image Stitching

  7. Least squares fit Find “average” translation vector Image Stitching

  8. Assembling the panorama • Stitch pairs together, blend, then crop Image Stitching

  9. Problem: Drift • Error accumulation • small (vertical) errors accumulate over time • apply correction so that sum = 0 (for 360° pan.) Image Stitching

  10. (xn,yn) copy of first image Problem: Drift (x1,y1) • Solution • add another copy of first image at the end • this gives a constraint: yn = y1 • there are a bunch of ways to solve this problem • add displacement of (y1 – yn)/(n -1) to each image after the first • compute a global warp: y’ = y + ax • run a big optimization problem, incorporating this constraint • best solution, but more complicated • known as “bundle adjustment” Image Stitching

  11. Full-view (360° spherical) panoramas

  12. Full-view Panorama + + + + Image Stitching

  13. Global alignment • Pairwise matching  global energy function that involves all of the per-image pose parameters • Local adjustments to reduce blurring due to local mis-registration • Panorama recogition Image Stitching

  14. Bundle Adjustment • Developed in photogrammetry in 50´s , for general structure from motion (Szeliski and Kang 1994) • Pairwise alignment

  15. Bundle Adjustment • N features, appears in M images

  16. Bundle Adjustment • Given initial set of {Rj, Kj}, refine them together • Xik depends on xij, error-in-variable • Overweighted for feature observed many times Predicted location of feature I in image k Observed location of feature I in image k

  17. Bundle Adjustment • True bundle adjustment to estimate camera pose and 3d points • Disadvantages • A lot of variables to solve • Slow convergence

  18. Bundle Adjustment • Sparse matrix techniques • Levenberg-Marquardt http://www.ics.forth.gr/~lourakis/sba/

  19. Non-linear least squares • Linear approximation of residual • allows quadratic approximation of sum- of-squares Minimization corresponds to finding zeros of derivative N Levenberg-Marquardt: extra term to deal with singular N (decrease/increase l if success/failure to descent) (extra term = descent term)

  20. P1 P2 P3 M U1 U2 W U3 WT V 3xn (in general much larger) 12xm Bundle Adjustment • Jacobian of has sparse block structure • cameras independent of other cameras, • points independent of other points im.pts. view 1 Needed for non-linear minimization

  21. U-WV-1WT WT V 3xn 11xm Bundle Adjustment Eliminate dependence of camera/motion parameters on structure parameters Note in general 3n >> 11m Allows much more efficient computations e.g. 100 views,10000 points, solve 1000x1000, not 30000x30000 Often still band diagonal use sparse linear algebra algorithms

  22. Up Vector Selection • Keep the upright direction at the final stitched image • Multiply each roation by a global rotation Rg

  23. Up Vector Selection • rg1 is the smallest eigenvector of the scatter or moment matrix spanned by the individual camera rotation x-vectors • Full rotation matrix Rg

  24. Parallax Removal • Blurry or ghosting • Unmodeled radial distortion • 3D parallax: failure to rotate the camera around its optical center • Small scene motion, large scale scene motion • Different approaches for them

  25. Parallax Removal • Based on bundle adjustment, compute 3D point location, then reproject it to images Deghosting a mosaic with motion parallax (Shum and Szeliski 2000) c 2000 IEEE: (a) composite with parallax; (b) after a single deghosting step (patch size 32); (c) after multiple steps (sizes 32, 16 and 8)

  26. Recognizing Panoramas Matthew Brown & David Lowe ICCV’2003

  27. Recognizing Panoramas [Brown & Lowe, ICCV’03] Image Stitching

  28. Finding the panoramas • Find all pairwise image overlaps • Find connected components in the overlap graph Image Stitching

  29. Finding the panoramas Image Stitching

  30. Finding the panoramas Image Stitching

  31. Get you own free copy http://get.live.com/photogallery/overview Image Stitching

  32. Rec.pano.: system components • Feature detection and description • more uniform point density • Fast matching (hash table) • RANSAC filtering of matches • Intensity-based verification • Incremental bundle adjustment • [M. Brown, R. Szeliski, and S. Winder. Multi-image matching using multi-scale oriented patches, CVPR'2005] Image Stitching

  33. Probabilistic Feature Matching Image Stitching

  34. RANSAC motion model Image Stitching

  35. RANSAC motion model Image Stitching

  36. RANSAC motion model Image Stitching

  37. How well does this work? Test on 100s of examples…

  38. How well does this work? Test on 100s of examples… …still too many failures (5-10%)for consumer application

  39. Matching Mistakes: False Positive Image Stitching

  40. Matching Mistakes: False Positive Image Stitching

  41. Matching Mistake: False Negative • Moving objects: large areas of disagreement Image Stitching

  42. Matching Mistakes • Accidental alignment • repeated / similar regions • Failed alignments • moving objects / parallax • low overlap • “feature-less” regions(more variety?) • No 100% reliable algorithm? Image Stitching

  43. How can we fix these? • Tune the feature detector • Tune the feature matcher (cost metric) • Tune the RANSAC stage (motion model) • Tune the verification stage • Use “higher-level” knowledge • e.g., typical camera motions • → Sounds like a big “learning” problem • Need a large training/test data set (panoramas) Image Stitching

  44. Direct vs. Feature-based Alignment • Feature-based • Confused in too textural or not textured enough region • Features should be distributed evenly over the image • Relies on robust feature detection and matching schemes • Direct • Limited range of convergence, even hierarchical framework • Work for video, fail for partially overlapping images

  45. Image Compositing • How to produce the final mosaic image? • Compositing surface (flat, cylindrical, spherical, etc) • View (reference image) • Select and blend pixels to minimize visible seams, blur, and ghosting • Surface parameterization, pixel and seam selection, blending, exposure compensation…

  46. Compositing Surface • If only a few images --- flat panorama • select one image as reference and warp all the others to the reference coordinate system • Still perspective projection

  47. mosaic plane Compositing Surface • Large field of view • Large distortion • Usually use a cylindrical or spherical projection • Environment map • Tradeoff between • keeping local appearance undistorted, and • providing a reasonably uniform sampling of the environment

  48. View Selection • Which part of the scene will be centered in the final composition • Average depth • Middle image in a 360 panorama • Image that contains the object of greatest interest • User control

  49. Y X Z Coordinate Transformations • Mapping between input and output pixels • Flat surface: homography • Analytic form (cylindrical, spherical…) • Pixel in panorama -> 3D view ray -> image • Precompute lookup tables cylindrical image

  50. Sampling Issues • Low-resolution panorama than input images • Prefilter input images to avoid aliasing

More Related