1 / 65

Robust Feature-Based Registration of Remotely Sensed Data

Robust Feature-Based Registration of Remotely Sensed Data. Nathan S. Netanyahu Dept. of Computer Science, Bar-Ilan University and Center for Automation Research, University of Maryland Collaborators: Jacqueline Le Moigne NASA / Goddard Space Flight Center

minniep
Download Presentation

Robust Feature-Based Registration of Remotely Sensed Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Robust Feature-Based Registration of Remotely Sensed Data Nathan S. Netanyahu Dept. of Computer Science, Bar-Ilan University and Center for Automation Research, University of Maryland Collaborators: Jacqueline Le Moigne NASA / Goddard Space Flight Center David M. Mount University of Maryland Arlene A. Cole-Rhodes, Kisha L. Johnson Morgan State University, Maryland Roger D. EastmanLoyola College of Maryland Ardeshir Goshtasby Wright State University, Ohio Jeffrey G. Masek, Jeffrey Morisette NASA / Goddard Space Flight Center Antonio Plaza University of Extremadura, Spain Harold Stone NEC Research Institute (Ret.) Ilya Zavorin CACI International, Maryland Shirley Barda, Boris Sherman Applied Materials, Inc., Israel Yair Kapach Bar-Ilan University

  2. What is Image Registration / Alignment / Matching? The above image is rotated and shifted with respect to the left image.

  3. Sensor Webs, Constellation, and Exploration Planning and Scheduling Automatic Multiple Source Integration Satellite/Orbiter, and In-Situ Data Intelligent Navigation and Decision Making

  4. Selected NASA Earth Science Missions

  5. MODIS Satellite System From the NASA MODIS website

  6. MODIS Satellite Specifications

  7. Landsat 7 Satellite System New Orleans, before and after Katrina 2005 (from the USGS Landsat website)

  8. Landsat 7 Satellite Specifications

  9. IKONOS Satellite System

  10. IKONOS Satellite Specifications

  11. Motivation • A crucial, fundamental step in image analysis tasks, where final information is obtained by the combination / integration of multiple data sources.

  12. Motivation / Applications • Computer Vision (target localization, quality control, stereo matching) • Medical Imaging (combining CT and MRI data, tumor growth monitoring, treatment verification) • Remote Sensing (classification, environmental monitoring, change detection, image mosaicing, weather forecasting, integration into GIS)

  13. Application Examples (cont’d) • Change Detection 2000 1975 Satellite images of Dead Sea, United Nations Environment Programme (UNEP) website

  14. Change Detection (cont’d) 2005 1990 Satellite images of Amona hilltop, Peace Now website

  15. Change Detection (cont’d) IKONOS images of Iran’s Bushehr nuclear plant, GlobalSecurity.org

  16. Change Detection (cont’d) IKONOS images of Iran’s Bushehr nuclear plant, GlobalSecurity.org

  17. Change Detection (cont’d) Satellite imagery of Sendai Airport before and after the 2001 earthquake

  18. What is the “Big Deal”? By matching control points, e.g., corners, high-curvature points. How do humans solve this? Zitova and Flusser, IVC 2003

  19. Automatic Image Registration • Books: • Medical Image Registration, J. Hajnal, D.J. Hawkes, and D. Hill (Eds.), CRC 2001 • Numerical Methods for Image Registration, J. Modersitzki, Oxford University Press 2004 • 2-D and 3-D Image Registration, A. Goshtasby, Wiley 2005 • Image Registration for Remote Sensing, J. LeMoigne, N.S. Netanyahu, and R.D. Eastman (Eds.), Cambridge University Press 2011. • Surveys: • A Survey of Image Registration Techniques, ACM Comp. Surveys, L.G. Brown, 1992 • A Survey of Medical Image Registration, Medical Image Analysis, J.B.A. Maintz and M.A. Viergever, 1998 • Image Registration Methods: A Survey, Image and Vision Computing, B. Zitova and J. Flusser, 2003

  20. Automatic Image Registration Components 0. Preprocessing • Image enhancement, cloud detection, region of interest masking 1. Feature extraction (control points) • Corners, edges, wavelet coefficients, segments, regions, contours 2. Feature matching • Spatial transformation (a priori knowledge) • Similarity metric (correlation, mutual information, Hausdorff distance) • Search strategy (global vs. local, multiresolution, optimization) 3. Resampling I2 Tp I1

  21. Example of Image Registration Steps Feature extraction Resampling Registered images after transformation Zitova and Flusser, IVC 2003 Feature matching

  22. Automatic Image Registrationfor Remote Sensing • Sensor webs, constellation, and exploration • Selected NASA Earth science missions • Domain-dependent characteristics

  23. Domain-Dependent Characteristics • Very large images (~ 6200 x 5700 of typical Landsat 7 scene) • Practically “flat”, 2D images • Rigid/similar transformations • A priori knowledge (e.g., small rotation and scale)

  24. Challenges in Processing of Remotely Sensed Data • Multisource data • Multi-temporal data • Various spatial resolutions • Various spectral resolutions • Subpixel accuracy • 1 pixel misregistration ≥ 50% error in NDVI classification • Computational efficiency • Fast procedures for very large data sets • Accuracy assessment • Synthetic data • “Ground Truth" (manual registration?) • Consistency ("circular" registrations) studies

  25. Fusion of Multi-temporal Images Improvement of NDVI classification accuracy due to fusion of multi-temporal SAR and Landsat TM over farmland in The Netherlands (source: The Remote Sensing Tutorial by N.M. Short, Sr.)

  26. Integration of Multiresolution Sensors Registration of Landsat ETM+ and IKONOS images over coastal VA and agricultural Konsa site (source: LeMoigne et al., IGARSS 2003)

  27. Feature Extraction Gray levels BPF wavelet coefficients Binary feature map Top 10% of wavelet coefficients (due to Simoncelli) of Landsat image over Washington, D.C. (source: N.S. Netanyahu, J. LeMoigne, and J.G. Masek, IEEE-TGRS, 2004)

  28. Feature Extraction (cont’d) Image features (extracted from two overlapping scenes over D.C.) to be matched

  29. Feature Matching / Transformations • Given a reference image, I1(x, y), and a sensed image I2(x, y),find the mapping (Tp, g) which “best” transforms I1 intoI2, i.e., where Tp denotes spatial mapping and g denotes radiometric mapping. • Spatial transformations: Translation, rigid, affine, projective, perspective, polynomial • Radiometric transformations (resampling): Nearest neighbor, bilinear, cubic convolution, spline

  30. Transformations (cont’d) Objective: Find parameters of a transformation Tp (consisting of a translation, a rotation, and an isometric scale) that maximize similarity measure.

  31. Similarity Measures (cont’d) • L2 norm: Minimize sum of squared errors over overlapping subimage • Normalized cross correlation (NCC): Maximize normalized correlation between the images

  32. Similarity Measures (cont’d) • Mutual information (MI): Maximize the degree of dependence between the images or using histograms, maximize

  33. Similarity Measures (cont’d), An Example MI vs. L2-norm and NCC applied to Landsat 5 images (source: Chen, Varshney, and Arora, IEEE-TGRS, 2003)

  34. Similarity Measures (cont’d), an MI Example Source: Cole-Rhodes et al., IEEE-TIP, 2003

  35. Similarity Measures (cont’d) • (Partial) Hausdorff distance (PHD): where

  36. Similarity Measures (cont’d), PHD Example PHD-based matching of Landsat images over D.C.(source: Netanyahu, LeMoigne, and Masek, IEEE-TGRS, 2004)

  37. Feature Matching / Search Strategy • Exhaustive search • Fast Fourier transform (FFT) • Optimization (e.g., gradient descent; Thévenaz, Ruttimann, and Unser (TRU), 1998; Spall, 1992) • Robust feature matching (e.g., efficient subdivision and pruning of transformation space; Huttenlocher et al., 1993, Mount et al., 1999)

  38. Search Strategy: Geometric Branch and Bound • Space of affine transformations: 6-dim space • Subdivide: Quadtree or kd-tree. Each cell T represents a set of transformations; T is active if it may contain ; o/w, it is killed • Uncertainty regions (UR’s): Rectangular approximation to the possible images for all • Bounds: Compute upper bound (on optimum similarity) by sampling a transformation and lower bound by computing nearest neighbors to each UR • Prune: If lower bound exceeds best upper bound, then kill the cell; o/w, split it

  39. Branch and Bound (cont’d) Illustration of uncertainty regions

  40. Algorithmic Outline of B & B (Sketch) • For all active cells do • Compute upper bound on similarity metric • For each active compute a lower bound on the similarity measure (can be done using a variant of efficient NN-searching) • Prune search space, i.e., discard if lower bound exceeds best (upper bound) seen thus far • O/w, split (e.g., along “longest dimension”) and enqueue in queue of active cells • If termination condition met, e.g., empty or , then report transformation and exit; o/w, goto 1)

  41. Computational Efficiency • Extraction of corresponding regions of interest(ROI) • Hierarchical, pyramid-like approach • Efficient search strategy

  42. Image Registration Subsystem Based on a Chip Database input scene UTM of 4 scene corners known from systematic correction (1) Find chips that correspond to the incoming scene (2) For each chip, extract window from scene, using UTM of: - 4 approx scene corners - 4 correct chip corners (3) Register each (chip-window) pair and record pairs of registered chip corners (4) Compute global registration from multiple local ones (5) Compute correct UTM of 4 scene corners of input scene Landmark Chip Database correct UTM of 4 chip corners

  43. Computational Efficiency (cont’d),An Example of a Pyramid-Like Approach 0 32 x 32 1 64 x 64 2 128 x 128 3 256 x 256

  44. IR Example Using Partial Hausdorff Distance 64 x 64 128 x 128 256 x 256

  45. IR Example Using PHD (cont’d) Source: Netanyahu, LeMoigne, and Masek, IEEE-TGRS, 2004

  46. Computational Efficiency (cont’d), ROI Extraction Input Scene UTM of 4 scene corners known from systematic correction Extract reference chips and corresponding input windows using mathematical morphology Register each (chip-window) pair and record pairs of registered chip corners (refinement step) Compute global registration from multiple local ones Compute correct UTM of 4 scene corners of input scene Reference Scene • Advantages: • Eliminates need for chip database • Cloud detection can easily be included in process • Process any size images • Initial registration closer to optimal registration => • reduces computation time and increases accuracy. Source: Plaza, LeMoigne, and Netanyahu, MultiTemp, 2005

  47. Step 1: Chip-Window Extraction UsingMathematical Morphology • Mathematical Morphology (MM) Concept: • Nonlinear spatial-based technique that provides a framework. • Relies on a partial ordering relation between image pixels. • In greyscale imagery, such relation is given by the digital value of image pixels Original image Grayscale MM Basic Operations: K K Structuring element (4-pixel radius Disk SE) Dilation Erosion

  48. Step 1 (Cont.) Binary Erosion Structuring element Structuring element Structuring element

  49. Step 1 (Cont.) Binary Dilation Structuring element Structuring element Structuring element

  50. Step 1 (Cont.) Grayscale Morphology: Combined Operations e.g., Erosion + Dilation = Opening K

More Related