Image Registration of Remotely Sensed Data

1. פברואר 1968, בסיום מסע למצדה פברואר 1968, בסיום מעלה האיסיים פברואר 1969, עובדת (מסע גדנ"ע) יוני 1969, אודיטוריום פיסיקה, בית-בירם

2. Image Registration of Remotely Sensed Data Nathan S. Netanyahu Dept. of Computer Science, Bar-Ilan University and Center for Automation Research, University of Maryland Collaborators: Jacqueline Le Moigne NASA / Goddard Space Flight Center David M. Mount University of Maryland Arlene A. Cole-Rhodes, Kisha L. Johnson Morgan State University, Maryland Roger D. EastmanLoyola College of Maryland Ardeshir Goshtasby Wright State University, Ohio Jeffrey G. Masek, Jeffrey Morisette NASA / Goddard Space Flight Center Antonio Plaza University of Extremadura, Spain Harold Stone NEC Research Institute (Ret.) Ilya Zavorin CACI International, Maryland Shirley Barda, Boris Sherman Applied Materials, Inc., Israel Yair Kapach Bar-Ilan University

3. What is Image Registration / Alignment / Matching? The above image is rotated and shifted with respect to the left image.

4. Motivation • A crucial, fundamental step in image analysis tasks, where final information is obtained by the combination / integration of multiple data sources.

5. Motivation / Applications • Computer Vision (target localization, quality control, stereo matching) • Medical Imaging (combining CT and MRI data, tumor growth monitoring, treatment verification) • Remote Sensing (classification, environmental monitoring, change detection, image mosaicing, weather forecasting, integration into GIS)

6. Application Examples • Global Wafer Alignment

7. Wafer Alignment (cont’d) Courtesy: Shirley Barda and Boris Sherman, Applied Materials, Inc.

8. Application Examples (cont’d) • Integration of medical images Registration of MR and PET images of the same person (courtesy: A. Goshtasby)

9. Application Examples (cont’d) • Change Detection 2000 1975 Satellite images of Dead Sea, United Nations Environment Programme (UNEP) website

10. Change Detection (cont’d) 2005 1990 Satellite images of Amona hilltop, Peace Now website

11. Change Detection (cont’d) IKONOS images of Iran’s Bushehr nuclear plant, GlobalSecurity.org

12. What is the “Big Deal”? By matching control points, e.g., corners, high-curvature points. How do humans solve this? Zitova and Flusser, IVC 2003

13. Automatic Image Registration • Books: • Medical Image Registration, J. Hajnal, D.J. Hawkes, and D. Hill (Eds.), CRC 2001 • Numerical Methods for Image Registration, J. Modersitzki, Oxford University Press 2004 • 2-D and 3-D Image Registration, A. Goshtasby, Wiley 2005 • Image Registration for Remote Sensing, J. LeMoigne, N.S. Netanyahu, and R.D. Eastman (Eds.), Cambridge University Press, in preparation. • Surveys: • A Survey of Image Registration Techniques, ACM Comp. Surveys, L.G. Brown, 1992 • A Survey of Medical Image Registration, Medical Image Analysis, J.B.A. Maintz and M.A. Viergever, 1998 • Image Registration Methods: A Survey, Image and Vision Computing, B. Zitova and J. Flusser, 2003 • Sample Papers: • Image Sequence Enhancement Using Sub-pixel Displacements, CVPR, D. Keren, S. Peleg, and R. Brada, 1988 • Improving Resolution by Image Registration, CVGIP, M. Irani and S. Peleg, 1991 • Computing Correspondence Based on Regions and Invariants without Feature Extraction and Segmentation, CVPR, C. Lee, D. Cooper, and D. Keren, 1993 • Robust Multi-Image Sensor Image Alignment, ICCV, M. Irani and P. Anandan, 1998 • Fast Block Motion Estimation Using Gray Code Kernels, Israel CV Workshop, Y. Moshe and H. Hel-Or, 2006 • Image Matching Using Photometric Information, Israel CV Workshop, M. Kolomenkin and I. Shimshoni, 2006

14. Automatic Image Registration Components 0. Preprocessing • Image enhancement, cloud detection, region of interest masking 1. Feature extraction (control points) • Corners, edges, wavelet coefficients, segments, regions, contours 2. Feature matching • Spatial transformation (a priori knowledge) • Similarity metric (correlation, mutual information, Hausdorff distance) • Search strategy (global vs. local, multiresolution, optimization) 3. Resampling I2 Tp I1

15. Example of Image Registration Steps Feature extraction Resampling Registered images after transformation Zitova and Flusser, IVC 2003 Feature matching

16. Automatic Image Registrationfor Remote Sensing • Sensor webs, constellation, and exploration • Selected NASA Earth science missions • Domain-dependent characteristics

17. Sensor Webs, Constellation, and Exploration Planning and Scheduling Automatic Multiple Source Integration Satellite/Orbiter, and In-Situ Data Intelligent Navigation and Decision Making

18. Selected NASA Earth Science Missions

19. MODIS Satellite System From the NASA MODIS website

20. MODIS Satellite Specifications

21. Landsat 7 Satellite System New Orleans, before and after Katrina 2005 (from the USGS Landsat website)

22. Landsat 7 Satellite Specifications

23. IKONOS Satellite System

24. IKONOS Satellite Specifications

25. Domain-Dependent Characteristics • Very large images (~ 6200 x 5700 of typical Landsat 7 scene) • Practically “flat”, 2D images • Rigid/similar transformations • A priori knowledge (e.g., small rotation and scale)

26. Challenges in Processing of Remotely Sensed Data • Multisource data • Multi-temporal data • Various spatial resolutions • Various spectral resolutions • Subpixel accuracy • 1 pixel misregistration ≥ 50% error in NDVI classification • Computational efficiency • Fast procedures for very large data sets • Accuracy assessment • Synthetic data • “Ground Truth" (manual registration?) • Consistency ("circular" registrations) studies

27. Fusion of Multi-temporal Images Improvement of NDVI classification accuracy due to fusion of multi-temporal SAR and Landsat TM over farmland in The Netherlands (source: The Remote Sensing Tutorial by N.M. Short, Sr.)

28. Integration of Multiresolution Sensors Registration of Landsat ETM+ and IKONOS images over coastal VA and agricultural Konsa site (source: LeMoigne et al., IGARSS 2003)

29. Feature Extraction Gray levels BPF wavelet coefficients Binary feature map Top 10% of wavelet coefficients (due to Simoncelli) of Landsat image over Washington, D.C. (source: N.S. Netanyahu, J. LeMoigne, and J.G. Masek, IEEE-TGRS, 2004)

30. Feature Extraction (cont’d) Image features (extracted from two overlapping scenes over D.C.) to be matched

31. Feature Matching / Transformations • Given a reference image, I1(x, y), and a sensed image I2(x, y),find the mapping (Tp, g) which “best” transforms I1 intoI2, i.e., where Tp denotes spatial mapping and g denotes radiometric mapping. • Spatial transformations: Translation, rigid, affine, projective, perspective, polynomial • Radiometric transformations (resampling): Nearest neighbor, bilinear, cubic convolution, spline

32. Transformations (cont’d) Objective: Find parameters of a transformation Tp (consisting of a translation, a rotation, and an isometric scale) that maximize similarity measure.

33. Similarity Measures (cont’d) • L2 norm: Minimize sum of squared errors over overlapping subimage • Normalized cross correlation (NCC): Maximize normalized correlation between the images

34. Similarity Measures (cont’d) • Mutual information (MI): Maximize the degree of dependence between the images or using histograms, maximize

35. Similarity Measures (cont’d), An Example MI vs. L2-norm and NCC applied to Landsat 5 images (source: Chen, Varshney, and Arora, IEEE-TGRS, 2003)

36. Similarity Measures (cont’d), an MI Example Source: Cole-Rhodes et al., IEEE-TIP, 2003

37. Similarity Measures (cont’d) • (Partial) Hausdorff distance (PHD): where

38. Similarity Measures (cont’d), PHD Example PHD-based matching of Landsat images over D.C.(source: Netanyahu, LeMoigne, and Masek, IEEE-TGRS, 2004)

39. Feature Matching / Search Strategy • Exhaustive search • Fast Fourier transform (FFT) • Optimization (e.g., gradient descent; Thévenaz, Ruttimann, and Unser (TRU), 1998; Spall, 1992) • Robust feature matching (e.g., efficient subdivision and pruning of transformation space)

40. Computational Efficiency • Extraction of corresponding regions of interest(ROI) • Hierarchical, pyramid-like approach • Efficient search strategy

41. Computational Efficiency (cont’d), ROI Extraction Input Scene UTM of 4 scene corners known from systematic correction Extract reference chips and corresponding input windows using mathematical morphology Register each (chip-window) pair and record pairs of registered chip corners (refinement step) Compute global registration from multiple local ones Compute correct UTM of 4 scene corners of input scene Reference Scene • Advantages: • Eliminates need for chip database • Cloud detection can easily be included in process • Process any size images • Initial registration closer to optimal registration => • reduces computation time and increases accuracy. Source: Plaza, LeMoigne, and Netanyahu, MultiTemp, 2005

42. Computational Efficiency (cont’d),An Example of a Pyramid-Like Approach 0 32 x 32 1 64 x 64 2 128 x 128 3 256 x 256

43. IR Example Using Partial Hausdorff Distance 64 x 64 128 x 128 256 x 256

44. IR Example Using PHD (cont’d) Source: Netanyahu, LeMoigne, and Masek, IEEE-TGRS, 2004

45. IR Components (Revisited) Correlation Mutual Information Spall’s Optimization Hausdorff Distance Gradient Descent L2-Norm Robust Feature Matching Thevenaz, Ruttimann, Unser Optimization Fast Fourier Transform Gray Levels Wavelets or Wavelet-Like Edges Features Similarity Measure Strategy

46. IR Components (Revisited) Correlation Gray Levels Simoncelli BPF Spline or Simoncelli LPF Spall’s Optimization Hausdorff Distance Gradient Descent L2-Norm L2-Norm MI MI Thevenaz, Ruttimann, Unser Optimization Features Similarity Measure Strategy Thevenaz, Ruttimann, Unser Optimization Robust Feature Matching Gradient Descent Spall’s Optimization FFT

47. Goals of a Modular Image Registration Framework • Testing framework to: • Assess various combinations of components • Assess a new registration component • Web-based registration tool would allow user to “schedule” combination of components, as a function of: • Application • Available computational resources • Required registration accuracy • Prototype of web-based registration toolbox: • Several modules based on wavelet decomposition • Java implementation; JNI-wrapped functions

48. Web-Based Image Registration ToolboxTARA (“Toolbox for Automated Registration & Analysis”)

49. Web-Based Image Registration ToolboxTARA (“Toolbox for Automated Registration & Analysis”)

50. Current and Future Work • Conclude component evaluation • Sensitivity to noise, radiometric transformations, initial conditions, and computational requirements • Integration of digital elevation map (DEM) information • Build operational registration framework/toolbox • Web-based • Applications: • EOS validation core sites • Other EOS satellites (e.g., Hyperion vs. ALI registration) and beyond • Image fusion, change detection