Feature Point Matching

1. Feature Point Matching Detection, Extraction & Matching Part 2 – Matching via template matching (Previous slides: Feature point detection Moravec detection Harris detection Scale space detection Feature point Extraction ).

2. Template matching – X-correlation • Template matching by Cross-correlation • You have already learnt template matching with sum of square differences – SSD: There you were trying to minimize a dissimilarity score. • Here you will learn about template matching by normalized cross-correlation: minimizing a similarity score.

3. Template matching – X-correlation • Cross-correlation: • In statistical parlance, the cross-correlation between two random variables is a measure of how closely the two variables vary together. • {A,B} = {1,2},{8,16},{3,6},{-5,-10} • The magnitude of these variables vary together closely – they have a high positive cross-correlation. • {C,D} = {-1,2},{-8,16},{-3,6},{5,-10} • The magnitude of these vary oppositely closely – they have a high negative cross-correlation. • {E,F} = {-1,2},{8,0},{-3,6},{-5,-10} • These don’t vary closely – they have a low cross-correlation.

4. Template matching – X-correlation • Cross-correlation: • Normalized cross-correlation (-1<0<-1): • {A,B} = {1,2},{8,16},{3,6},{-5,-10} • X(A,B) = 43 C(A,B) = 1.0 • {C,D} = {-1,2},{-8,16},{-3,6},{5,-10} • X(C,D) = -43 C(A,B) = -1.0 • {E,F} = {-1,2},{8,0},{-3,6},{-5,-10} • X(E,F) = 7.4 C(A,B) = 0.25

5. Template matching – X-correlation • Lingo alert: • Cross-correlation is usually called Covariance • Normalized cross-correlation is usually called Correlation:

6. Template matching – X-correlation • Template matching with Cross-correlation • Take an image (I) • Take a target area to be matched (T) • For each point in the image (I) • Find the normalized cross-correlation: • Where: and

7. Template matching – X-correlation • How to calculate the cross-correlation function • Pick a pixel to be matched (u,v) • For each (x,y) in the template • Multiply its value (minus the template average) with the equivalent value in the image (minus the image patch average). Add to running total. • Then divide the total by the square root of the product of the variances of the image and template. C = 0.86 here

8. Template matching – X-correlation • Template matching with Cross-correlation The cross-correlation function at the point is a measure of the degree of match between template and that point in the image. • Q: It looks similar to convolution. How does this relate to convolution? • A: It is indeed very similar. What you learnt as convolution is actually almost the cross-correlation that you learnt here (you just don’t divide by N). But this that is yet another definition of cross-correlation, from signal detection theory… • …so how would you do cross-correlation in matlab? Matlab: help conv2 help corr2

9. Template matching – X-correlation • Template matching with Cross-correlation • But... • How well would this template match the image? • Answer: Undefined! Divide by zero! • It’s the normalization factor’s fault! (Doesn’t affect non-normalized cross-correlation).

10. Template matching • Problems with naïve template matching • And... • How well would this template match the image? • Answer: -0.16 • Not rotationally invariant.

11. Template matching • Problems with naïve template matching • And... • Problem: Rotational invariance. • Possible solution: Match every possible rotation.

12. Template matching • Problems with naïve template matching Direction of slowest change • And... • Problem: Rotational invariance. • Possible solution: Align orientations. Direction of fastest change • E.g. by gradient direction

13. Template matching • Problems with naïve template matching • Also… • How well would this template match the image? • Answer: 0.6 • Needs to handle scale changes better too. • Could check multiple scales…

14. Template matching • Naïve cross-correlation is not the only way to do template matching. • More sophisticated methods have to be able to handle • Scale changes • Rotation • Affine transformation • Lighting changes • Etc • More efficient approaches are needed. E.g. • Using pyramidal (down-sampled) images • Frequency domain filtering • Iterative refinement

15. Alternatives to template matching • Template matching can be refined to be made very useful. • But other approaches exist too. • These approaches generalize template matching by extracting a description (or descriptor) of the target feature point so that the descriptor can be matched. For example: • Training a classifier from local image information • Extracting local image statistics • E.g. histogram of spatial distribution of local intensity gradient (as per SIFT) • More? • This improvement can be made at the extraction phase.

16. Summary • Criteria for a good feature point detector are: • Repeatably detectable under transforms: • Scale • Rotation • Affine transformation • Illumination • Etc • Computationally efficient • Accurate at localisation • High detection rate • Robust to noise

17. Summary • Criteria for a good feature point matcher are: • Consistent • Invariant to: • Scale • Rotation • Illumination • Affine transformation • Etc • Robust to noise • Accurate