1 / 49

Coherency Sensitive Hashing

Efficient Search Methods. Coherency Sensitive Hashing. &. PatchMatch. Simon Korman and Shai Avidan. C. Barnes , E. Shechtman , A. Finkelstein, and D. B. Goldman. Presented by: Itamar Friedman. Nearest Neighbor Fields ( NNF ). For each patch – find nearest patch in other image

adora
Download Presentation

Coherency Sensitive Hashing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Efficient Search Methods Coherency Sensitive Hashing & PatchMatch Simon KormanandShaiAvidan C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman. Presented by: Itamar Friedman

  2. Nearest Neighbor Fields (NNF) • For each patch – find nearest patch in other image • Dense : Number of patches ≈ image size • Exact NN is infeasible • Error is mean(patch-distances) A B NNF

  3. Why NNF? Non-Local Means denoising Non Local Means. Baudes, Coll, Morel. CVPR 2005

  4. Why NNF? Image Completion Object Removal by Examplar Based Inpainting. Criminisi et al’, CVPR 2003 Space-Time Completion. Wexler et al’, CVPR 2004

  5. Appearance Space k-by-kpatch = k2 dimensional vector Ordinary Nearest Neighbor search • Two popular methods are: • KD-trees [Arya et al, 1998] • Locality Sensitive Hashing (LSH) [Indyk et al, 1998] • These methods • Can be very accurate • Are relatively slow

  6. Image Space • Based on coherency

  7. Each iteration, for each pixel: Compare two kinds of candidates (patches): 1. Randomly located patches (with probability that drops exponentially with distance from current match) Image Space - the PatchMatch algorithm[Barnes et al, SIGRAPH 2009] current match candidates

  8. Each iteration, for each pixel: Compare two kinds of candidates (patches): 2. Propagated neighbor’s matches Image Space - the PatchMatch algorithm[Barnes et al, SIGRAPH 2009] left neighbor current match of left neighbor candidate

  9. Each iteration, for each pixel: Compare two kinds of candidates (patches): 2. Propagated neighbor’s matches Image Space - the PatchMatch algorithm[Barnes et al, SIGRAPH 2009] current match of top neighbor top neighbor left neighbor current match of left neighbor candidates

  10. An iterative randomized algorithm Simple and fast (runtime is in the seconds) Based on local propagation and mostly-local search Heavily relies on the fact that images are generally coherent Image Space - the PatchMatch algorithm[Barnes et al, SIGRAPH 2009]

  11. Coherency Sensitive Hashing (CSH) Combine image space and appearance space • Find good matches • Propagate them across patches that either: • Are neighbors (image space) • Share hash values (appearance space)

  12. Walsh Hadamard (WH) Basis 3 4 5 6 7 8 11 10 6 7 8 9 4 5 6 7 8 9 CSH - The LSH Framework • Project each point v on a random segmented line Hash function (code) Bin number LSH table Datar, Immorlica, Indyk, Mirrokni, 2004 The Gray-Code Filter Kernels, Hel-Or, Hel-Or, PAMI 2007

  13. CSH indexing • Hash all patches of both images (with hash function g) Hash Table Image ‘A’ Image ‘B’ g g

  14. CSH indexing • Hash all patches of both images (with hash function g) • Keep k (=2) representatives of each image in each entry Hash Table k k Image ‘A’ Image ‘B’ g g-1 g-1 g

  15. CSH indexing - candidates Candidates = potential matching patches Image ‘A’ Image ‘B’

  16. CSH indexing – candidate types Type I (appearance only) = Standard LSH Image ‘A’ Image ‘B’

  17. CSH indexing – candidate types Type II (appearance and coherence) Image ‘A’ Image ‘B’ PM candidate

  18. CSH indexing – candidate types Type III (appearance only) Image ‘A’ Image ‘B’

  19. CSH indexing – candidate types Type 1 Type 2 Type 3 (> PM) (LSH)

  20. CSH Algorithm – recap Preprocessing: • Project A’s and B’s patches on WH vectors Iterate (typically 1-3 times): • Hash patches into a table • For each patch in A • Create candidate patches in B • Pick the best one *further details in the paper A B

  21. CSH Algorithm – recap Preprocessing: • Project A’s and B’s patches on WH vectors Iterate (typically 1-3 times): • Hash patches into a table • For each patch in A • Create candidate patches in B • Pick the best one *further details in the paper A B

  22. CSH Algorithm – Hashing 1 6 5 2 4 3 3 4 2 1

  23. CSH Algorithm – Hashing kernel (DC) Y channel – 32 bins kernel (DC) Cb channel – 4 bins kernel Y channel - 8 bins kernel Y channel - 2 bins

  24. CSH Algorithm – Hashing • The reasoning being: • number of bins - relative to the dispersion of points along the projection line • Less bins the higher the kernel frequency • Less bins for color channels compared to Y • spacing of bins – attempting to capture a constant number of points in each bin • Use fixed percentiles • over the distribution

  25. CSH Algorithm – recap Preprocessing: • Project A’s and B’s patches on WH vectors Iterate (typically 1-3 times): • Hash patches into a table • For each patch in A • Create candidate patches in B • Pick the best one *further details in the paper A B

  26. CSH Algorithm – comparing candidates

  27. Experiments - setup • Vid-Pairs data-set • 133 related pairs of ~2 MP images • Up to 30 frames apart in the video • Include both camera and object motion A B

  28. Experiments - accuracy/efficiency

  29. Experiments – Image Reconstruction • Reconstruct image A, given: B and A->B • Each pixel is an avg. of corresponding pixels through the mapping • Main ingredient in many patch based algorithms (retargetting, inpainting, denoising, super-resolution, etc.) Image ‘A’ Image ‘B’ ? PM NNF CSH GT

  30. Experiments – Image Reconstruction

  31. Experiments – Image Reconstruction

  32. Experiments – Image Reconstruction

  33. Experiments – Image Reconstruction

  34. Experiments – Image Reconstruction PM vs. GT CSH vs. GT

  35. Dependence on patch energy

  36. Experiments – Image Reconstruction Image ‘A’ Image ‘B’ ? NNF

  37. Experiments – Image Reconstruction • Reconstruct image A, • given only image B and an NNF mapping A->B

  38. Experiments – Image Reconstruction • Reconstruct image A, • given only image B and an NNF mapping A->B

  39. Experiments – Image Reconstruction • Reconstruct image A, • given only image B and an NNF mapping A->B

  40. Experiments – Image Reconstruction • Reconstruct image A, • given only image B and an NNF mapping A->B

  41. Experiments – Image Reconstruction PM vs. GT CSH vs. GT

  42. Experiments – Mapping Incoherency CSH PM

  43. Experiments – Mapping (In)coherency Image ‘A’ Image ‘B’ CSH PM ‘x’-coordinate of PM mapping ‘x’-coordinate of CSH mapping

  44. Reconstruction-Error vs. Incoherency Good (exact NN) bad

  45. Summary - CSH • A new algorithm for NNF • combines appearance space and image space accuracy speed • other pleasing properties • Reconstruction, Incoherency, Texture • Limitations • Small data (images, patches) • Closely related image pairs • Additional memory for hash tables • Currently doesn’t support scale, rotation, illumination

  46. Evaluation – CSH & PatchMatch • Well written. Self contained. • Novel. Technically convincing. • Well tested and evaluated. • Provide materials and code.

  47. Future Work • The CSH shows inferior results with flat patches. Is it because of: (???) • The WH representation/functions • Noise in the flat patches • Distribution of selected/matching patches • Overcome limitations • “Smart” selection from a big set of data (images, patches) • Support scale, rotation, illumination by using a different representation such as {SIFT,DAISY,…}

  48. Reference • PatchMatch: source code, data-set, more info, available at: • http://gfx.cs.princeton.edu/pubs/Barnes_2009_PAR/ • CSH: source code, data-set, more info, available at: • http://www.eng.tau.ac.il/~simonk/CSH • Huge gratitude to CSH team for their presentation published in their site.

  49. THANK YOU!

More Related