1 / 60

Mixed-Resolution Patch-Matching

Mixed-Resolution Patch-Matching. Harshit Sureka Guide: Dr. P. J. Narayanan. Patch-Matching. 01/08/13. 2. Scope of Patch-Matching Algorithms. Image credits: Image Montage (Simakov et al. 2008). Object Detection,

landon
Download Presentation

Mixed-Resolution Patch-Matching

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Mixed-Resolution Patch-Matching Harshit Sureka Guide: Dr. P. J. Narayanan

  2. Patch-Matching 01/08/13 2

  3. Scope of Patch-Matching Algorithms Image credits: Image Montage (Simakov et al. 2008). Object Detection, Forgery Detection and Image Reshuffling (Barnes et al. 2010). Texture synthesis and Image completion (Xiao et al. 2011) 01/08/13 3

  4. Brute force v/s state-of-the-art Image Credit: Kumar et al. (ECCV 2008) • Find the matching patches to the patch in red • Brute-force: 2125 msec • State-of-the-art approaches: 1.5msec (1400x faster) 01/08/13 4

  5. The Patch-Matching Problem Image Credit: Barnes et al. (SIGGRAPH 2009) 01/08/13 5

  6. Types of Patch-Matching • Sparse • Finding matches for a few key-points • Dense • Matches for all patches in the source Image • Local • Search for matches in a specific region in the target image • Global • Search everywhere for matches in the target image • Local is computationally less expensive • But, will it suffice? Sparse Dense Global Local 01/08/13 6

  7. Where do similar patches lie? • Our focus dense and global • Since a lot of patches lie at large distances 01/08/13 7

  8. Till now.. • What is a patch? • The formal patch-matching problem • Motivation for making patch-matching algorithms faster • Types of patch-matching Next.. • Parameters of Patch-matching algorithms • Related Work 01/08/13 8

  9. Parameters of Patch-Matching Algorithms 01/08/13 9

  10. Previous Efforts • To use general-purpose Approximate Nearest Neighbor (ANN) algorithms • KD trees • PCA trees • etc. • They provide good solutions to general ANN problems • Do not take advantage of the structure and properties specific to images 01/08/13 10

  11. Previous Efforts: PatchMatch • Barnes et al., SIGGRAPH 2009 • Randomized cooperative Hill-Climbing Search • Based on the property of coherence in image patches • 3 steps of the algorithm: • Up-to 2x faster than all previous approaches • Enables several interactive applications • Not as accurate as one would hope • Coherency Assumption: Not true in highly textured images 01/08/13 11 Image credit: Barnes et al. 2009

  12. Previous state-of-the-art: Coherency Sensitive Hashing (CSH) • Korman and Avidan, ICCV 2011 • Based on PatchMatch • Replaces the Random step of PatchMatch with a Hash Table • Combines cues of location (of coherence) and appearance in a novel manner Coherence Appearance Type 3 Type 2 Type 1 01/08/13 12

  13. Coherency Sensitive Hashing (CSH) • Targeted propagation of information • Lower RMS error values in early iterations compared to PatchMatch • Faster and more accurate than Patchmatch • Less dependent on the coherency assumption • Lowest reconstruction errors • Current state-of-the-art 01/08/13 13

  14. Till now.. • Introduction to Patch-Matching • Previous techniques • General purpose ANN • PatchMatch • Coherency Sensitive Hashing (CSH) Next.. • Image Pyramids • Pyramids for Patch-Matching 01/08/13 14

  15. Pyramids • Image represented at multiple scales 512x512 32x32 Level 4 Level 0 Level 3 Level 2 Level 1 01/08/13 15

  16. Pyramids • Pyramid coarse-to-fine framework is well established (Burt et al. 1984) • Help a variety of problems • Image compression • Image alignment • Optical flow • Object tracking • Matching: do away with fine details, look at only high level information Can they also be used for Patch-Matching? Why not! 01/08/13 16

  17. Pyramid Patch-Matching • Novel patch-matching algorithm • Pyramid representation of original images • Coarsest level image size determined by a threshold ‘Pthold’ • Exhaustive global search at small images of coarsest level • Search localization at finer levels by up-sampling the matches found at coarser levels 01/08/13 17

  18. Pyramid Patch-Matching (PPM) • Coarsest level • Finer levels 01/08/13 18 a* is the patch in IA(p+1) corresponding to a

  19. PPM: Real Image example 01/08/13 19

  20. Pyramid Patch Matching • Increase K • Best match at coarsest resolution may not be the best match at original resolution • Store more number of matches • Wider search range at finer levels • Up to 2% better accuracy • Reduce Patch-Size • Small size patches matched at coarser resolutions • Early Termination • Of distance calculation if it exceeds the distance of the farthest patch currently in the matched list of patches • Up to 1.7x speed-up 01/08/13 20

  21. Parallel Implementation • PPM processes each patch independently • Pyramid construction • Standard parallel implementation • Coarsest level • Each thread assigned a block of source patches • Searches for best matches in the whole target image • Finer levels • Up-sampling of matches; candidate-list generation • Each thread searches the candidate space of the patches in its block • Near-linear time speed up w.r.t. number of cores 01/08/13 21

  22. GPU Implementation • Pyramid construction on the GPU • Standard implementation • Each patch is matched independently • Distance calculation of each query patch with its candidate matches is also independent 01/08/13 22

  23. GPU Implementation • Algorithm • Exhaustive search at the coarsest image resolution • Localized search at finer resolution levels • For each patch • threads equal to the number of candidate patches are invoked • Each thread handles one vector-distance calculation • Up-sampling of matches; candidate-list generation also in parallel • Additional 70x compared to implementation run on 4-core (~250x compared to single-core) 01/08/13 23

  24. GPU Timings • Values for CSH Dataset Images (shown in next slide) • Brute force for Lena (256x256 image) • Most gain in speed achieved for brute-force global exhaustive search 01/08/13 24

  25. CSH Dataset for Experiments • 133 image-pairs from the dataset provided by Korman and Avidan, ICCV 2011 (CSH) • Both images of the same scene of a movie • between 1 and 30 frames apart in the video 01/08/13 25

  26. Parameters Error: RMS distance between RGB values of patches, averaged over all matches of a patch and over all patches in the image Platform: Intel i7 920 processor, 2.67 GHz Varying KNN and Patch-size Varying Search-range Varying Pthold Values over all 133 pairs of images in the CSH dataset (shown in next slide) 01/08/13 26

  27. Experiment I • Proximity to ground-truth • Brute-force search for ground-truth matches • PPM captures up to 4% more ground truth matches • Better performance for more number of matches required 01/08/13 27

  28. Experiment II • Error (RMS error between RGB values of patches and their matches) • Parameters set to achieve similar run-times • PPM achieves consistently lower error values than CSH • Better performance than state-of-the-art • Much closer to ground truth (Calculated by Brute-force on GPU) 01/08/13 28

  29. Till now.. • Patch-Matching and previous techniques • Pyramid Patch-Matching • Improvements: Increase K, Reduce patch-size, early-termination • Experiments: State-of-the-art performance Next.. • Mixed-Resolution Vectors (MR-vectors) • Motivation for mixing resolutions • Mixed-Resolution Patch-Matching (MRPM) 01/08/13 29

  30. Mixed-Resolution: Motivation • Images at coarser resolutions contain approximate information • Small objects may get smoothed out due to downsampling • A match missed at the coarsest resolution will never be recovered later in the algorithm • Solution: More information already exists at finer resolutions. Use it! 01/08/13 30

  31. Mixed-Resolution Vectors • Idea • Concatenate corresponding patch-vectors of finer resolutions and current resolution • Including Information • Currently we pick a same-sized patch around the center of the corresponding up-sampled patch • Available prior knowledge about the image or patches can be used to pick information more wisely! 01/08/13 31

  32. MR-Vectors • Primary contribution • Can include information from multiple levels 01/08/13 32

  33. Mixed-Resolution Patch-Matching • In the framework of PPM, we replace traditional patch-vectors with MR-vectors • Rest of the algorithm remains same • We call this the “Mixed-Resolution Patch-Matching” Algorithm (MRPM). 01/08/13 33

  34. Comparison with increasing search range • Accuracy increases with cost of time when using a wider search range around up-sampled matches • Similar behavior is observed while mixing resolutions Mixed-Resolution achieves lower error values with less cost of time 01/08/13 34

  35. Parallel Implementation • Fast and efficient parallel implementations of MRPM follow from parallel implementations of PPM • Multicore: Near-linear time speed-up • GPU: All levels already on GPU memory 01/08/13 35

  36. PPM v/s MRPM (l = 2) • Lower error values with marginal cost of time • Less effect of time cost on GPU performance • Note: error not averaged over patch-size 01/08/13 36

  37. Experiment I • Proximity to ground truth • Up to 8% more of the ground truth matches are captured by MRPM than CSH • 4% more than PPM 01/08/13 37

  38. Experiment II • Error • Parameters set to achieve similar run time • MRPM achieves near optimal patch-matching 01/08/13 38

  39. Till now.. • Patch-Matching and previous techniques • Pyramid Patch-Matching • Mixed-Resolution Vectors • Mixed-Resolution Patch-Matching Next.. • Applications • Conclusion 01/08/13 39

  40. Applications • A variety of applications have a patch-matching engine at their core • MRPM can act as that engine and help improve quality and speed of several of these applications • We discuss 3 of these • Image de-noising • Image summarization (Retargeting) • Auto-crop 01/08/13 40

  41. Image denoising • Performs a weighted averaging over similar patches to de-noise the image (Non-local means, Buades et al., 2005) • Currently local due to computational constraints • Also improves accuracy in some cases • MRPM can be used to extend it to global 01/08/13 41

  42. Dataset of 36 images Experiments • Dataset of 36 images collected for evaluating denoising performance Combining matches of local (non-local means) and global (MRPM) 01/08/13 42

  43. Self-similar Image • Evaluation: Method-noise • Less structure removed from Image PSNR: 33.35 PSNR: 34.03 01/08/13 43

  44. Image Retargeting (Summarization) • Simakov et al. 2008, Bidirectional Similarity 01/08/13 44

  45. Results • MRPM achieves better visual qualityof results 01/08/13 45

  46. Auto-crop • Similar to Bi-directional Similarity • Coherence term muted: since all cropping windows would anyway be coherent with the image • Cropping based on completeness term similarity 01/08/13 46

  47. More Results 01/08/13 47

  48. Limitation: Image Reconstruction • Given an Image B and a dense patch-matching from A to B. Reconstruct image A • Each pixel is replaces with the average of the corresponding pixels that it is mapped to by all patches that contain it 01/08/13 48

  49. Summary • Presented a Pyramid based approach to the problem of patch-matching • 3 Improvements to improve accuracy and efficiency • Achieved state-of-the-art matching accuracy • Introduced the concept of Mixed-Resolution Vectors • Proposed Mixed-Resolution Patch-Matching outperforming state-of-the-art by up-to 8% more ground-truth matches captured 01/08/13 49

  50. Future work • Include coherency cues in the framework of the algorithm • Up-sampling maintains some coherency • Matching across all rotations and scale • Matching in videos • Explore the scope of MR-vectors • We believe it is a generic concept and the idea can be used in several other problems that use the multi-resolution technique 01/08/13 50

More Related