1 / 67

Hybrid Texture Synthesis

Hybrid Texture Synthesis. Andrew Nealen Marc Alexa Discrete Geometric Modeling Group (DGM) Technische Universität Darmstadt Eurographics Symposium on Rendering 2003. Outline. Introduction. n x m Input Texture. N x M Output Texture.

cole-lynch
Download Presentation

Hybrid Texture Synthesis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Hybrid Texture Synthesis Andrew NealenMarc AlexaDiscrete Geometric Modeling Group (DGM)Technische Universität Darmstadt Eurographics Symposium on Rendering 2003

  2. Outline

  3. Introduction nxm Input Texture NxM Output Texture • The goal: Synthesize an output texture which isperceptually similarto the input texture. Also ensure that the result contains sufficient variation. 2D Texture Synthesis

  4. Introduction • Existing (and Impressive) Technology • Pixel-Based: • Non-parametric Sampling [Efros and Leung 1999] • Tree-structured Vector Quantization [Wei and Levoy 2000] • Image Analogies [Hertzmann et al. 2001] • Patch-Based: • Patch-Based Sampling [Guo et al. 2001] • Image Quilting [Efros and Freeman 2001]

  5. Introduction Wei/Levoy Algorithm: Can occur when using structured Textures with rich histograms. This is due to the L2 norm. Possible Drawbacks ? • Loss of global structure • Loss of scale • Boundary mismatch • Blurring

  6. Introduction Image Quilting Algorithm: boundary mismatch artifacts are noticeable Possible Drawbacks ? • Loss of global structure • Loss of scale • Boundary mismatch • Blurring

  7. Introduction Patch-Based Sampling Algorithm: Can occur when texture features are not well-alignedacross patch boundaries Possible Drawbacks ? • Loss of global structure • Loss of scale • Boundary mismatch • Blurring

  8. Hybrid Texture Synthesis • Combine best aspects from other approacheswhile avoiding (or improving over) known pitfalls: • Patch-based algorithms • Are good at preserving global structure • Can introduce artifacts along patch boundaries • Pixel-based algorithms • Preserve local coherence (MRF model: local and stationary) • Possibly fail to preserve global structure • This is especially problematic for textures with rich histograms and many high frequency features, due to the smoothing nature of the distance metric (L2norm)

  9. Hybrid Texture Synthesis (HTS) • Hybrid, two-fold approach to texture synthesis • Adaptive patch sampling [Soler et al. 2002] • Start with a uniform quadrilateral grid of patches (the output) • Recursively split a patch if the best fit taken from the input texture is not good enough (a user-defined tradeoffmax) • Overlap re-synthesis(novel overlap repair strategy) • Mark invalid pixels in the overlap region • Validitymap is defined by the error surface E(x)=(xsrc – xdst)2 and a user-defined tolerance parametermax • Define an ordering for the invalid pixels and re-synthesize them using a per-pixel strategy

  10. HybridSynthesizeAlgorithm 1: HYBRIDSYNTHESIZE(T,P,ov,Δmax,δmax,R) : R 2: for all patches pi ∈P do 3: [Pi,Δi]← FINDBESTPATCH(T,Ri−1, pi,ov) 4: if (Δi < Δmax or ISSINGLEPIXEL(pi)) then 5: Si ← ERRORSURFACE(Pi,Ri−1) 6: Pi ← MARKINVALIDPIXELS(Pi,Si,δmax) 7: Mi ← BUILDTRAVERSALMAP(Pi) 8: Ri,composite ← COMPOSE(Pi,Ri−1) 9: Ri ← OVERLAPRESYNTHESIS(T,Ri,composite,Mi) 10: else 11: Ps ← SPLITPATCH(pi) 12: ovs ←max(3, ov/2) 13: Ri ← HYBRIDSYNTHESIZE(T,Ps,ovs,Δmax,δmax,Ri−1) 14: end if 15: end for 16: return R

  11. FindBestPatch(Algo.) 1: FINDBESTPATCH(T,Ri−1, pi,ov) : [Pi,Δi] 2: Ii, Ji ← BUILDIMAGEMASK(Ri−1, pi,ov) 3: Ei ← ERRORIMAGE(T, Ii, Ji) 4: Ei,trim ← TRIMINVALIDREGIONS(Ei, Ji, pi) 5: [Pi,Δi]← BESTPATCH(Ei,trim,T) 6: return [Pi,Δi]

  12. In Search of Good PatchesSynthesizing a single patch Synthesize Black Patch i

  13. In Search of Good PatchesExtracting the image mask • Grow patch by overlap (toroidally) Synthesize Black Patch i

  14. In Search of Good PatchesExtracting the image mask • Extract image mask (Ii) and binary support (Ji), and circularly shift to upper left corner Synthesize Black Patch i Image Mask (Ii) Binary Support (Ji)

  15. In Search of Good PatchesComputing the error image * Compute error Ei(x0) between Ii and T for each circular shift x0=(x,y) of the input texture T Input Texture (T) Image Mask (Ii) Binary Support (Ji)

  16. In Search of Good PatchesComputing the error image C={R,G,B} (both with RGB color values in [0,1]) WR,G,B = {0.299,0.587,0.114} X0: circular shift of T Input Texture (T) Error Image(Ei) (1)

  17. In Search of Good PatchesSelecting a Patch Input Texture (T) Error Image(E) Selected Patch (P) (2) • Given that Ji(x) Ii(x) = Ii(x),and the cross correlation between two images (functions) f ◇ g is defined as ( f ◇ g )(x0) = Σx f (x)g(x+x0) • The correlation f◇ g between two functions can be computed in O(N logN) • in fourier space as f ◇g = F-1(F( f )∗F(g)) • (N :number of pixels in the input texture) • In implementation it’s can be pre-compute the fourier transform for Tand need only recompute the fourier transforms of Iiand Jifor each new patch pi.

  18. In Search of Good PatchesSelecting a Patch • Note: error image can be computed efficiently in the Fourier domain. Complexity: O(n log n)per patch. Input Texture (T) Error Image(E) Selected Patch (P)

  19. Adaptive Patch SamplingPrinciple • Essentially: Hierarchical Pattern Mapping applied to the plane[Soler et al. 2002] Synthesize White Patch i > max  Split Patch i < max

  20. Adaptive Patch SamplingTrade-off • Trade-off between preserving global structure and avoiding detail artifacts max = 0.01 max = 1.0 max = 0.04

  21. Adaptive Patch SamplingTrade-off • Trade-off between preserving global structure and avoiding detail artifacts max = 0.01 max = 1.0 max = 0.04

  22. Adaptive Patch SamplingTrade-off • Trade-off between preserving global structure and avoiding detail artifacts max = 0.01 max = 1.0 max = 0.04

  23. Overlap Re-synthesisOverlap error and pixel invalidation Patch and Image Mask Selected Patch (P)

  24. Overlap Re-synthesisOverlap error and pixel invalidation Image Mask (I) Selected Patch (P)

  25. Overlap Re-synthesisOverlap error and pixel invalidation Image Mask (I) Selected Patch (P) Error Surface (S)

  26. Overlap Re-synthesisOverlap error and pixel invalidation • If(Ji(x) !=0 and Si(x)> δmax ) as invalid and in need of per-pixel re-synthesis. else if( Ji(x) != 0 and Si(x) < δmax ) are preserved and a stepwise alpha mask (feathering) is applied during Compositing. (Trade-off: user-defined maxis the pixel error tolerance. Setting to 1 results in pure feathering.) Patch and Image Mask P with Invalid Pixels (blue) Error Surface (S)

  27. Overlap Re-synthesisCompositing P OVER Ri-1 Intermediate Result Patch with Invalid Pixels

  28. Overlap Re-synthesisCompositing • Simple scanline re-synthesis: possibly insufficient causal neighborhood within valid pixelregion • Solution: alternative ordering (Pixel Traversal Map) Composited Result Patch with Invalid Pixels

  29. Pixel Traversal-Map 1: BUILDTRAVERSALMAP(Pi) : Mi 2: level = 1 3: Dlevel ←binary image of size Pi initialized to 0 4: Dlevel ←set all valid pixels ∈ Pi to 1 5: Mi ←Dlevel 6: while (∃pixel ∈ Dlevel ∧ pixel = 0) do 7: level = level+1 8: Dlevel ← DILATE(Dlevel−1,Disk) 9: Mi ←Mi +(Dlevel ∧Dlevel−1) ∗level 10: end while 11: return Mi

  30. Overlap Re-synthesisPixel Traversal Map (Ordering) • Pixel Traversal Map: repeated morphological dilation of the binary support for valid pixels with a euclidian disk of radius 1 (city-block distance transform). Composited Result Patch with Invalid Pixels Pixel Traversal Map (M)

  31. Overlap Re-synthesisPixel Traversal Map (Ordering) • Pixel Traversal Map:step 1 Composited Result Patch with Invalid Pixels Pixel Traversal Map (M)

  32. Overlap Re-synthesisPixel Traversal Map (Ordering) • Pixel Traversal Map: step 2 ... Composited Result Patch with Invalid Pixels Pixel Traversal Map (M)

  33. Overlap Re-synthesisPixel Traversal Map (Ordering) • Pixel Traversal Map: ... step n Composited Result Patch with Invalid Pixels Pixel Traversal Map (M)

  34. Overlap Re-synthesisPixel Traversal Map (Ordering) • Art Restorer Analogy: stepwise restoration of the hole from the boundary of the existing image. Composited Result Patch with Invalid Pixels Pixel Traversal Map (M)

  35. Overlap Re-synthesisPer-Pixel Re-synthesis Synthesize red pixel from valid (or already re-synthesized) pixels Intermediate Result Pixel Traversal Map (M)

  36. Overlap Re-synthesisPer-Pixel Re-synthesis Extract image mask (Ij) and binary support (Jj) for best-pixel search in the input texture. Complexity: O(n log n) per pixel. Image Mask (Ij) Binary Support (Jj) Intermediate Result

  37. Overlap Re-synthesisPer-Pixel Re-synthesis Intermediate Result Pixel Traversal Map (M)

  38. Overlap Re-synthesisPer-Pixel Re-synthesis Intermediate Result Shifted Pixel Traversal Map (M)

  39. Overlap Re-synthesisPer-Pixel Re-synthesis Intermediate Result with Traversal Map Pixel Traversal Map (M)

  40. Overlap Re-synthesisPer-Pixel Re-synthesis Overlap Re-synthesis Step 1 Pixel Traversal Map (M)

  41. Overlap Re-synthesisPer-Pixel Re-synthesis Overlap Re-synthesis Step 2 Pixel Traversal Map (M)

  42. Overlap Re-synthesisPer-Pixel Re-synthesis Overlap Re-synthesis Step 3 Pixel Traversal Map (M)

  43. Overlap Re-synthesisPer-Pixel Re-synthesis Overlap Re-synthesis Result Pixel Traversal Map (M)

  44. ResultsOverlap Repair Comparisons No Repair IQ PBS HTS

  45. ResultsOverlap Repair Comparisons No Repair IQ PBS HTS

  46. ResultsOverlap Repair Comparisons No Repair IQ PBS HTS

  47. ResultsOverlap Repair Comparisons No Repair IQ PBS HTS

  48. ResultsOverlap Repair Comparisons No Repair IQ PBS HTS

  49. ResultsOverlap Repair Comparisons No Repair IQ PBS HTS

  50. ResultsOverlap Repair Comparisons No Repair IQ PBS HTS

More Related