hybrid texture synthesis
Download
Skip this Video
Download Presentation
Hybrid Texture Synthesis

Loading in 2 Seconds...

play fullscreen
1 / 67

Hybrid Texture Synthesis - PowerPoint PPT Presentation


  • 142 Views
  • Uploaded on

Hybrid Texture Synthesis. Andrew Nealen Marc Alexa Discrete Geometric Modeling Group (DGM) Technische Universität Darmstadt Eurographics Symposium on Rendering 2003. Outline. Introduction. n x m Input Texture. N x M Output Texture.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Hybrid Texture Synthesis' - cole-lynch


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
hybrid texture synthesis

Hybrid Texture Synthesis

Andrew NealenMarc AlexaDiscrete Geometric Modeling Group (DGM)Technische Universität Darmstadt

Eurographics Symposium on Rendering 2003

introduction
Introduction

nxm Input Texture

NxM Output Texture

  • The goal: Synthesize an output texture which isperceptually similarto the input texture. Also ensure that the result contains sufficient variation.

2D Texture Synthesis

introduction1
Introduction
  • Existing (and Impressive) Technology
    • Pixel-Based:
      • Non-parametric Sampling [Efros and Leung 1999]
      • Tree-structured Vector Quantization [Wei and Levoy 2000]
      • Image Analogies [Hertzmann et al. 2001]
    • Patch-Based:
      • Patch-Based Sampling [Guo et al. 2001]
      • Image Quilting [Efros and Freeman 2001]
introduction2
Introduction

Wei/Levoy Algorithm:

Can occur when using structured

Textures with rich histograms.

This is due to the L2 norm.

Possible Drawbacks ?

  • Loss of global structure
  • Loss of scale
  • Boundary mismatch
  • Blurring
introduction3
Introduction

Image Quilting Algorithm:

boundary mismatch artifacts are

noticeable

Possible Drawbacks ?

  • Loss of global structure
  • Loss of scale
  • Boundary mismatch
  • Blurring
introduction4
Introduction

Patch-Based Sampling Algorithm:

Can occur when texture features are not

well-alignedacross patch boundaries

Possible Drawbacks ?

  • Loss of global structure
  • Loss of scale
  • Boundary mismatch
  • Blurring
hybrid texture synthesis1
Hybrid Texture Synthesis
  • Combine best aspects from other approacheswhile avoiding (or improving over) known pitfalls:
    • Patch-based algorithms
      • Are good at preserving global structure
      • Can introduce artifacts along patch boundaries
    • Pixel-based algorithms
      • Preserve local coherence (MRF model: local and stationary)
      • Possibly fail to preserve global structure
      • This is especially problematic for textures with rich histograms and many high frequency features, due to the smoothing nature of the distance metric (L2norm)
hybrid texture synthesis hts
Hybrid Texture Synthesis (HTS)
  • Hybrid, two-fold approach to texture synthesis
    • Adaptive patch sampling [Soler et al. 2002]
      • Start with a uniform quadrilateral grid of patches (the output)
      • Recursively split a patch if the best fit taken from the input texture is not good enough (a user-defined tradeoffmax)
    • Overlap re-synthesis(novel overlap repair strategy)
      • Mark invalid pixels in the overlap region
      • Validitymap is defined by the error surface E(x)=(xsrc – xdst)2 and a user-defined tolerance parametermax
      • Define an ordering for the invalid pixels and re-synthesize them using a per-pixel strategy
hybridsynthesize algorithm
HybridSynthesizeAlgorithm

1: HYBRIDSYNTHESIZE(T,P,ov,Δmax,δmax,R) : R

2: for all patches pi ∈P do

3: [Pi,Δi]← FINDBESTPATCH(T,Ri−1, pi,ov)

4: if (Δi < Δmax or ISSINGLEPIXEL(pi)) then

5: Si ← ERRORSURFACE(Pi,Ri−1)

6: Pi ← MARKINVALIDPIXELS(Pi,Si,δmax)

7: Mi ← BUILDTRAVERSALMAP(Pi)

8: Ri,composite ← COMPOSE(Pi,Ri−1)

9: Ri ← OVERLAPRESYNTHESIS(T,Ri,composite,Mi)

10: else

11: Ps ← SPLITPATCH(pi)

12: ovs ←max(3, ov/2)

13: Ri ← HYBRIDSYNTHESIZE(T,Ps,ovs,Δmax,δmax,Ri−1)

14: end if

15: end for

16: return R

findbestpatch algo
FindBestPatch(Algo.)

1: FINDBESTPATCH(T,Ri−1, pi,ov) : [Pi,Δi]

2: Ii, Ji ← BUILDIMAGEMASK(Ri−1, pi,ov)

3: Ei ← ERRORIMAGE(T, Ii, Ji)

4: Ei,trim ← TRIMINVALIDREGIONS(Ei, Ji, pi)

5: [Pi,Δi]← BESTPATCH(Ei,trim,T)

6: return [Pi,Δi]

in search of good patches extracting the image mask
In Search of Good PatchesExtracting the image mask
  • Grow patch by overlap (toroidally)

Synthesize Black Patch i

in search of good patches extracting the image mask1
In Search of Good PatchesExtracting the image mask
  • Extract image mask (Ii) and binary support (Ji), and circularly shift to upper left corner

Synthesize Black Patch i

Image Mask (Ii)

Binary Support (Ji)

in search of good patches computing the error image
In Search of Good PatchesComputing the error image

* Compute error Ei(x0) between Ii and T for each circular shift x0=(x,y) of the input texture T

Input Texture (T)

Image Mask (Ii)

Binary Support (Ji)

in search of good patches computing the error image1
In Search of Good PatchesComputing the error image

C={R,G,B}

(both with RGB color values in [0,1])

WR,G,B = {0.299,0.587,0.114}

X0: circular shift of T

Input Texture (T)

Error Image(Ei)

(1)

in search of good patches selecting a patch
In Search of Good PatchesSelecting a Patch

Input Texture (T)

Error Image(E)

Selected Patch (P)

(2)

  • Given that Ji(x) Ii(x) = Ii(x),and the cross correlation between two images (functions) f ◇ g is defined as ( f ◇ g )(x0) = Σx f (x)g(x+x0)
  • The correlation f◇ g between two functions can be computed in O(N logN)
  • in fourier space as f ◇g = F-1(F( f )∗F(g))
  • (N :number of pixels in the input texture)
  • In implementation it’s can be pre-compute the fourier transform for Tand need only recompute the fourier transforms of Iiand Jifor each new patch pi.
in search of good patches selecting a patch1
In Search of Good PatchesSelecting a Patch
  • Note: error image can be computed efficiently in the Fourier domain. Complexity: O(n log n)per patch.

Input Texture (T)

Error Image(E)

Selected Patch (P)

adaptive patch sampling principle
Adaptive Patch SamplingPrinciple
  • Essentially: Hierarchical Pattern Mapping applied to the plane[Soler et al. 2002]

Synthesize White Patch

i > max  Split Patch

i < max

adaptive patch sampling trade off
Adaptive Patch SamplingTrade-off
  • Trade-off between preserving global structure and avoiding detail artifacts

max = 0.01

max = 1.0

max = 0.04

adaptive patch sampling trade off1
Adaptive Patch SamplingTrade-off
  • Trade-off between preserving global structure and avoiding detail artifacts

max = 0.01

max = 1.0

max = 0.04

adaptive patch sampling trade off2
Adaptive Patch SamplingTrade-off
  • Trade-off between preserving global structure and avoiding detail artifacts

max = 0.01

max = 1.0

max = 0.04

overlap re synthesis overlap error and pixel invalidation
Overlap Re-synthesisOverlap error and pixel invalidation

Patch and Image Mask

Selected Patch (P)

overlap re synthesis overlap error and pixel invalidation2
Overlap Re-synthesisOverlap error and pixel invalidation

Image Mask (I)

Selected Patch (P)

Error Surface (S)

overlap re synthesis overlap error and pixel invalidation3
Overlap Re-synthesisOverlap error and pixel invalidation
  • If(Ji(x) !=0 and Si(x)> δmax )

as invalid and in need of per-pixel re-synthesis.

else if( Ji(x) != 0 and Si(x) < δmax )

are preserved and a stepwise alpha mask (feathering) is applied during Compositing.

(Trade-off: user-defined maxis the pixel error tolerance. Setting to 1 results in pure feathering.)

Patch and Image Mask

P with Invalid Pixels (blue)

Error Surface (S)

overlap re synthesis compositing
Overlap Re-synthesisCompositing

P OVER Ri-1

Intermediate Result

Patch with Invalid Pixels

overlap re synthesis compositing1
Overlap Re-synthesisCompositing
  • Simple scanline re-synthesis: possibly insufficient causal neighborhood within valid pixelregion
  • Solution: alternative ordering (Pixel Traversal Map)

Composited Result

Patch with Invalid Pixels

pixel traversal map
Pixel Traversal-Map

1: BUILDTRAVERSALMAP(Pi) : Mi

2: level = 1

3: Dlevel ←binary image of size Pi initialized to 0

4: Dlevel ←set all valid pixels ∈ Pi to 1

5: Mi ←Dlevel

6: while (∃pixel ∈ Dlevel ∧ pixel = 0) do

7: level = level+1

8: Dlevel ← DILATE(Dlevel−1,Disk)

9: Mi ←Mi +(Dlevel ∧Dlevel−1) ∗level

10: end while

11: return Mi

overlap re synthesis pixel traversal map ordering
Overlap Re-synthesisPixel Traversal Map (Ordering)
  • Pixel Traversal Map: repeated morphological dilation of the binary support for valid pixels with a euclidian disk of radius 1 (city-block distance transform).

Composited Result

Patch with Invalid Pixels

Pixel Traversal Map (M)

overlap re synthesis pixel traversal map ordering1
Overlap Re-synthesisPixel Traversal Map (Ordering)
  • Pixel Traversal Map:step 1

Composited Result

Patch with Invalid Pixels

Pixel Traversal Map (M)

overlap re synthesis pixel traversal map ordering2
Overlap Re-synthesisPixel Traversal Map (Ordering)
  • Pixel Traversal Map: step 2 ...

Composited Result

Patch with Invalid Pixels

Pixel Traversal Map (M)

overlap re synthesis pixel traversal map ordering3
Overlap Re-synthesisPixel Traversal Map (Ordering)
  • Pixel Traversal Map: ... step n

Composited Result

Patch with Invalid Pixels

Pixel Traversal Map (M)

overlap re synthesis pixel traversal map ordering4
Overlap Re-synthesisPixel Traversal Map (Ordering)
  • Art Restorer Analogy: stepwise restoration of the hole from the boundary of the existing image.

Composited Result

Patch with Invalid Pixels

Pixel Traversal Map (M)

overlap re synthesis per pixel re synthesis
Overlap Re-synthesisPer-Pixel Re-synthesis

Synthesize red pixel from valid (or already re-synthesized) pixels

Intermediate Result

Pixel Traversal Map (M)

overlap re synthesis per pixel re synthesis1
Overlap Re-synthesisPer-Pixel Re-synthesis

Extract image mask (Ij) and binary support (Jj) for best-pixel search in the input texture. Complexity: O(n log n) per pixel.

Image Mask (Ij)

Binary Support (Jj)

Intermediate Result

overlap re synthesis per pixel re synthesis2
Overlap Re-synthesisPer-Pixel Re-synthesis

Intermediate Result

Pixel Traversal Map (M)

overlap re synthesis per pixel re synthesis3
Overlap Re-synthesisPer-Pixel Re-synthesis

Intermediate Result

Shifted

Pixel Traversal Map (M)

overlap re synthesis per pixel re synthesis4
Overlap Re-synthesisPer-Pixel Re-synthesis

Intermediate Result

with Traversal Map

Pixel Traversal Map (M)

overlap re synthesis per pixel re synthesis5
Overlap Re-synthesisPer-Pixel Re-synthesis

Overlap Re-synthesis

Step 1

Pixel Traversal Map (M)

overlap re synthesis per pixel re synthesis6
Overlap Re-synthesisPer-Pixel Re-synthesis

Overlap Re-synthesis

Step 2

Pixel Traversal Map (M)

overlap re synthesis per pixel re synthesis7
Overlap Re-synthesisPer-Pixel Re-synthesis

Overlap Re-synthesis

Step 3

Pixel Traversal Map (M)

overlap re synthesis per pixel re synthesis8
Overlap Re-synthesisPer-Pixel Re-synthesis

Overlap Re-synthesis

Result

Pixel Traversal Map (M)

results synthesis c omparisons
ResultsSynthesis Comparisons

Input

Efros/Leung

Wei/Levoy

IQ

PBS

HTS

results synthesis c omparisons1
ResultsSynthesis Comparisons

Input

Efros/Leung

Wei/Levoy

IQ

PBS

HTS

results synthesis c omparisons2
ResultsSynthesis Comparisons

Input

Efros/Leung

Wei/Levoy

IQ

PBS

HTS

results synthesis c omparisons3
ResultsSynthesis Comparisons

Input

Efros/Leung

Wei/Levoy

IQ

PBS

HTS

results synthesis c omparisons4
ResultsSynthesis Comparisons

Input

Efros/Leung

Wei/Levoy

IQ

PBS

HTS

results synthesis c omparisons5
ResultsSynthesis Comparisons

Input

Efros/Leung

Wei/Levoy

IQ

PBS

HTS

results synthesis c omparisons6
ResultsSynthesis Comparisons

Input

PBS

HTS

Input

PBS

HTS

results synthesis c omparisons7
ResultsSynthesis Comparisons

Input

PBS

HTS

Input

PBS

HTS

results synthesis c omparisons8
ResultsSynthesis Comparisons

Input

PBS

HTS

Input

PBS

HTS

results
Results

presence of high frequency structure cause the error metric (L2 Norm)

conclusions and future work
Conclusions and Future Work
  • Improve Computational Complexity
    • Pixel neighborhoods of patch and pixel stage are not known a priori, so precomputation is not straightforward
    • We can solve this in the pixel stage by employing k-coherence search [Tong et al. 2002] [Ashikhmin 2001]
  • Improve Error Metric
    • Still using the L2 norm due to its simplicity
    • Develop a metric which takes feature mismatch into account
ad