Hybrid texture synthesis
This presentation is the property of its rightful owner.
Sponsored Links
1 / 67

Hybrid Texture Synthesis PowerPoint PPT Presentation


  • 103 Views
  • Uploaded on
  • Presentation posted in: General

Hybrid Texture Synthesis. Andrew Nealen Marc Alexa Discrete Geometric Modeling Group (DGM) Technische Universität Darmstadt Eurographics Symposium on Rendering 2003. Outline. Introduction. n x m Input Texture. N x M Output Texture.

Download Presentation

Hybrid Texture Synthesis

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Hybrid texture synthesis

Hybrid Texture Synthesis

Andrew NealenMarc AlexaDiscrete Geometric Modeling Group (DGM)Technische Universität Darmstadt

Eurographics Symposium on Rendering 2003


Outline

Outline


Introduction

Introduction

nxm Input Texture

NxM Output Texture

  • The goal: Synthesize an output texture which isperceptually similarto the input texture. Also ensure that the result contains sufficient variation.

2D Texture Synthesis


Introduction1

Introduction

  • Existing (and Impressive) Technology

    • Pixel-Based:

      • Non-parametric Sampling [Efros and Leung 1999]

      • Tree-structured Vector Quantization [Wei and Levoy 2000]

      • Image Analogies [Hertzmann et al. 2001]

    • Patch-Based:

      • Patch-Based Sampling [Guo et al. 2001]

      • Image Quilting [Efros and Freeman 2001]


Introduction2

Introduction

Wei/Levoy Algorithm:

Can occur when using structured

Textures with rich histograms.

This is due to the L2 norm.

Possible Drawbacks ?

  • Loss of global structure

  • Loss of scale

  • Boundary mismatch

  • Blurring


Introduction3

Introduction

Image Quilting Algorithm:

boundary mismatch artifacts are

noticeable

Possible Drawbacks ?

  • Loss of global structure

  • Loss of scale

  • Boundary mismatch

  • Blurring


Introduction4

Introduction

Patch-Based Sampling Algorithm:

Can occur when texture features are not

well-alignedacross patch boundaries

Possible Drawbacks ?

  • Loss of global structure

  • Loss of scale

  • Boundary mismatch

  • Blurring


Hybrid texture synthesis1

Hybrid Texture Synthesis

  • Combine best aspects from other approacheswhile avoiding (or improving over) known pitfalls:

    • Patch-based algorithms

      • Are good at preserving global structure

      • Can introduce artifacts along patch boundaries

    • Pixel-based algorithms

      • Preserve local coherence (MRF model: local and stationary)

      • Possibly fail to preserve global structure

      • This is especially problematic for textures with rich histograms and many high frequency features, due to the smoothing nature of the distance metric (L2norm)


Hybrid texture synthesis hts

Hybrid Texture Synthesis (HTS)

  • Hybrid, two-fold approach to texture synthesis

    • Adaptive patch sampling [Soler et al. 2002]

      • Start with a uniform quadrilateral grid of patches (the output)

      • Recursively split a patch if the best fit taken from the input texture is not good enough (a user-defined tradeoffmax)

    • Overlap re-synthesis(novel overlap repair strategy)

      • Mark invalid pixels in the overlap region

      • Validitymap is defined by the error surface E(x)=(xsrc – xdst)2 and a user-defined tolerance parametermax

      • Define an ordering for the invalid pixels and re-synthesize them using a per-pixel strategy


Hybridsynthesize algorithm

HybridSynthesizeAlgorithm

1: HYBRIDSYNTHESIZE(T,P,ov,Δmax,δmax,R) : R

2: for all patches pi ∈P do

3: [Pi,Δi]← FINDBESTPATCH(T,Ri−1, pi,ov)

4: if (Δi < Δmax or ISSINGLEPIXEL(pi)) then

5: Si ← ERRORSURFACE(Pi,Ri−1)

6: Pi ← MARKINVALIDPIXELS(Pi,Si,δmax)

7: Mi ← BUILDTRAVERSALMAP(Pi)

8: Ri,composite ← COMPOSE(Pi,Ri−1)

9: Ri ← OVERLAPRESYNTHESIS(T,Ri,composite,Mi)

10: else

11: Ps ← SPLITPATCH(pi)

12: ovs ←max(3, ov/2)

13: Ri ← HYBRIDSYNTHESIZE(T,Ps,ovs,Δmax,δmax,Ri−1)

14: end if

15: end for

16: return R


Findbestpatch algo

FindBestPatch(Algo.)

1: FINDBESTPATCH(T,Ri−1, pi,ov) : [Pi,Δi]

2: Ii, Ji ← BUILDIMAGEMASK(Ri−1, pi,ov)

3: Ei ← ERRORIMAGE(T, Ii, Ji)

4: Ei,trim ← TRIMINVALIDREGIONS(Ei, Ji, pi)

5: [Pi,Δi]← BESTPATCH(Ei,trim,T)

6: return [Pi,Δi]


In search of good patches synthesizing a single patch

In Search of Good PatchesSynthesizing a single patch

Synthesize Black Patch i


In search of good patches extracting the image mask

In Search of Good PatchesExtracting the image mask

  • Grow patch by overlap (toroidally)

Synthesize Black Patch i


In search of good patches extracting the image mask1

In Search of Good PatchesExtracting the image mask

  • Extract image mask (Ii) and binary support (Ji), and circularly shift to upper left corner

Synthesize Black Patch i

Image Mask (Ii)

Binary Support (Ji)


In search of good patches computing the error image

In Search of Good PatchesComputing the error image

* Compute error Ei(x0) between Ii and T for each circular shift x0=(x,y) of the input texture T

Input Texture (T)

Image Mask (Ii)

Binary Support (Ji)


In search of good patches computing the error image1

In Search of Good PatchesComputing the error image

C={R,G,B}

(both with RGB color values in [0,1])

WR,G,B = {0.299,0.587,0.114}

X0: circular shift of T

Input Texture (T)

Error Image(Ei)

(1)


In search of good patches selecting a patch

In Search of Good PatchesSelecting a Patch

Input Texture (T)

Error Image(E)

Selected Patch (P)

(2)

  • Given that Ji(x) Ii(x) = Ii(x),and the cross correlation between two images (functions) f ◇ g is defined as ( f ◇ g )(x0) = Σx f (x)g(x+x0)

  • The correlation f◇ g between two functions can be computed in O(N logN)

  • in fourier space as f ◇g = F-1(F( f )∗F(g))

  • (N :number of pixels in the input texture)

  • In implementation it’s can be pre-compute the fourier transform for Tand need only recompute the fourier transforms of Iiand Jifor each new patch pi.


In search of good patches selecting a patch1

In Search of Good PatchesSelecting a Patch

  • Note: error image can be computed efficiently in the Fourier domain. Complexity: O(n log n)per patch.

Input Texture (T)

Error Image(E)

Selected Patch (P)


Adaptive patch sampling principle

Adaptive Patch SamplingPrinciple

  • Essentially: Hierarchical Pattern Mapping applied to the plane[Soler et al. 2002]

Synthesize White Patch

i > max  Split Patch

i < max


Adaptive patch sampling trade off

Adaptive Patch SamplingTrade-off

  • Trade-off between preserving global structure and avoiding detail artifacts

max = 0.01

max = 1.0

max = 0.04


Adaptive patch sampling trade off1

Adaptive Patch SamplingTrade-off

  • Trade-off between preserving global structure and avoiding detail artifacts

max = 0.01

max = 1.0

max = 0.04


Adaptive patch sampling trade off2

Adaptive Patch SamplingTrade-off

  • Trade-off between preserving global structure and avoiding detail artifacts

max = 0.01

max = 1.0

max = 0.04


Overlap re synthesis overlap error and pixel invalidation

Overlap Re-synthesisOverlap error and pixel invalidation

Patch and Image Mask

Selected Patch (P)


Overlap re synthesis overlap error and pixel invalidation1

Overlap Re-synthesisOverlap error and pixel invalidation

Image Mask (I)

Selected Patch (P)


Overlap re synthesis overlap error and pixel invalidation2

Overlap Re-synthesisOverlap error and pixel invalidation

Image Mask (I)

Selected Patch (P)

Error Surface (S)


Overlap re synthesis overlap error and pixel invalidation3

Overlap Re-synthesisOverlap error and pixel invalidation

  • If(Ji(x) !=0 and Si(x)> δmax )

    as invalid and in need of per-pixel re-synthesis.

    else if( Ji(x) != 0 and Si(x) < δmax )

    are preserved and a stepwise alpha mask (feathering) is applied during Compositing.

    (Trade-off: user-defined maxis the pixel error tolerance. Setting to 1 results in pure feathering.)

Patch and Image Mask

P with Invalid Pixels (blue)

Error Surface (S)


Overlap re synthesis compositing

Overlap Re-synthesisCompositing

P OVER Ri-1

Intermediate Result

Patch with Invalid Pixels


Overlap re synthesis compositing1

Overlap Re-synthesisCompositing

  • Simple scanline re-synthesis: possibly insufficient causal neighborhood within valid pixelregion

  • Solution: alternative ordering (Pixel Traversal Map)

Composited Result

Patch with Invalid Pixels


Pixel traversal map

Pixel Traversal-Map

1: BUILDTRAVERSALMAP(Pi) : Mi

2: level = 1

3: Dlevel ←binary image of size Pi initialized to 0

4: Dlevel ←set all valid pixels ∈ Pi to 1

5: Mi ←Dlevel

6: while (∃pixel ∈ Dlevel ∧ pixel = 0) do

7: level = level+1

8:Dlevel ← DILATE(Dlevel−1,Disk)

9: Mi ←Mi +(Dlevel ∧Dlevel−1) ∗level

10: end while

11: return Mi


Overlap re synthesis pixel traversal map ordering

Overlap Re-synthesisPixel Traversal Map (Ordering)

  • Pixel Traversal Map: repeated morphological dilation of the binary support for valid pixels with a euclidian disk of radius 1 (city-block distance transform).

Composited Result

Patch with Invalid Pixels

Pixel Traversal Map (M)


Overlap re synthesis pixel traversal map ordering1

Overlap Re-synthesisPixel Traversal Map (Ordering)

  • Pixel Traversal Map:step 1

Composited Result

Patch with Invalid Pixels

Pixel Traversal Map (M)


Overlap re synthesis pixel traversal map ordering2

Overlap Re-synthesisPixel Traversal Map (Ordering)

  • Pixel Traversal Map: step 2 ...

Composited Result

Patch with Invalid Pixels

Pixel Traversal Map (M)


Overlap re synthesis pixel traversal map ordering3

Overlap Re-synthesisPixel Traversal Map (Ordering)

  • Pixel Traversal Map: ... step n

Composited Result

Patch with Invalid Pixels

Pixel Traversal Map (M)


Overlap re synthesis pixel traversal map ordering4

Overlap Re-synthesisPixel Traversal Map (Ordering)

  • Art Restorer Analogy: stepwise restoration of the hole from the boundary of the existing image.

Composited Result

Patch with Invalid Pixels

Pixel Traversal Map (M)


Overlap re synthesis per pixel re synthesis

Overlap Re-synthesisPer-Pixel Re-synthesis

Synthesize red pixel from valid (or already re-synthesized) pixels

Intermediate Result

Pixel Traversal Map (M)


Overlap re synthesis per pixel re synthesis1

Overlap Re-synthesisPer-Pixel Re-synthesis

Extract image mask (Ij) and binary support (Jj) for best-pixel search in the input texture. Complexity: O(n log n) per pixel.

Image Mask (Ij)

Binary Support (Jj)

Intermediate Result


Overlap re synthesis per pixel re synthesis2

Overlap Re-synthesisPer-Pixel Re-synthesis

Intermediate Result

Pixel Traversal Map (M)


Overlap re synthesis per pixel re synthesis3

Overlap Re-synthesisPer-Pixel Re-synthesis

Intermediate Result

Shifted

Pixel Traversal Map (M)


Overlap re synthesis per pixel re synthesis4

Overlap Re-synthesisPer-Pixel Re-synthesis

Intermediate Result

with Traversal Map

Pixel Traversal Map (M)


Overlap re synthesis per pixel re synthesis5

Overlap Re-synthesisPer-Pixel Re-synthesis

Overlap Re-synthesis

Step 1

Pixel Traversal Map (M)


Overlap re synthesis per pixel re synthesis6

Overlap Re-synthesisPer-Pixel Re-synthesis

Overlap Re-synthesis

Step 2

Pixel Traversal Map (M)


Overlap re synthesis per pixel re synthesis7

Overlap Re-synthesisPer-Pixel Re-synthesis

Overlap Re-synthesis

Step 3

Pixel Traversal Map (M)


Overlap re synthesis per pixel re synthesis8

Overlap Re-synthesisPer-Pixel Re-synthesis

Overlap Re-synthesis

Result

Pixel Traversal Map (M)


Results overlap repair comparisons

ResultsOverlap Repair Comparisons

No Repair

IQ

PBS

HTS


Results overlap repair comparisons1

ResultsOverlap Repair Comparisons

No Repair

IQ

PBS

HTS


Results overlap repair comparisons2

ResultsOverlap Repair Comparisons

No Repair

IQ

PBS

HTS


Results overlap repair comparisons3

ResultsOverlap Repair Comparisons

No Repair

IQ

PBS

HTS


Results overlap repair comparisons4

ResultsOverlap Repair Comparisons

No Repair

IQ

PBS

HTS


Results overlap repair comparisons5

ResultsOverlap Repair Comparisons

No Repair

IQ

PBS

HTS


Results overlap repair comparisons6

ResultsOverlap Repair Comparisons

No Repair

IQ

PBS

HTS


Results overlap repair comparisons7

ResultsOverlap Repair Comparisons

No Repair

IQ

PBS

HTS


Results overlap repair comparisons8

ResultsOverlap Repair Comparisons

No Repair

IQ

PBS

HTS


Results overlap repair comparisons9

ResultsOverlap Repair Comparisons

No Repair

IQ

PBS

HTS


Results overlap repair comparisons10

ResultsOverlap Repair Comparisons

No Repair

IQ

PBS

HTS


Results overlap repair comparisons11

ResultsOverlap Repair Comparisons

No Repair

IQ

PBS

HTS


Results overlap repair comparisons12

ResultsOverlap Repair Comparisons

No Repair

IQ

PBS

HTS


Results synthesis c omparisons

ResultsSynthesis Comparisons

Input

Efros/Leung

Wei/Levoy

IQ

PBS

HTS


Results synthesis c omparisons1

ResultsSynthesis Comparisons

Input

Efros/Leung

Wei/Levoy

IQ

PBS

HTS


Results synthesis c omparisons2

ResultsSynthesis Comparisons

Input

Efros/Leung

Wei/Levoy

IQ

PBS

HTS


Results synthesis c omparisons3

ResultsSynthesis Comparisons

Input

Efros/Leung

Wei/Levoy

IQ

PBS

HTS


Results synthesis c omparisons4

ResultsSynthesis Comparisons

Input

Efros/Leung

Wei/Levoy

IQ

PBS

HTS


Results synthesis c omparisons5

ResultsSynthesis Comparisons

Input

Efros/Leung

Wei/Levoy

IQ

PBS

HTS


Results synthesis c omparisons6

ResultsSynthesis Comparisons

Input

PBS

HTS

Input

PBS

HTS


Results synthesis c omparisons7

ResultsSynthesis Comparisons

Input

PBS

HTS

Input

PBS

HTS


Results synthesis c omparisons8

ResultsSynthesis Comparisons

Input

PBS

HTS

Input

PBS

HTS


Results

Results

presence of high frequency structure cause the error metric (L2 Norm)


Conclusions and future work

Conclusions and Future Work

  • Improve Computational Complexity

    • Pixel neighborhoods of patch and pixel stage are not known a priori, so precomputation is not straightforward

    • We can solve this in the pixel stage by employing k-coherence search [Tong et al. 2002] [Ashikhmin 2001]

  • Improve Error Metric

    • Still using the L2 norm due to its simplicity

    • Develop a metric which takes feature mismatch into account


  • Login