Overview of texture synthesis
This presentation is the property of its rightful owner.
Sponsored Links
1 / 56

Overview of Texture Synthesis PowerPoint PPT Presentation


  • 100 Views
  • Uploaded on
  • Presentation posted in: General

Overview of Texture Synthesis. Ganesh Ramanarayanan Cornell Graphics Seminar. ???. small texture. big texture. The Problem. ???. small texture. big texture. A Simple Solution …. small texture. A Simple Solution … Tiling. ???. big texture. A Simple Solution … Tiling.

Download Presentation

Overview of Texture Synthesis

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Overview of texture synthesis

Overview of Texture Synthesis

Ganesh Ramanarayanan

Cornell Graphics Seminar


The problem

???

small texture

big texture

The Problem


A simple solution

???

small texture

big texture

A Simple Solution …


A simple solution tiling

small texture

A Simple Solution … Tiling

???

big texture


A simple solution tiling1

A Simple Solution … Tiling

small texture

Tiling is obvious!

big texture?


Ideal solution

Ideal Solution?


The problem in words

The Problem … in Words

  • Given texture I, generate a texture J which

    • Looks like the same texture

    • Has no obvious copying or tiling from I

    • Difference between I and J should be the same as the way I “differs from itself” [DeBonet97]

  • Things to watch for:

    • ‘Looks the same’: what is the texture model?

    • ‘Obvious copying’: how is it avoided?

    • Underlined text: indicates algorithm parameter


Classes of algorithms

Classes of Algorithms

  • Multiresolution pyramids

    • [HeegerBergen95]

  • Pixel-by-pixel synthesis

    • [EfrosLeung99]

  • Multiresolution pixel-by-pixel

    • [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01], [Ashikhmin01]

  • Patch quilting

    • [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04]

  • Geometric feature matching

    • [WuYu04], [Liu et.al. 04]


Heeger bergen 1995

Heeger Bergen 1995

  • Seminal paper that introduced texture synthesis to the graphics community

  • Algorithm:

    • Initialize J to noise

    • Create multiresolution pyramids for I and J

    • Match the histograms of J’s pyramid levels with I’s pyramid levels

    • Loop until convergence

    • Can be generalized to 3D


Heeger bergen 1995 algorithm

Heeger Bergen 1995 - Algorithm

  • Image pyramids

    • Gaussian

    • Laplacian

  • Steerable pyramids [SimoncelliFreeman95]

    • b): multiple scales of oriented filters

    • c): a sample image

    • d): results of filters in b) applied to c)


Heeger bergen 1995 results

I

I

J

J

Heeger Bergen 1995 - Results

Successes

Failures


Heeger bergen 1995 results1

Heeger Bergen 1995 - Results


Heeger bergen 1995 verdict

Heeger Bergen 1995 - Verdict

  • Texture model:

    • Histograms of responses to various filters

  • Avoiding copying:

    • Inherent in algorithm

  • No user intervention required

  • Captures stochastic textures well

  • Does not capture structure

    • Lack of inter-scale constraints


De bonet 1997

De Bonet 1997

  • Propagate constraints downwards by matching statistics all the way up the pyramid

  • Feature vector:multiscale collection of filter responses for a given pixel

  • Algorithm:

    • Initialize J to empty image

    • Create multiresolution pyramids for I and J

    • For each pixel in level of J, randomly choose pixel from corresponding level of I that has similar feature vector


De bonet 1997 algorithm

De Bonet 1997 - Algorithm

  • 6 feature vectors shown

  • Notice how they share parent information


De bonet 1997 results

De Bonet 1997 - Results


De bonet 1997 verdict

De Bonet 1997 - Verdict

  • Texture model:

    • Feature vector containing multiscale responses to various filters

  • Avoiding copying:

    • Random choice of pixels with ‘close’ feature vectors, but copying still frequent on small scale

  • Individual per-filter thresholds are cumbersome

  • Feature vectors used in later synthesis work


Efros leung 1999

Efros Leung 1999

  • Markov Random Fields: pixels synthesized probabilistically based on causal neighborhood

  • Algorithm:

    • Initialize J to empty

    • Synthesize new pixels in a spiral pattern

    • Select new pixel from I based on conditional probabilities given the current neighborhood in J and similar neighborhoods in I


Efros leung 1999 algorithm

Efros Leung 1999 - Algorithm

I

J

‘causal’

neighborhood


Efros leung 1999 results

Efros Leung 1999 - Results


Efros leung 1999 verdict

Efros Leung 1999 – Verdict

  • Texture model:

    • MRF

  • Avoiding copying:

    • MRF

  • Neighborhood size = largest feature size

  • Markov model is surprisingly good

    • “I spent an interesting evening recently with a grain of salt.”

  • Search is very slow with large neighborhoods


Wei levoy 2000

Wei Levoy 2000

  • Adds tree-structured vector quantization speedup and multiresolution matching

  • Algorithm:

    • Initialize J to noise

    • Synthesize new pixels in scanline order

    • Select new pixel from I that has the closest matching neighborhood to J

    • In multiresolution case, look instead at neighborhood of feature vectors


Wei levoy 2000 algorithm

Wei Levoy 2000 - Algorithm

  • Scanline synthesis with neighborhoods

    • a) input texture

    • b) algorithm start

    • c) midway point

    • d) ending


Wei levoy 2000 results

Wei Levoy 2000 - Results


Wei levoy 2000 verdict

Wei Levoy 2000 - Verdict

  • Texture model:

    • Multiscale filters and MRF

  • Avoiding copying:

    • Not explicitly, but in practice inherently avoided; TSVQ adds additional unpredictability

  • Results comparable to [EfrosLeung99]

  • TSVQ speedup comes at a price (quality loss)


Ashikhmin 2001

Ashikhmin 2001

  • Adds more verbatim image copying and user-constrained synthesis

  • Algorithm:

    • New array J* stores location in I of J’s pixels

    • Initialize J to empty, J* to random

    • Synthesize new pixels in scanline order

    • Select new pixel by copying from patches of I, as determined by locations in J*

    • In user-constrained case, also include error comparison with user-specified target image T


Ashikhmin 2001 algorithm

Ashikhmin 2001 - Algorithm

  • Candidates:

  • x=4, y=5

  • x=18, y=16

  • x=11, y=12

x=3

y=4

x=4

y=4

x=12

y=11

x=17

y=16

x=18

y=16

J*


Ashikhmin 2001 results

Ashikhmin 2001 - Results


Ashikhmin 2001 verdict

Ashikhmin 2001 - Verdict

  • Texture model:

    • MRF

  • Avoiding copying:

    • Actually, here it is encouraged on a small scale, but in practice it doesn’t occur on a large scale

  • Recognized that copying preserves fine detail

  • Features can get ‘cut off’ because of the abrupt jumping from patch to patch


Hertzmann et al 2001

Hertzmann et. al. 2001

  • Framework for solving image analogies: learning an image filter f by example

    • Given A, f(A), and B, generate f(B)

  • To handle texture synthesis, ignore A, B, and let f(A) = I, f(B) = J

  • Algorithm:

    • Bring image histograms into correspondence

    • Combine [WeiLevoy00] and [Ashikhmin01] Each search method is weighted appropriately and then the best choice is used


Hertzmann et al 2001 algorithm

Hertzmann et. al. 2001 - Algorithm

  • A provides set of constraints

  • Histogram matching critical because of color differences between A and B sets of images


Hertzmann et al 2001 results

I

I

J

J

Hertzmann et. al. 2001 - Results


Hertzmann et al 2001 results1

Hertzmann et. al. 2001 - Results


Hertzmann et al 2001 verdict

Hertzmann et. al. 2001 - Verdict

  • Texture model:

    • Multiscale filters and MRF

  • Avoiding copying:

    • See [WeiLevoy00] and [Ashikhmin01]

  • Unique approach to image manipulation and texture synthesis

  • Same fundamental limitations as [WeiLevoy00] and [Ashikhmin01]


Efros freeman 2001

Efros Freeman 2001

  • Inspired by chaos mosaics [Xu et. al. 00]

  • Make an ‘image quilt’ by pasting together patches of the input and fixing seams

  • Algorithm:

    • Initialize J to empty

    • Copy new patches of certain size in scanline order, with fixed overlap width

    • Randomly select new patch that has error in overlap region less than given threshold

    • Copy pixels based on minimal error boundary cut


Efros freeman 2001 algorithm

B1

B1

B2

B2

Neighboring blocks

constrained by overlap

Minimal error

boundary cut

Efros Freeman 2001 - Algorithm

Input texture

block


Efros freeman 2001 results

Efros Freeman 2001 - Results


Efros freeman 2001 verdict

Efros Freeman 2001 - Verdict

  • Texture model:

    • ??

  • Avoiding copying:

    • Randomized patch selection, but still noticeable

  • Patch size is a hard parameter to understand

  • Results are surprisingly good given algorithm

  • Multiscale goes on a brief hiatus


Kwatra et al 2003

Kwatra et. al. 2003

  • Generalizes seam computation in overlap regions as a graph cut problem

    • Based on [Boykov et. al. 99] (with Ramin Zabih)

  • Algorithm:

    • Initialize J to empty

    • Copy pieces of I to J using a variety of methods

    • Formulate flow graph in overlap region based on errors and compute minimum cut

    • Copy sink-side pixels to J

    • Variety of strategies to further hide seams


Kwatra et al 2003 algorithm

Kwatra et. al. 2003 - Algorithm

(assume cut region is 3x3 for simplicity)


Kwatra et al 2003 results

Kwatra et. al. 2003 - Results


Kwatra et al 2003 results1

Kwatra et. al. 2003 - Results


Kwatra et al 2003 verdict

Kwatra et. al. 2003 - Verdict

  • Texture model:

    • MRF

  • Avoiding copying:

    • Even with a multitude of patch selection methods, still noticeable when it happens repeatedly

  • Paper presents a bag of synthesis tricks without much intuition for when to use what

  • Graph cut formalization is useful and powerful


Wu yu 2004

Wu Yu 2004

  • Synthesizes a feature image J* in parallel

  • Tries to match features in overlap region

  • Algorithm:

    • Compute feature image I*

    • Initialize J* to empty

    • Synthesize J and J* in parallel

    • When searching for new patch, add in weighted feature matching error

    • Prior to copying, warp selected patch to match features exactly


Wu yu 2004 algorithm

Wu Yu 2004 - Algorithm

  • Match feature pixels based on

    • Tangent orientation

    • Distance

    • Bijectivity

  • Example:

    • L1out matches L1in

    • L2out matches L3in

    • Warping will move matching features closer to each other

Feature correspondence


Wu yu 2004 results

Wu Yu 2004: Results


Wu yu 2004 verdict

Wu Yu 2004: Verdict

  • Texture model:

    • Feature lines and MRF

  • Avoiding copying:

    • No; in fact, mirroring is often very noticeable

  • Unclear how to combine this metric with others

  • Insight to use warping is valuable, although the next paper does a much better job with it


Liu et al 2004

Liu et. al. 2004

  • Regular textures: can be tiled

  • Near-regular textures: statistical distortion of a regular texture

  • Algorithm (simplified):

    • Mark lattice points in I

    • Transform I into regular texture R

    • Express transformation as deformation texture I*

    • Synthesize larger deformation J*

    • Synthesize J by combining J* with R


Liu et al 2004 algorithm

Liu et. al. 2004 - Algorithm

I

R

Deformation Texture I*


Liu et al 2004 results

Liu et. al. 2004 - Results


Liu et al 2004 results1

Liu et. al. 2004 - Results

texture transfer

original


Liu et al 2004 verdict

Liu et. al. 2004 - Verdict

  • Texture model:

    • Regular texture with modulated geometry and color

  • Avoiding copying:

    • Depends on the synthesis algorithm

  • Proposes specific model to handle specific types of textures

  • Interestingly, this method depends on previous texture synthesis methods


Classes of algorithms recap

Classes of Algorithms - Recap

  • Multiresolution pyramids

    • [HeegerBergen95]

  • Pixel-by-pixel synthesis

    • [EfrosLeung99]

  • Multiresolution pixel-by-pixel

    • [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01], [Ashikhmin01]

  • Patch quilting

    • [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04]

  • Geometric feature matching

    • [WuYu04], [Liu et.al. 04]


Evolution of texture model

Evolution of Texture Model

  • Responses to image filters / decompositions

  • Markov Random Fields

  • Copies of small pieces of image

  • Feature data

  • Various combinations of the above


What s next

=

+

What’s Next?

  • Texture synthesis is really hard

  • Rather than creating a silver bullet, focus on algorithms for more specific problems

  • Superresolution:

???


What s next1

=

+

What’s Next?

  • Texture synthesis is really hard

  • Rather than creating a silver bullet, focus on algorithms for more specific problems

  • Superresolution:

[Hertzmann et. al. 01]


  • Login