overview of texture synthesis n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Overview of Texture Synthesis PowerPoint Presentation
Download Presentation
Overview of Texture Synthesis

Loading in 2 Seconds...

play fullscreen
1 / 56

Overview of Texture Synthesis - PowerPoint PPT Presentation


  • 148 Views
  • Uploaded on

Overview of Texture Synthesis. Ganesh Ramanarayanan Cornell Graphics Seminar. ???. small texture. big texture. The Problem. ???. small texture. big texture. A Simple Solution …. small texture. A Simple Solution … Tiling. ???. big texture. A Simple Solution … Tiling.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Overview of Texture Synthesis' - dixon


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
overview of texture synthesis

Overview of Texture Synthesis

Ganesh Ramanarayanan

Cornell Graphics Seminar

the problem

???

small texture

big texture

The Problem
a simple solution

???

small texture

big texture

A Simple Solution …
a simple solution tiling1
A Simple Solution … Tiling

small texture

Tiling is obvious!

big texture?

the problem in words
The Problem … in Words
  • Given texture I, generate a texture J which
    • Looks like the same texture
    • Has no obvious copying or tiling from I
    • Difference between I and J should be the same as the way I “differs from itself” [DeBonet97]
  • Things to watch for:
    • ‘Looks the same’: what is the texture model?
    • ‘Obvious copying’: how is it avoided?
    • Underlined text: indicates algorithm parameter
classes of algorithms
Classes of Algorithms
  • Multiresolution pyramids
    • [HeegerBergen95]
  • Pixel-by-pixel synthesis
    • [EfrosLeung99]
  • Multiresolution pixel-by-pixel
    • [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01], [Ashikhmin01]
  • Patch quilting
    • [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04]
  • Geometric feature matching
    • [WuYu04], [Liu et.al. 04]
heeger bergen 1995
Heeger Bergen 1995
  • Seminal paper that introduced texture synthesis to the graphics community
  • Algorithm:
    • Initialize J to noise
    • Create multiresolution pyramids for I and J
    • Match the histograms of J’s pyramid levels with I’s pyramid levels
    • Loop until convergence
    • Can be generalized to 3D
heeger bergen 1995 algorithm
Heeger Bergen 1995 - Algorithm
  • Image pyramids
    • Gaussian
    • Laplacian
  • Steerable pyramids [SimoncelliFreeman95]
    • b): multiple scales of oriented filters
    • c): a sample image
    • d): results of filters in b) applied to c)
heeger bergen 1995 results

I

I

J

J

Heeger Bergen 1995 - Results

Successes

Failures

heeger bergen 1995 verdict
Heeger Bergen 1995 - Verdict
  • Texture model:
    • Histograms of responses to various filters
  • Avoiding copying:
    • Inherent in algorithm
  • No user intervention required
  • Captures stochastic textures well
  • Does not capture structure
    • Lack of inter-scale constraints
de bonet 1997
De Bonet 1997
  • Propagate constraints downwards by matching statistics all the way up the pyramid
  • Feature vector:multiscale collection of filter responses for a given pixel
  • Algorithm:
    • Initialize J to empty image
    • Create multiresolution pyramids for I and J
    • For each pixel in level of J, randomly choose pixel from corresponding level of I that has similar feature vector
de bonet 1997 algorithm
De Bonet 1997 - Algorithm
  • 6 feature vectors shown
  • Notice how they share parent information
de bonet 1997 verdict
De Bonet 1997 - Verdict
  • Texture model:
    • Feature vector containing multiscale responses to various filters
  • Avoiding copying:
    • Random choice of pixels with ‘close’ feature vectors, but copying still frequent on small scale
  • Individual per-filter thresholds are cumbersome
  • Feature vectors used in later synthesis work
efros leung 1999
Efros Leung 1999
  • Markov Random Fields: pixels synthesized probabilistically based on causal neighborhood
  • Algorithm:
    • Initialize J to empty
    • Synthesize new pixels in a spiral pattern
    • Select new pixel from I based on conditional probabilities given the current neighborhood in J and similar neighborhoods in I
efros leung 1999 algorithm
Efros Leung 1999 - Algorithm

I

J

‘causal’

neighborhood

efros leung 1999 verdict
Efros Leung 1999 – Verdict
  • Texture model:
    • MRF
  • Avoiding copying:
    • MRF
  • Neighborhood size = largest feature size
  • Markov model is surprisingly good
    • “I spent an interesting evening recently with a grain of salt.”
  • Search is very slow with large neighborhoods
wei levoy 2000
Wei Levoy 2000
  • Adds tree-structured vector quantization speedup and multiresolution matching
  • Algorithm:
    • Initialize J to noise
    • Synthesize new pixels in scanline order
    • Select new pixel from I that has the closest matching neighborhood to J
    • In multiresolution case, look instead at neighborhood of feature vectors
wei levoy 2000 algorithm
Wei Levoy 2000 - Algorithm
  • Scanline synthesis with neighborhoods
    • a) input texture
    • b) algorithm start
    • c) midway point
    • d) ending
wei levoy 2000 verdict
Wei Levoy 2000 - Verdict
  • Texture model:
    • Multiscale filters and MRF
  • Avoiding copying:
    • Not explicitly, but in practice inherently avoided; TSVQ adds additional unpredictability
  • Results comparable to [EfrosLeung99]
  • TSVQ speedup comes at a price (quality loss)
ashikhmin 2001
Ashikhmin 2001
  • Adds more verbatim image copying and user-constrained synthesis
  • Algorithm:
    • New array J* stores location in I of J’s pixels
    • Initialize J to empty, J* to random
    • Synthesize new pixels in scanline order
    • Select new pixel by copying from patches of I, as determined by locations in J*
    • In user-constrained case, also include error comparison with user-specified target image T
ashikhmin 2001 algorithm
Ashikhmin 2001 - Algorithm
  • Candidates:
  • x=4, y=5
  • x=18, y=16
  • x=11, y=12

x=3

y=4

x=4

y=4

x=12

y=11

x=17

y=16

x=18

y=16

J*

ashikhmin 2001 verdict
Ashikhmin 2001 - Verdict
  • Texture model:
    • MRF
  • Avoiding copying:
    • Actually, here it is encouraged on a small scale, but in practice it doesn’t occur on a large scale
  • Recognized that copying preserves fine detail
  • Features can get ‘cut off’ because of the abrupt jumping from patch to patch
hertzmann et al 2001
Hertzmann et. al. 2001
  • Framework for solving image analogies: learning an image filter f by example
    • Given A, f(A), and B, generate f(B)
  • To handle texture synthesis, ignore A, B, and let f(A) = I, f(B) = J
  • Algorithm:
    • Bring image histograms into correspondence
    • Combine [WeiLevoy00] and [Ashikhmin01] Each search method is weighted appropriately and then the best choice is used
hertzmann et al 2001 algorithm
Hertzmann et. al. 2001 - Algorithm
  • A provides set of constraints
  • Histogram matching critical because of color differences between A and B sets of images
hertzmann et al 2001 verdict
Hertzmann et. al. 2001 - Verdict
  • Texture model:
    • Multiscale filters and MRF
  • Avoiding copying:
    • See [WeiLevoy00] and [Ashikhmin01]
  • Unique approach to image manipulation and texture synthesis
  • Same fundamental limitations as [WeiLevoy00] and [Ashikhmin01]
efros freeman 2001
Efros Freeman 2001
  • Inspired by chaos mosaics [Xu et. al. 00]
  • Make an ‘image quilt’ by pasting together patches of the input and fixing seams
  • Algorithm:
    • Initialize J to empty
    • Copy new patches of certain size in scanline order, with fixed overlap width
    • Randomly select new patch that has error in overlap region less than given threshold
    • Copy pixels based on minimal error boundary cut
efros freeman 2001 algorithm

B1

B1

B2

B2

Neighboring blocks

constrained by overlap

Minimal error

boundary cut

Efros Freeman 2001 - Algorithm

Input texture

block

efros freeman 2001 verdict
Efros Freeman 2001 - Verdict
  • Texture model:
    • ??
  • Avoiding copying:
    • Randomized patch selection, but still noticeable
  • Patch size is a hard parameter to understand
  • Results are surprisingly good given algorithm
  • Multiscale goes on a brief hiatus
kwatra et al 2003
Kwatra et. al. 2003
  • Generalizes seam computation in overlap regions as a graph cut problem
    • Based on [Boykov et. al. 99] (with Ramin Zabih)
  • Algorithm:
    • Initialize J to empty
    • Copy pieces of I to J using a variety of methods
    • Formulate flow graph in overlap region based on errors and compute minimum cut
    • Copy sink-side pixels to J
    • Variety of strategies to further hide seams
kwatra et al 2003 algorithm
Kwatra et. al. 2003 - Algorithm

(assume cut region is 3x3 for simplicity)

kwatra et al 2003 verdict
Kwatra et. al. 2003 - Verdict
  • Texture model:
    • MRF
  • Avoiding copying:
    • Even with a multitude of patch selection methods, still noticeable when it happens repeatedly
  • Paper presents a bag of synthesis tricks without much intuition for when to use what
  • Graph cut formalization is useful and powerful
wu yu 2004
Wu Yu 2004
  • Synthesizes a feature image J* in parallel
  • Tries to match features in overlap region
  • Algorithm:
    • Compute feature image I*
    • Initialize J* to empty
    • Synthesize J and J* in parallel
    • When searching for new patch, add in weighted feature matching error
    • Prior to copying, warp selected patch to match features exactly
wu yu 2004 algorithm
Wu Yu 2004 - Algorithm
  • Match feature pixels based on
    • Tangent orientation
    • Distance
    • Bijectivity
  • Example:
    • L1out matches L1in
    • L2out matches L3in
    • Warping will move matching features closer to each other

Feature correspondence

wu yu 2004 verdict
Wu Yu 2004: Verdict
  • Texture model:
    • Feature lines and MRF
  • Avoiding copying:
    • No; in fact, mirroring is often very noticeable
  • Unclear how to combine this metric with others
  • Insight to use warping is valuable, although the next paper does a much better job with it
liu et al 2004
Liu et. al. 2004
  • Regular textures: can be tiled
  • Near-regular textures: statistical distortion of a regular texture
  • Algorithm (simplified):
    • Mark lattice points in I
    • Transform I into regular texture R
    • Express transformation as deformation texture I*
    • Synthesize larger deformation J*
    • Synthesize J by combining J* with R
liu et al 2004 algorithm
Liu et. al. 2004 - Algorithm

I

R

Deformation Texture I*

liu et al 2004 results1
Liu et. al. 2004 - Results

texture transfer

original

liu et al 2004 verdict
Liu et. al. 2004 - Verdict
  • Texture model:
    • Regular texture with modulated geometry and color
  • Avoiding copying:
    • Depends on the synthesis algorithm
  • Proposes specific model to handle specific types of textures
  • Interestingly, this method depends on previous texture synthesis methods
classes of algorithms recap
Classes of Algorithms - Recap
  • Multiresolution pyramids
    • [HeegerBergen95]
  • Pixel-by-pixel synthesis
    • [EfrosLeung99]
  • Multiresolution pixel-by-pixel
    • [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01], [Ashikhmin01]
  • Patch quilting
    • [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04]
  • Geometric feature matching
    • [WuYu04], [Liu et.al. 04]
evolution of texture model
Evolution of Texture Model
  • Responses to image filters / decompositions
  • Markov Random Fields
  • Copies of small pieces of image
  • Feature data
  • Various combinations of the above
what s next

=

+

What’s Next?
  • Texture synthesis is really hard
  • Rather than creating a silver bullet, focus on algorithms for more specific problems
  • Superresolution:

???

what s next1

=

+

What’s Next?
  • Texture synthesis is really hard
  • Rather than creating a silver bullet, focus on algorithms for more specific problems
  • Superresolution:

[Hertzmann et. al. 01]