Loading in 5 sec....

Overview of Texture SynthesisPowerPoint Presentation

Overview of Texture Synthesis

- 135 Views
- Uploaded on
- Presentation posted in: General

Download Presentation
## PowerPoint Slideshow about 'Overview of Texture Synthesis ' - dixon

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

The Problem … in Words

- Given texture I, generate a texture J which
- Looks like the same texture
- Has no obvious copying or tiling from I
- Difference between I and J should be the same as the way I “differs from itself” [DeBonet97]

- Things to watch for:
- ‘Looks the same’: what is the texture model?
- ‘Obvious copying’: how is it avoided?
- Underlined text: indicates algorithm parameter

Classes of Algorithms

- Multiresolution pyramids
- [HeegerBergen95]

- Pixel-by-pixel synthesis
- [EfrosLeung99]

- Multiresolution pixel-by-pixel
- [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01], [Ashikhmin01]

- Patch quilting
- [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04]

- Geometric feature matching
- [WuYu04], [Liu et.al. 04]

Heeger Bergen 1995

- Seminal paper that introduced texture synthesis to the graphics community
- Algorithm:
- Initialize J to noise
- Create multiresolution pyramids for I and J
- Match the histograms of J’s pyramid levels with I’s pyramid levels
- Loop until convergence
- Can be generalized to 3D

Heeger Bergen 1995 - Algorithm

- Image pyramids
- Gaussian
- Laplacian

- Steerable pyramids [SimoncelliFreeman95]
- b): multiple scales of oriented filters
- c): a sample image
- d): results of filters in b) applied to c)

Heeger Bergen 1995 - Verdict

- Texture model:
- Histograms of responses to various filters

- Avoiding copying:
- Inherent in algorithm

- No user intervention required
- Captures stochastic textures well
- Does not capture structure
- Lack of inter-scale constraints

De Bonet 1997

- Propagate constraints downwards by matching statistics all the way up the pyramid
- Feature vector:multiscale collection of filter responses for a given pixel
- Algorithm:
- Initialize J to empty image
- Create multiresolution pyramids for I and J
- For each pixel in level of J, randomly choose pixel from corresponding level of I that has similar feature vector

De Bonet 1997 - Algorithm

- 6 feature vectors shown
- Notice how they share parent information

De Bonet 1997 - Verdict

- Texture model:
- Feature vector containing multiscale responses to various filters

- Avoiding copying:
- Random choice of pixels with ‘close’ feature vectors, but copying still frequent on small scale

- Individual per-filter thresholds are cumbersome
- Feature vectors used in later synthesis work

Efros Leung 1999

- Markov Random Fields: pixels synthesized probabilistically based on causal neighborhood
- Algorithm:
- Initialize J to empty
- Synthesize new pixels in a spiral pattern
- Select new pixel from I based on conditional probabilities given the current neighborhood in J and similar neighborhoods in I

Efros Leung 1999 – Verdict

- Texture model:
- MRF

- Avoiding copying:
- MRF

- Neighborhood size = largest feature size
- Markov model is surprisingly good
- “I spent an interesting evening recently with a grain of salt.”

- Search is very slow with large neighborhoods

Wei Levoy 2000

- Adds tree-structured vector quantization speedup and multiresolution matching
- Algorithm:
- Initialize J to noise
- Synthesize new pixels in scanline order
- Select new pixel from I that has the closest matching neighborhood to J
- In multiresolution case, look instead at neighborhood of feature vectors

Wei Levoy 2000 - Algorithm

- Scanline synthesis with neighborhoods
- a) input texture
- b) algorithm start
- c) midway point
- d) ending

Wei Levoy 2000 - Verdict

- Texture model:
- Multiscale filters and MRF

- Avoiding copying:
- Not explicitly, but in practice inherently avoided; TSVQ adds additional unpredictability

- Results comparable to [EfrosLeung99]
- TSVQ speedup comes at a price (quality loss)

Ashikhmin 2001

- Adds more verbatim image copying and user-constrained synthesis
- Algorithm:
- New array J* stores location in I of J’s pixels
- Initialize J to empty, J* to random
- Synthesize new pixels in scanline order
- Select new pixel by copying from patches of I, as determined by locations in J*
- In user-constrained case, also include error comparison with user-specified target image T

Ashikhmin 2001 - Algorithm

- Candidates:
- x=4, y=5
- x=18, y=16
- x=11, y=12

x=3

y=4

x=4

y=4

x=12

y=11

x=17

y=16

x=18

y=16

J*

Ashikhmin 2001 - Verdict

- Texture model:
- MRF

- Avoiding copying:
- Actually, here it is encouraged on a small scale, but in practice it doesn’t occur on a large scale

- Recognized that copying preserves fine detail
- Features can get ‘cut off’ because of the abrupt jumping from patch to patch

Hertzmann et. al. 2001

- Framework for solving image analogies: learning an image filter f by example
- Given A, f(A), and B, generate f(B)

- To handle texture synthesis, ignore A, B, and let f(A) = I, f(B) = J
- Algorithm:
- Bring image histograms into correspondence
- Combine [WeiLevoy00] and [Ashikhmin01] Each search method is weighted appropriately and then the best choice is used

Hertzmann et. al. 2001 - Algorithm

- A provides set of constraints
- Histogram matching critical because of color differences between A and B sets of images

Hertzmann et. al. 2001 - Verdict

- Texture model:
- Multiscale filters and MRF

- Avoiding copying:
- See [WeiLevoy00] and [Ashikhmin01]

- Unique approach to image manipulation and texture synthesis
- Same fundamental limitations as [WeiLevoy00] and [Ashikhmin01]

Efros Freeman 2001

- Inspired by chaos mosaics [Xu et. al. 00]
- Make an ‘image quilt’ by pasting together patches of the input and fixing seams
- Algorithm:
- Initialize J to empty
- Copy new patches of certain size in scanline order, with fixed overlap width
- Randomly select new patch that has error in overlap region less than given threshold
- Copy pixels based on minimal error boundary cut

B1

B2

B2

Neighboring blocks

constrained by overlap

Minimal error

boundary cut

Efros Freeman 2001 - AlgorithmInput texture

block

Efros Freeman 2001 - Verdict

- Texture model:
- ??

- Avoiding copying:
- Randomized patch selection, but still noticeable

- Patch size is a hard parameter to understand
- Results are surprisingly good given algorithm
- Multiscale goes on a brief hiatus

Kwatra et. al. 2003

- Generalizes seam computation in overlap regions as a graph cut problem
- Based on [Boykov et. al. 99] (with Ramin Zabih)

- Algorithm:
- Initialize J to empty
- Copy pieces of I to J using a variety of methods
- Formulate flow graph in overlap region based on errors and compute minimum cut
- Copy sink-side pixels to J
- Variety of strategies to further hide seams

Kwatra et. al. 2003 - Algorithm

(assume cut region is 3x3 for simplicity)

Kwatra et. al. 2003 - Verdict

- Texture model:
- MRF

- Avoiding copying:
- Even with a multitude of patch selection methods, still noticeable when it happens repeatedly

- Paper presents a bag of synthesis tricks without much intuition for when to use what
- Graph cut formalization is useful and powerful

Wu Yu 2004

- Synthesizes a feature image J* in parallel
- Tries to match features in overlap region
- Algorithm:
- Compute feature image I*
- Initialize J* to empty
- Synthesize J and J* in parallel
- When searching for new patch, add in weighted feature matching error
- Prior to copying, warp selected patch to match features exactly

Wu Yu 2004 - Algorithm

- Match feature pixels based on
- Tangent orientation
- Distance
- Bijectivity

- Example:
- L1out matches L1in
- L2out matches L3in
- Warping will move matching features closer to each other

Feature correspondence

Wu Yu 2004: Verdict

- Texture model:
- Feature lines and MRF

- Avoiding copying:
- No; in fact, mirroring is often very noticeable

- Unclear how to combine this metric with others
- Insight to use warping is valuable, although the next paper does a much better job with it

Liu et. al. 2004

- Regular textures: can be tiled
- Near-regular textures: statistical distortion of a regular texture
- Algorithm (simplified):
- Mark lattice points in I
- Transform I into regular texture R
- Express transformation as deformation texture I*
- Synthesize larger deformation J*
- Synthesize J by combining J* with R

Liu et. al. 2004 - Verdict

- Texture model:
- Regular texture with modulated geometry and color

- Avoiding copying:
- Depends on the synthesis algorithm

- Proposes specific model to handle specific types of textures
- Interestingly, this method depends on previous texture synthesis methods

Classes of Algorithms - Recap

- Multiresolution pyramids
- [HeegerBergen95]

- Pixel-by-pixel synthesis
- [EfrosLeung99]

- Multiresolution pixel-by-pixel
- [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01], [Ashikhmin01]

- Patch quilting
- [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04]

- Geometric feature matching
- [WuYu04], [Liu et.al. 04]

Evolution of Texture Model

- Responses to image filters / decompositions
- Markov Random Fields
- Copies of small pieces of image
- Feature data
- Various combinations of the above

+

What’s Next?- Texture synthesis is really hard
- Rather than creating a silver bullet, focus on algorithms for more specific problems
- Superresolution:

???

+

What’s Next?- Texture synthesis is really hard
- Rather than creating a silver bullet, focus on algorithms for more specific problems
- Superresolution:

[Hertzmann et. al. 01]