slide1 n.
Skip this Video
Loading SlideShow in 5 Seconds..
Presentation by Gyozo Gidofalvi Computer Science and Engineering Department PowerPoint Presentation
Download Presentation
Presentation by Gyozo Gidofalvi Computer Science and Engineering Department

Loading in 2 Seconds...

play fullscreen
1 / 49

Presentation by Gyozo Gidofalvi Computer Science and Engineering Department - PowerPoint PPT Presentation

  • Uploaded on

Texture Synthesis by Non-parametric Sampling / Image Quilting for Texture Synthesis & Transfer by Efros and Leung / Efros and Freeman ICCV ’99 / SIGGRAPH ’01. Presentation by Gyozo Gidofalvi Computer Science and Engineering Department University of California, San Diego

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Presentation by Gyozo Gidofalvi Computer Science and Engineering Department' - zelda

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Texture Synthesis by Non-parametric Sampling / Image Quiltingfor Texture Synthesis & TransferbyEfros and Leung / Efros and Freeman ICCV ’99 / SIGGRAPH ’01

Presentation by Gyozo Gidofalvi

Computer Science and Engineering Department

University of California, San Diego

November 15, 2001

  • Background information on texture
  • Growing texture pixel-by-pixel using a simple non-parametric method
  • Image quilting: a very simple texture synthesis algorithm
  • Simple extension to image quilting for texture transfer
  • Applications
  • Summary
what is a texture
What is a texture?
  • “What features and statistics are characteristics of a texture pattern, so that texture pairs that share the same features and statistics cannot be told apart by pre-attentive human visual perception?” ---- Julesz 1960s-1980s
  • The concept of texture is intuitively obvious but has no precise definition: something consisting of mutually related elements.
  • On can describe texture by: tone and structure
    • Tone is based on pixel intensity properties
    • Structure describes spatial relationships of primitives
  • Texture can be described by the number and types of primitives and by their spatial relationships.
texture analysis techniques
Texture Analysis Techniques
  • Statistical methods: gather information about textures by exploiting pixel first and second order statistics.
  • Structural methods: describe textures as composed of well defined texture primitives (texels), which are placed according to some syntactic rules.
  • Model based methods: construct a generative or stochastic model of textures, called random field models.
  • Transform methods: represent an image in a new form, in which the characteristics of the texture become more easily accessible. Some examples include: Fourier-transforms and multi-resolution methods.
what is texture synthesis
What is texture synthesis?

Given an input sample texture synthesize a texture that is sufficiently different from the given sample texture, yet appears perceptually to be generated by the same underlying stochastic process.

input image


True (infinite) texture

generated image

classification of texture
Classification of texture
  • Traditionally textures has been classified as:
    • regular : repeated texels
    • stochastic without explicit texels
  • Most real world textures are mixtures of these basic types
  • Challenge is to model the whole spectrum: from regular to stochastic texture




some previous approaches
Some previous approaches
  • multi-scale filter response histogram matching [Heeger and Bergen,’95]
  • sampling from conditional distribution over multiple scales [DeBonet,’97]
  • filter histograms with Gibbs sampling [Zhu et al,’98]
  • matching 1st and 2nd order properties of wavelet coefficients [Simoncelli and Portilla,’98]
new method by efros et al
New method by Efros et al.
  • goals:
    • preserve local structure
    • model wide range of real textures
    • ability to do constrained synthesis
  • method:
    • inspired by N-gram language model of Shannon, texture is modelled as Markov Random Field (MRF)
    • texture is “grown” one pixel at a time
    • conditional pdf of a pixel given its neighbors synthesized thus far is estimated by searching the the sample image for similar neighborhoods
n gram model of the english language
N-gram model of the English language
  • Shannon: Model language as a generalized Markov chain, where a set of n letters (words) completely determine the pdf of the next letter (word).
  • Results (using corpus) [Mark V. Shaney]:
  • "One morning I shot an elephant in my arms and kissed him.”
  • "I spent an interesting evening recently with a grain of salt“
  • Assuming Markov property, texture can be modeled as a MRF
synthesizing one pixel
Synthesizing one pixel



finite sample image

Generated image

To synthesize a pixel p, search the sample image for pixels with similar neighborhood to p, construct a histogram for the distribution of these pixels, finally sample this distribution to obtain a value for p.

Similarity is based on the Gaussian-weighted sum squared difference, to preserve local structure.

growing texture on pixel at the time
Growing texture on pixel at the time
  • User defined window size indicates the randomness of the texture
  • To grow from from scratch a 3x3 random seed from the sample is used
  • Unless no close match is found pixels with most neighbors are synthesized first
  • Importance of Gaussian-weighted similarity measure
more synthesis results
More Synthesis Results

Increasing window size


reptile skin

aluminium wire

more results
More results

French canvas

rafia weave

more results1
More results



more results2
More results

brick wall

white bread

visual comparison
Visual comparison

Synthetic tilable


[DeBonet, ‘97]

Simple tiling

Our approach

failure cases
Failure cases

Growing garbage

Verbatim copying

what we have so far
What we have so far…
  • An algorithm that is:
    • simple
    • models a wide range of real-world textures
    • naturally well-suited for constrained texture synthesis
  • but it:
    • very slow
    • sometimes “grows garbage”
  • How can it be improved?
why only synthesize on pixel at the time
Why only synthesize on pixel at the time?
  • For most complex textures only a very few pixels actually have a choice of values wasted search effort
  • Example: Pattern of circles on the plane
    • Once the algorithm starts synthesizing a particular circle, the values of the remaining pixels are completely determined.
  • Unit of synthesis should be more than just a pixel patch
  • Texture synthesis would be like jigsaw puzzle
  • Questions:
    • What are the patches?
    • How to put them together?
chaos mosaic xu guo shum 00
Chaos Mosaic [Xu, Guo & Shum, ‘00]




  • Process: 1) tile input image; 2) pick random blocks and place them in random locations 3) Smooth edges

Used in Lapped Textures [Praun,’00]


Chaos Mosaic [Xu, Guo & Shum, ‘00]



The approach works well on stochastic textures but fails on structures.

image quilting



Synthesizing a block

  • Idea: let’s combine random block placement of Chaos Mosaic with spatial constraints of Efros & Leung
  • Unit of synthesis is a block
  • Exactly the same but now we want P(B|N(B))
  • Much faster: synthesize all pixels in a block at once
Image Quilting



Input image






Neighboring blocks

constrained by overlap

Minimal error

boundary cut


Input texture



Random placement

of blocks

minimal error boundary




overlap error

min. error boundary

Minimal error boundary

overlapping blocks

vertical boundary

image quilting algorithm
Image Quilting algorithm
  • Pick size of block and size of overlap
  • Synthesize blocks in raster order
  • Search input texture for block that satisfies overlap constraints (above and left)

Easy to optimize using NN search [Liang, ’01]

  • Paste new block into resulting texture

use dynamic programming to compute minimal error boundary cut



Portilla & Simoncelli

Xu, Guo & Shum

input image

Wei & Levoy

Image Quilting



Portilla & Simoncelli

Xu, Guo & Shum

input image

Wei & Levoy

Image Quilting


Homage to Shannon

Portilla & Simoncelli

Xu, Guo & Shum

input image

Wei & Levoy

Image Quilting





texture transfer
Texture transfer

Take the texture from one object and “paint” it onto another object

  • This requires separating texture and shape
  • That’s HARD, but we can cheat
  • Assume we can capture shape by boundary and rough shading

Idea: just add another constraint when sampling: similarity to underlying image at that spot

Correspondence can be based on: image intensity, blured image intensity, local image orientation angles, etc…

There is a tradeoff between the legitimacy of synthesized texture and the correctness of the correspondence mapping.













correspondence image



Target image




applications of texture synthesis and transfer
Applications of texture synthesis and transfer
  • Occlusion fill-in
    • for 3D reconstruction
  • region-based image and video compression
    • a small sample of textured region is stored
  • Texturing non-developable objects
    • growing texture directly on surface
  • Motion synthesis
  • Synthesizing and transferring music and environmental sounds?
  • Rendering object in a different style without explicit 3D information

We have seen:

  • A non-parametric method that grows texture pixel-by-pixel based on neighborhood statistics of pixels gathered from a sample texture
    • Conceptually simple
    • Spatial locality principle lends itself to constraint synthesis
  • An image based method that where images are synthesized by stitching together patches of existing images
  • A simple method based on similarity obtained by correspondence for texture transfer and rendering texture to objects that does not need explicit 3D information

Alexei Efros and his fellow researchers for sharing their slides.

references and related literature
References and Related Literature

A. A. Efros and W. T. Freeman. Image Quilting for Texture Synthesis and Transfer, SIGGRAPH01.

A. A. Efros and T. K. Leung. Texture Synthesis by Non-parametric Sampling. In ICCV99.

S. Livens. Image Analysis for Material Characterization, 1998.

C. E. Shannon, A mathematical theory of computation. Bell Sys. Tech. Journal, 27, 1948.

M. Ashikhmin. Synthesizing natural textures. In Symposium on Interactive 3D Graphics, 2001.

J. S. De Bonet. Multiresolution sampling procedure for analysis and synthesis of texture images. In SIGGRAPH 97, pages 361-368, 1997.

D. D. Garber. Computational Models for Texture Analysis and Texture Synthesis. PhD thesis, University of Southern California, Image Processing Institute, 1981.

P. Harrison. A non-hierarchical procedure for re-synthesis of complex textures. In WSCG 2001 Conference Proceedings, pages 190-197, 2001.

D. J. Heeger and J. R. Bergen. Pyramid based texture analysis/synthesis. In SIGGRAPH 95, pages 229-238, 1995.

A Hertzmann, C. E. Jacobs, D. Oliver, B. Curless, and D. H. Salesin. Image analogies. In SIGGRAPH 01, 2001.

L. Liang , C. Liu, Y, Xu, B. Guo, and H.-Y. Shum. Real-time texture synthesis by patch-based sampling. Technical Report MSR-TR-2001-40, Microsoft Research, March 2001.

Y. XU, B. Guo, and H.-Y. Shum. Chaos Mosaic: Fast and memory efficient texture synthesis. Technical Report MSR-TR-2000-32, Microsoft Research, April 2000.