slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Advanced Multimedia PowerPoint Presentation
Download Presentation
Advanced Multimedia

Loading in 2 Seconds...

play fullscreen
1 / 85

Advanced Multimedia - PowerPoint PPT Presentation


  • 141 Views
  • Uploaded on

Advanced Multimedia. Warping & Morphing Tamara Berg. Image Warping. http://www.jeffrey-martin.com. Slides from: Alexei Efros & Steve Seitz. Image Warping in Biology. D'Arcy Thompson http://www-groups.dcs.st-and.ac.uk/~history/Miscellaneous/darcy.html

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

Advanced Multimedia


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Advanced Multimedia

Warping & Morphing

Tamara Berg

image warping
Image Warping

http://www.jeffrey-martin.com

Slides from: Alexei Efros & Steve Seitz

image warping in biology
Image Warping in Biology
  • D'Arcy Thompson
  • http://www-groups.dcs.st-and.ac.uk/~history/Miscellaneous/darcy.html
  • http://en.wikipedia.org/wiki/D'Arcy_Thompson
  • Importance of shape and structure in evolution

Slide by Durand and Freeman

image transformations

f

f

f

T

T

x

x

x

f

x

Image Transformations
  • image filtering: change range of image
  • g(x) = T(f(x))

image warping: change domain of image

g(x) = f(T(x))

image transformations1

T

T

Image Transformations
  • image filtering: change range of image
  • g(x) = T(f(x))

f

g

image warping: change domain of image

g(x) = f(T(x))

f

g

parametric global warping
Parametric (global) warping
  • Examples of parametric warps:

aspect

rotation

translation

perspective

cylindrical

affine

parametric global warping1

T

Parametric (global) warping
  • Transformation T is a coordinate-changing machine:
  • p’ = T(p)
  • What does it mean that T is global?
    • Is the same for any point p
    • can be described by just a few numbers (parameters)
  • Let’s represent T as a matrix:
  • p’ = Mp

p = (x,y)

p’ = (x’,y’)

scaling
Scaling
  • Scaling a coordinate means multiplying each of its components by a scalar
  • Uniform scaling means this scalar is the same for all components:

2

scaling1

X  2,Y  0.5

Scaling
  • Non-uniform scaling: different scalars per component:
scaling2
Scaling
  • Scaling operation:
  • Or, in matrix form:

scaling matrix S

What’s inverse of S?

2 d rotation

(x’, y’)

(x, y)

2-D Rotation

x’ = x cos() - y sin()

y’ = x sin() + y cos()

2 d rotation1

(x’, y’)

(x, y)

2-D Rotation

x = r cos (f)

y = r sin (f)

x’ = r cos (f + )

y’ = r sin (f + )

f

2 d rotation2

(x’, y’)

(x, y)

2-D Rotation

x = r cos (f)

y = r sin (f)

x’ = r cos (f + )

y’ = r sin (f + )

Trig Identity…

x’ = r cos(f) cos() – r sin(f) sin()

y’ = r sin(f) cos() + r cos(f) sin()

f

2 d rotation3

(x’, y’)

(x, y)

2-D Rotation

x = r cos (f)

y = r sin (f)

x’ = r cos (f + )

y’ = r sin (f + )

Trig Identity…

x’ = r cos(f) cos() – r sin(f) sin()

y’ = r sin(f) cos() + r cos(f) sin()

Substitute…

x’ = x cos() - y sin()

y’ = x sin() + y cos()

f

2 d rotation4
2-D Rotation
  • This is easy to capture in matrix form:
  • What is the inverse transformation?

R

2 d rotation5
2-D Rotation
  • This is easy to capture in matrix form:
  • What is the inverse transformation?
    • Rotation by –q
    • For rotation matrices

R

2x2 matrices
2x2 Matrices
  • What types of transformations can be represented with a 2x2 matrix?

2D Identity?

2x2 matrices1
2x2 Matrices
  • What types of transformations can be represented with a 2x2 matrix?

2D Identity?

2x2 matrices2
2x2 Matrices
  • What types of transformations can be represented with a 2x2 matrix?

2D Identity?

2D Scale around (0,0)?

2x2 matrices3
2x2 Matrices
  • What types of transformations can be represented with a 2x2 matrix?

2D Identity?

2D Scale around (0,0)?

2x2 matrices4
2x2 Matrices
  • What types of transformations can be represented with a 2x2 matrix?

2D Rotate around (0,0)?

2D Shear?

2x2 matrices5
2x2 Matrices
  • What types of transformations can be represented with a 2x2 matrix?

2D Mirror about Y axis?

2x2 matrices6
2x2 Matrices
  • What types of transformations can be represented with a 2x2 matrix?

2D Mirror about Y axis?

2x2 matrices7
2x2 Matrices
  • What types of transformations can be represented with a 2x2 matrix?

2D Mirror about Y axis?

2D Mirror over (0,0)?

2x2 matrices8
2x2 Matrices
  • What types of transformations can be represented with a 2x2 matrix?

2D Mirror about Y axis?

2D Mirror over (0,0)?

2x2 matrices9
2x2 Matrices
  • What types of transformations can be represented with a 2x2 matrix?

2D Translation?

2x2 matrices10
2x2 Matrices
  • What types of transformations can be represented with a 2x2 matrix?

2D Translation?

NO!

Only linear 2D transformations

can be represented with a 2x2 matrix

all 2d linear transformations
All 2D Linear Transformations
  • Linear transformations are combinations of …
    • Scale,
    • Rotation,
    • Shear, and
    • Mirror
  • Properties of linear transformations:
    • Origin maps to origin
    • Lines map to lines
    • Parallel lines remain parallel
    • Ratios are preserved
    • Closed under composition
homogeneous coordinates
Homogeneous Coordinates
  • Q: How can we represent translation as a 3x3 matrix?
homogeneous coordinates1
Homogeneous Coordinates
  • Homogeneous coordinates
    • represent coordinates in 2 dimensions with a 3-vector
homogeneous coordinates2

y

2

(2,1,1)

or (4,2,2)

or (6,3,3)

1

x

2

1

Homogeneous Coordinates
  • Add a 3rd coordinate to every 2D point
    • (x, y, w) represents a point at location (x/w, y/w)
    • (x, y, 0) represents a point at infinity
    • (0, 0, 0) is not allowed

Convenient coordinate system to represent many useful transformations

homogeneous coordinates3
Homogeneous Coordinates
  • Q: How can we represent translation as a 3x3 matrix?
homogeneous coordinates4
Homogeneous Coordinates
  • Q: How can we represent translation as a 3x3 matrix?
  • A: Using the rightmost column:
translation
Translation
  • Example of translation

Homogeneous Coordinates

tx = 2ty= 1

basic 2d transformations
Basic 2D Transformations
  • Basic 2D transformations as 3x3 matrices

Translate

Scale

Rotate

Shear

matrix composition
Matrix Composition
  • Transformations can be combined by matrix multiplication

p’ = T(tx,ty) R(Q) S(sx,sy) p

affine transformations
Affine Transformations
  • Affine transformations are combinations of …
    • Linear transformations, and
    • Translations
  • Properties of affine transformations:
    • Origin does not necessarily map to origin
    • Lines map to lines
    • Parallel lines remain parallel
    • Ratios are preserved
    • Closed under composition
    • Models change of basis
projective transformations
Projective Transformations
  • Projective transformations …
    • Affine transformations, and
    • Projective warps
  • Properties of projective transformations:
    • Origin does not necessarily map to origin
    • Lines map to lines
    • Parallel lines do not necessarily remain parallel
    • Ratios are not preserved
    • Closed under composition
    • Models change of basis
2d image transformations
2D image transformations
  • These transformations are a nested set of groups
    • Closed under composition and inverse is a member
recovering transformations
Recovering Transformations
  • What if we know f and g and want to recover the transform T?
    • willing to let user provide correspondences
      • How many do we need?

?

T(x,y)

y

y’

x

x’

f(x,y)

g(x’,y’)

translation correspondences
Translation: # correspondences?
  • How many correspondences needed for translation?

?

T(x,y)

y

y’

x

x’

translation correspondences1
Translation: # correspondences?
  • How many correspondences needed for translation?
  • How many Degrees of Freedom?

?

T(x,y)

y

y’

x

x’

translation correspondences2
Translation: # correspondences?
  • How many correspondences needed for translation?
  • How many Degrees of Freedom?
  • What is the transformation matrix?

?

T(x,y)

y

y’

x

x’

euclidian correspondences
Euclidian: # correspondences?
  • How many correspondences needed for translation+rotation?
  • How many DOF?

?

T(x,y)

y

y’

x

x’

affine correspondences
Affine: # correspondences?
  • How many correspondences needed for affine?
  • How many DOF?

?

T(x,y)

y

y’

x

x’

projective correspondences
Projective: # correspondences?
  • How many correspondences needed for projective?
  • How many DOF?

?

T(x,y)

y

y’

x

x’

example warping triangles
Example: warping triangles

B’

  • Given two triangles: ABC and A’B’C’ in 2D (12 numbers)
  • Need to find transform T to transfer all pixels from one to the other.
  • What kind of transformation is T?
  • How can we compute the transformation matrix:

B

?

T(x,y)

C’

A

C

A’

Source

Destination

image warping1
Image warping
  • Given a coordinate transform (x’,y’)=T(x,y) and a source image f(x,y), how do we compute a transformed image g(x’,y’)=f(T(x,y))?

T(x,y)

y

y’

x

x’

f(x,y)

g(x’,y’)

forward warping
Forward warping
  • Send each pixel f(x,y) to its corresponding location
  • (x’,y’)=T(x,y) in the second image

T(x,y)

y

y’

x

x’

f(x,y)

g(x’,y’)

Q: what if pixel lands “between” two pixels?

forward warping1
Forward warping
  • Send each pixel f(x,y) to its corresponding location
  • (x’,y’)=T(x,y) in the second image

T(x,y)

y

y’

x

x’

f(x,y)

g(x’,y’)

Q: what if pixel lands “between” two pixels?

  • A: distribute color among neighboring pixels (x’,y’)
    • Known as “splatting”
    • Check out griddata in Matlab
inverse warping

T-1(x,y)

Inverse warping
  • Get each pixel g(x’,y’) from its corresponding location
  • (x,y)=T-1(x’,y’) in the first image

y

y’

x

x

x’

f(x,y)

g(x’,y’)

Q: what if pixel comes from “between” two pixels?

inverse warping1

T-1(x,y)

Inverse warping
  • Get each pixel g(x’,y’) from its corresponding location
  • (x,y)=T-1(x’,y’) in the first image

y

y’

x

x

x’

f(x,y)

g(x’,y’)

Q: what if pixel comes from “between” two pixels?

  • A: Interpolate color value from neighbors
    • nearest neighbor, bilinear, Gaussian, bicubic
    • Check out interp2 in Matlab
forward vs inverse warping
Forward vs. inverse warping
  • Q: which is better?
  • A: usually inverse—eliminates holes
    • however, it requires an invertible warp function—not always possible...
homework 3a
Homework 3A

On course webpage. Due Tues, March 31. Start now and come ask me questions asap. My solution is < 20 lines of code, but hardest part will be conceptually figuring out how to do it.

Homework 3B will build on the implementation of 3A. So, I will release a solution to 3A on Thurs, April 2 and no late homeworks will be accepted after that.

Overview: Implement image warping and composition in matlab.

homework 3a1
Homework 3A

+

Target

Source

homework 3a2
Homework 3A

+

Target

Source

User clicks on 4 points in target, 4 points in source.

homework 3a3
Homework 3A

+

Target

Source

User clicks on 4 points in target, 4 points in source.

Find best affine transformation between source subimg and target subimg.

homework 3a4
Homework 3A

+

Target

Source

User clicks on 4 points in target, 4 points in source.

Find best affine transformation between source subimg and target subimg.

Transform the source subimg accordingly.

homework 3a5
Homework 3A

+

Target

Source

User clicks on 4 points in target, 4 points in source.

Find best affine transformation between source subimg and target subimg.

Transform the source subimg accordingly.

Insert the transformed source subimg into the target image.

homework 3a6
Homework 3A

+

Target

Source

User clicks on 4 points in target, 4 points in source.

Find best affine transformation between source subimg and target subimg.

Transform the source subimg accordingly.

Insert the transformed source subimg into the target image.

Useful matlab functions: ginput, cp2transform, imtransform. Look these up!

homework 3a7
Homework 3A

+

Target

Source

=

Result

image morphing women in art video
Image Morphing - Women in Art video
  • http://youtube.com/watch?v=nUDIoN-_Hxs
morphing object averaging
Morphing = Object Averaging
  • The aim is to find “an average” between two objects
    • Not an average of two images of objects…
    • …but an image of the average object!
    • How can we make a smooth transition in time?
      • Do a “weighted average” over time t
  • How do we know what the average object looks like?
    • We haven’t a clue!
    • But we can often fake something reasonable
      • Usually required user/artist input
averaging points
Averaging Points
  • P and Q can be anything:
    • points on a plane (2D) or in space (3D)
    • Colors in RGB or HSV (3D)
    • Whole images (m-by-n D)… etc.

Q

v= Q - P

What’s the average

of P and Q?

P

P + 1.5v

= P + 1.5(Q – P)

= -0.5P + 1.5 Q

(extrapolation)

P + 0.5v

= P + 0.5(Q – P)

= 0.5P + 0.5 Q

Linear Interpolation

(Affine Combination):

New point aP + bQ,

defined only when a+b = 1

So aP+bQ = aP+(1-a)Q

idea 1 cross dissolve
Idea #1: Cross-Dissolve
  • Interpolate whole images:
  • Imagehalfway = (1-t)*Image1 + t*image2
  • This is called cross-dissolve in film industry
  • But what if the images are not aligned?
idea 2 align then cross disolve
Idea #2: Align, then cross-disolve
  • Align first, then cross-dissolve
    • Alignment using global warp – picture still valid
dog averaging
Dog Averaging
  • What to do?
    • Cross-dissolve doesn’t work
    • Global alignment doesn’t work
      • Cannot be done with a global transformation (e.g. affine)
    • Any ideas?
  • Feature matching!
    • Nose to nose, tail to tail, etc.
    • This is a local (non-parametric) warp
idea 3 local warp then cross dissolve
Idea #3: Local warp, then cross-dissolve
  • Morphing procedure:
  • for every t,
  • Find the average shape (the “mean dog”)
    • local warping
  • Find the average color
    • Cross-dissolve the warped images
local non parametric image warping
Local (non-parametric) Image Warping
  • Need to specify a more detailed warp function
    • Global warps were functions of a few (2,4,8) parameters
    • Non-parametric warps u(x,y) and v(x,y) can be defined independently for every single location x,y!
    • Once we know vector field u,v we can easily warp each pixel (use backward warping with interpolation)
image warping non parametric
Image Warping – non-parametric
  • Move control points to specify a spline warp
  • Spline produces a smooth vector field
warp specification dense
Warp specification - dense
  • How can we specify the warp?

Specify corresponding spline control points

      • interpolate to a complete warping function

But we want to specify only a few points, not a grid

warp specification sparse
Warp specification - sparse
  • How can we specify the warp?

Specify corresponding points

      • interpolate to a complete warping function
      • How do we do it?

How do we go from feature points to pixels?

triangular mesh
Triangular Mesh
  • Input correspondences at key feature points
  • Define a triangular mesh over the points
    • Same mesh in both images!
    • Now we have triangle-to-triangle correspondences
  • Warp each triangle separately from source to destination
    • How do we warp a triangle?
    • 3 points = affine warp!
    • Just like texture mapping
triangulations
Triangulations
  • A triangulation of set of points in the plane is a partition of the convex hull to triangles whose vertices are the points, and do not contain other points.
  • There are an exponential number of triangulations of a point set.
an o n 3 triangulation algorithm
An O(n3) Triangulation Algorithm
  • Repeat until impossible:
    • Select two sites.
    • If the edge connecting them does not intersect previous edges, keep it.
quality triangulations
“Quality” Triangulations
  • Let (T) = (1, 2 ,.., 3t) be the vector of angles in the triangulation T in increasing order.
  • A triangulation T1 will be “better” than T2 if (T1) > (T2) lexicographically.
  • The Delaunay triangulation is the “best”
    • Maximizes smallest angles

good

bad

improving a triangulation
Improving a Triangulation
  • In any convex quadrangle, an edge flip is possible. If this flip improves the triangulation locally, it also improves the global triangulation.

If an edge flip improves the triangulation, the first edge is called illegal.

illegal edges
Illegal Edges
  • Lemma: An edge pq is illegal iff one of its opposite vertices is inside the circle defined by the other three vertices.
  • Proof: By Thales’ theorem.

p

q

Theorem: A Delaunay triangulation does not contain illegal edges.

Corollary: A triangle is Delaunay iff the circle through its vertices is empty of other sites.

Corollary: The Delaunay triangulation is not unique if more than three sites are co-circular.

na ve delaunay algorithm
Naïve Delaunay Algorithm
  • Start with an arbitrary triangulation. Flip any illegal edge until no more exist.
  • Could take a long time to terminate.
delaunay triangulation by duality
Delaunay Triangulation by Duality
  • General position assumption: There are no four co-circular points.
  • Draw the dual to the Voronoi diagram by connecting each two neighboring sites in the Voronoi diagram.
  • Corollary: The DT may be constructed in O(nlogn) time.
  • This is what Matlab’s delaunay function uses.
image morphing
Image Morphing
  • We know how to warp one image into the other, but how do we create a morphing sequence?
    • Create an intermediate shape (by interpolation)
    • Warp both images towards it
    • Cross-dissolve the colors in the newly warped images
warp interpolation
Warp interpolation
  • How do we create an intermediate warp at time t?
    • Assume t = [0,1]
    • Simple linear interpolation of each feature pair
    • (1-t)*p1+t*p0 for corresponding features p0 and p1
morphing matting
Morphing & matting
  • Extract foreground first to avoid artifacts in the background

Slide by Durand and Freeman

other issues
Other Issues
  • Beware of folding
    • You are probably trying to do something 3D-ish
  • Morphing can be generalized into 3D
    • If you have 3D data, that is!
  • Extrapolation can sometimes produce interesting effects
    • Caricatures