slide1 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Real-time Combined 2D+3D Active Appearance Models Jing Xiao, Simon Baker,Iain Matthew, and Takeo Kanade CVPR 2004 PowerPoint Presentation
Download Presentation
Real-time Combined 2D+3D Active Appearance Models Jing Xiao, Simon Baker,Iain Matthew, and Takeo Kanade CVPR 2004

Loading in 2 Seconds...

play fullscreen
1 / 29

Real-time Combined 2D+3D Active Appearance Models Jing Xiao, Simon Baker,Iain Matthew, and Takeo Kanade CVPR 2004 - PowerPoint PPT Presentation


  • 359 Views
  • Uploaded on

Real-time Combined 2D+3D Active Appearance Models Jing Xiao, Simon Baker,Iain Matthew, and Takeo Kanade CVPR 2004. Presented by Pat Chan 23/11/2004. Outline. Introduction Active Appearance Models AAMs 3D Morphable Models 3DMMs Representational Power of AAM Combined 2D+3D AAMs Conclusion.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Real-time Combined 2D+3D Active Appearance Models Jing Xiao, Simon Baker,Iain Matthew, and Takeo Kanade CVPR 2004' - aubrianna


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Real-time Combined 2D+3D Active Appearance ModelsJing Xiao, Simon Baker,Iain Matthew, and Takeo KanadeCVPR 2004

Presented by Pat Chan

23/11/2004

outline
Outline
  • Introduction
  • Active Appearance Models AAMs
  • 3D Morphable Models 3DMMs
  • Representational Power of AAM
  • Combined 2D+3D AAMs
  • Conclusion
introduction
Introduction
  • Active Appearance Models are generative models commonly used to model faces
  • Another closely related type of face models are 3D Morphable Models
  • In this paper, it tries to model 3D phenomena by using the 2D AAM
  • Constrain the AAM with the 3D models to achieve a real-time algorithm for fitting the AAM
active appearance models aams
Active Appearance Models (AAMs)
  • 2D linear shape is defined by 2D triangulated mesh and in particular the vertex locations of the mesh.
  • Shape scan be expressed as a base shape s0.
  • pi are the shape parameter.
  • s0 is the mean shape and the matrices si are the eigenvectors corresponding to the m largest eigenvalues
active appearance models aams5

A0(u) A1(u) A2(u) A3(u)

Active Appearance Models (AAMs)
  • The appearanceof an independent AAM is defined within the base mesh s0. A(u) defined over the pixels u∈ s0
  • A(u) can be expressed as a base appearance A0(u) plus a linear combination of l appearance
  • Coefficients λi are the appearance parameters.
active appearance models aams6
Active Appearance Models (AAMs)
  • The AAM model instance with shape parameters pand appearance parameters λ is then created by warping the appearance Afrom the base mesh s0 to the model shape s.

Piecewise affine warp W(u; p):

(1) for any pixel u in s0 find out which triangle it lies in,

(2) warp u with the affine warp for that triangle.

M(W(u;p))

fitting aams

u

u

u

u

Fitting AAMs
  • Minimize the error between I (u) and M(W(u; p)) = A(u).
  • If u is a pixel in s0, then the corresponding pixel in the input image I is W(u; p).
  • At pixel u the AAM has the appearance
  • At pixel W(u; p), the input image has the intensity I (W(u; p)).
  • Minimize the sum of squares of the difference between these two quantities:
3d morphable models 3dmms
3D Morphable Models (3DMMS)
  • 3D shape of 3DMM is defined by 3D triangulated mesh and in particular the vertex location of the mesh.
  • The ŝ can be expressed as a based shape ŝ0 plus a linear combination of m shape matrices ŝi:
3d morphable models 3dmms11
3D Morphable Models (3DMMS)
  • The appearance of a 3DMM is defined within a 2D triangulated mesh that has the same topology as the base mesh ŝ0.
  • The appearance Â(u) can be expressed as a based appearance Â0(u) plus a linear combination of l appearance images Âi(u).
3d morphable models 3dmms12
3D Morphable Models (3DMMS)
  • To generate a 3DMM model instance, an image formation model is used to convert the 3D shape ŝ in to 2D mesh.
  • The result of the imaging 3D point x = (x, y, z)T is:
  • i, j are the projection axes, o is the offset of the origin
  • Given shape parameters picompute 3D shape  map to 2D  compute appearance  warp onto 2D mesh (defined by mapping from 2D vertices in ŝ0 to 2D vertices for 3D ŝ.)
representational power of aam
Representational Power of AAM

Can 2D shape models represent 3D?

  • The 2D shape variation of the 3D model:
  • The projection matrix can be expressed as:
  • Therefore 3D model can be represented as combination of:
  • The variation of the 3D model can therefore be represented by an appropriate set of 2D shape vectors, such as:

6*(m^+1) 2D shape vectors needed to represent a 3D model

representational power of aam14
Representational Power of AAM
  • Experiments
    • Use 3D-cube:ŝ = ŝ0 + p1ŝ1
    • Generate 60 sets p1 and P randomly
    • Synthesize 2D shapes of 60 3D model instances
    • Compute 2D shape model by performing PCA on 60 2D shapes
    • Result: 12 shapes vectors for each 2D shape mode
    • Confirm: 6*(m^+1) 2D vector is required
  • However, 2D models generate invalid cases.
  • Constrains is need to add on the model
combined 2d 3d aams
Combined 2D + 3D AAMs
  • At time t, we have
  • 2D AAM shape vector in all N images into a matrix:
  • Represent as a 3D linear shape modesW = MB =
compute the 3d model
Compute the 3D Model
  • Perform singular value decomposition (SVD) on W and factorize it into:
  • The scaled projection matrix M and the shape vector matrix B are given by:
  • G is the corrective matrix.
  • Additional rotational and basis constrain to compute G  M and B can be determined
  • Thus, the 3D shape modes can be computed from the 2D AMM shape modes and the 2D AAM tracking results.
calculate the corrective matrix
Calculate the Corrective Matrix
  • Rotational constraints and basis constraints are used.
  • Rotational constraints (denote GGTby Q):
  • where ˜M2*i−1:2*irepresents the ithtwo-row of ˜M
  • c is the coefficient and R is rotation matrices
  • Due to orthogonormality of rotation matrices and Q is symmetric,
calculate the corrective matrix18
Calculate the Corrective Matrix
  • Basis constraints:
  • We find K frames including independent shapes and treat those shapes as a set of bases, the bases are determined uniquely, we have
compute the 3d model19
Compute the 3D Model

AAM shapes

AAM appearance

First three 3D shapes modes

constraining an aam with 3d shape
Constraining an AAM with 3D Shape
  • Constraints on the 2D AAM shape parameters p = (p1, … , pm) that force the AAM to only move in a way that is consistent with the 3D shape modes:
  • and the 2D shape variation of the 3D shape modes over all imaging condition is:
  • Legitimate values of P and p such that the 2D projected 3D shape equals the 2D shape of AAM. The constraint is written as:
fitting with 3d shape constraints
Fitting with 3D Shape Constraints
  • AAM fitting is to minimize:
  • I.e the error between the appearance and the original image
  • Impose the constrains of 2D projected 3D shape equals the 2D shape of AAM as soft constrains on the above equation with a large K:
fitting with 3d shape constraints22
Fitting with 3D Shape Constraints
  • Optimize for the AAM shape p, q, and the appearance λ parameters:
  • Calculate the square difference between the appearance and the original image and project the difference into orthogonal complement of the linear subspace spanned by the vectors A1, …, Al.
  • It is optimized by using inverse compositional algorithm, I.e. iteratively minimizing:
  • Then, solve the appearance parameters using the linear closed form solution:
experimental results
Experimental Results

Estimated 3D shape

Estimates of the 3D

Pose extracted from

the current estimate

of the camera matrix P

Initialization

2D AAM

After 30 Iterations

Converged

experimental results24
Experimental Results

Results of using the algorithm to track a face in 180

frame video sequence by fitting the model in each frame

2d 3d aam model reconstruction
2D+3D AAM Model Reconstruction

Input Image

Tracked result (2D+3D fit result)

Shows two new view reconstruction

2D+3D model reconstruction

compare the fitting speed with 2d aams
Compare the fitting speed with 2D AAMs
  • Frames per second of 2D+3D > 2D AAM
  • Iteration per second of 2D > 2D+3D, but 2D need more iteration for convergence
conclusion
Conclusion
  • 2D AAMs can represent any phenomena that 3DMMs can.
  • Showed how to compute the equivalent 3D shape models from a 2D AAM with basis constrains, rotational constrains.
  • Improve the fitting speed of the 2D AAMs with 3D shapes constrains
  • 2D + 3D AAM is the ability to render the 3D model from novel viewpoint.