Variations
This presentation is the property of its rightful owner.
Sponsored Links
1 / 84

Variations PowerPoint PPT Presentation


  • 61 Views
  • Uploaded on
  • Presentation posted in: General

Variations. Dimensionality reduction. Computer Vision: Stages. Image formation Low-level Single image processing Multiple views Mid-level Grouping information Segmentation High-level Estimation, Recognition Classification. Why Model variations.

Download Presentation

Variations

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Variations

Variations


Computer vision stages

Dimensionality reduction

Computer Vision: Stages

  • Image formation

  • Low-level

    • Single image processing

    • Multiple views

  • Mid-level

    • Grouping information

    • Segmentation

  • High-level

    • Estimation,

    • Recognition

    • Classification


Why model variations

Why Model variations

Some objects have similar basic form but some variety in the contour shape as and perhaps also pixel values


Today

Segmentation using snakes (from segmentation)

Modeling variations (PCA)

Eigen faces and active shape models

Combining shape and pixel values ( Active Appearance models)

Today


Deformable contours

Deformable contours

a.k.a. active contours, snakes

Given: initial contour (model) near desired object

(Single frame)

[Snakes: Active contour models, Kass, Witkin, & Terzopoulos, ICCV1987]

Fig: Y. Boykov


Deformable contours1

Deformable contours

a.k.a. active contours, snakes

Given: initial contour (model) near desired object

Goal: evolve the contour to fit exact object boundary

(Single frame)

[Snakes: Active contour models, Kass, Witkin, & Terzopoulos, ICCV1987]

Fig: Y. Boykov


Deformable contours intuition

Deformable contours: intuition

Image from http://www.healthline.com/blogs/exercise_fitness/uploaded_images/HandBand2-795868.JPG

Figure from Shapiro & Stockman


Deformable contours2

initial

final

intermediate

Deformable contours

a.k.a. active contours, snakes

  • Initialize near contour of interest

  • Iteratively refine: elastic band is adjusted so as to

  • be near image positions with high gradients, and

  • satisfy shape “preferences” or contour priors

Fig: Y. Boykov


Deformable contours3

initial

final

Deformable contours

a.k.a. active contours, snakes

Like generalized Hough transform, useful for shape fitting; but

intermediate

Hough

Fixed model shape

Single voting pass can detect multiple instances

Snakes

Prior on shape types, but shape iteratively adjusted (deforms)

Requires initialization nearby

One optimization “pass” to fit a single contour


Deformable contours4

initial

intermediate

final

Deformable contours

a.k.a. active contours, snakes

  • How is the current contour adjusted to find the new contour at each iteration?

  • Define a cost function (“energy” function) that says how good a possible configuration is.

  • Seek next configuration that minimizes that cost function.

What are examples of problems with energy functions that we have seen previously?


Snakes energy function

Snakes energy function

The total energy (cost) of the current snake is defined as:

Internal energy: encourage prior shape preferences: e.g., smoothness, elasticity, particular known shape.

External energy (“image” energy): encourage contour to fit on places where image structures exist, e.g., edges.

A good fit between the current deformable contour and the target shape in the image will yield a low value for this cost function.


Variations

Parametric curve representation

(continuous case)

Fig from Y. Boykov


Parametric curve representation discrete form

Parametric curve representation(discrete form)

  • Represent the curve with a set of n points


External energy intuition

External energy: intuition

  • Measure how well the curve matches the image data

  • “Attract” the curve toward different image features

    • Edges, lines, etc.


External image energy

External image energy

How do edges affect “snap” of rubber band?

Think of external energy from image as gravitational pull towards areas of high contrast

Magnitude of gradient

- (Magnitude of gradient)


External image energy1

External image energy

  • Image I(x,y)

  • Gradient images and

  • External energy at a point v(s) on the curve is

  • External energy for the whole curve:


Internal energy intuition

Internal energy: intuition

A priori, we want to favor smooth shapes, contours with low curvature, contours similar to a known shape, etc. to balance what is actually observed (i.e., in the gradient image).

http://www3.imperial.ac.uk/pls/portallive/docs/1/52679.JPG


Internal energy

Internal energy

For a continuous curve, a common internal energy term is the “bending energy”.

At some point v(s) on the curve, this is:

The more the curve bends  the larger this energy value is.

The weights α and β dictate how much influence each component has.

Elasticity,

Tension

Stiffness,

Curvature

Internal energy for whole curve:


Dealing with missing data

Dealing with missing data

  • The smoothness constraint can deal with missing data:

[Figure from Kass et al. 1987]


Total energy continuous form

Total energy(continuous form)

// bending energy

// total edge strength

under curve


Variations

Discrete energy function:external term

  • If the curve is represented by n points

Discrete image gradients


Discrete energy function internal term

Discrete energy function:internal term

  • Curve is represented by n points

Elasticity,

Tension

Stiffness

Curvature


Penalizing elasticity

Penalizing elasticity

  • Current elastic energy definition uses a discrete estimate of the derivative, and can be re-written as:

Possible problem with this definition?

This encourages a closed curve to shrink to a cluster.


Penalizing elasticity1

Penalizing elasticity

  • To stop the curve from shrinking to a cluster of points, we can adjust the energy function to be:

  • This encourages chains of equally spaced points.

Average distance between pairs of points – updated at each iteration


Function of the weights

large

small

medium

weight controls the penalty for internal elasticity

Function of the weights

Fig from Y. Boykov


Optional specify shape prior

Optional: specify shape prior

  • If object is some smooth variation on a known shape, we can use a term that will penalize deviation from that shape (more about this later):

    where are the points of the known shape.

Fig from Y. Boykov


Summary elastic snake

Summary: elastic snake

  • A simple elastic snake is defined by

    • A set of n points,

    • An internal elastic energy term

    • An external edge based energy term

  • To use this to locate the outline of an object

    • Initialize in the vicinity of the object

    • Modify the points to minimize the total energy

How should the weights in the energy function be chosen?


Energy minimization greedy

  • For each point, search window around it and move to where energy function is minimal

    • Typical window size, e.g., 5 x 5 pixels

  • Stop when predefined number of points have not changed in last iteration, or after max number of iterations

  • Note

    • Convergence not guaranteed

    • Need decent initialization

Energy minimization: greedy


Deformable contours5

Deformable contours

Tracking Heart Ventricles

(multiple frames)


Shape

Shape

How to describe the shape of the human face?


Face database

Face database


Objective formulation

Objective Formulation

  • Millions of pixels

  • Transform into a few parameterse.g. Man / woman, fat / skinny etc.


Key idea

Key Idea

Images are points in a high dimensional space

Images in the possible set are highly correlated.

So, compress them to a low-dimensional subspace that

captures key appearance characteristics of the visual DOFs.

Today we will use PCA


Dimensionality reduction

Dimensionality Reduction

The set of faces is a “subspace” of the set

of images

  • Suppose it is K dimensional

  • We can find the best subspace using PCA (see later)

  • This is like fitting a “hyper-plane” to the set of faces

Any face is spanned by basis vectors:


Eigenfaces the idea

Eigenfaces: the idea

Think of a face as being a weighted combination of some “component” or “basis” faces. These basis faces are called eigenfaces

-8029 2900 1751 1445 4238 6193


Eigenfaces representing faces

Eigenfaces: representing faces

The basis faces can be differently weighted to represent any face

-8029 -1183 2900 -2088 1751 -4336 1445 -669 4238 -4221 6193 10549


Learning the basis images

Learning the basis images

Learn a set of basis faces which best represent the differences between the examples

Store each face as a set of weights for those basis faces


Eigenfaces

Eigenfaces

Eigenfaces look somewhat like generic faces.


Recognition reconstruction

recognition & reconstruction

Store and reconstruct a face from a set of weights

Recognise a new picture of a familiar face

Representation

Synthesis


Learning variations

Learning Variations

Use Principle Components Analysis (PCA)

Need to understand

  • What is an eigenvector

  • What is covariance


Principal component analysis

Principal Component analysis

A sample of nobservations in the 2-D space

Goal: to account for the variation in a sample

in as few variables as possible, to some accuracy


Subspaces

Subspaces

Imagine that our face is simply a (high dimensional) vector of pixels

We can think more easily about 2d vectors

Here we have data in two dimensions

But we only really need one dimension to represent it


Finding subspaces

Finding Subspaces

Suppose we take a line through the space

And then take the projection of each point onto that line

This could represent our data in “one” dimension


Finding subspaces1

Finding Subspaces

Some lines will represent the data in this way well, some badly

This is because the projection onto some lines separates the data well, while on others result in bad separation


Finding subspaces2

Finding Subspaces

Rather than a line we can perform roughly the same trick with a vector

Scale the vector to obtain any point on the line


Eigenvectors

Eigenvectors

Aneigenvectorof a matrix A is a vector such that:

Where is a matrix, is a scalar (called the eigenvalue)


Example

Example

one eigenvector of A is

so for this eigenvector of this matrix the eigenvalue is 4

Matlab: [eigvecs, eigVals] = eigs(C);


Facts about eigenvectors

Facts about Eigenvectors

  • The eigenvectors of a matrix are special vectors (for a given matrix) that are only scaled bythematrix

  • Different matrices have different eigenvectors

  • Only square (but not all) matrices have eigenvectors

  • AnN xNmatrix has at mostNdistinct eigenvectors

  • All the distinct eigenvectors of a matrix are orthogonal (ie perpendicular)


How to separate the 2d points

How to separate the 2D points?


Covariance

Covariance

The covariance of two variables is:

The diagonal elements are the variances e.g. Var(x1)

For data that have been centred around the mean


Example1

Example

Matlab: C= cov(X);


Principal component analysis1

Principal Component analysis

The 1stPC is a minimum distance fit to a line in spacealong the direction of most variance (eigenvalue=variance)

The 2nd PCa minimum distance fit to a line

in the plane perpendicular to the 1st PC

PCs are a series of linear least squares fits to a sample,

each orthogonal to all the previous. Combined they constitute a change of basis

A point’s position in this new coordinate system is what we earlier referred to as its “weight vector”


Dimensionality reduction1

i =

K

NM

Dimensionality Reduction

eigenvalues

  • We can represent the points with only their v1 coordinates

    • since v2 coordinates are all essentially 0

    • The eigenvalues are directly related to the variance along a particular direction (ignore directions with low eigenvalue).

  • This makes it much cheaper to store and compare points


Eigenfaces1

Eigenfaces

  • PCA extracts the eigenvectors of A

    • Gives a set of vectors v1, v2, v3, ...

    • Each one of these vectors is a direction in face space


Eigenfaces summary

Eigenfaces - summary

  • Treat images as points in a high-dimensional space

  • Training:

    • Calculate the covariance matrix of the face

    • Calculate eigenvectors of the covariance matrix

    • Eigenfaces with bigger eigenvalues will explain more of the variation in the set of faces, i.e. will be more distinguishing. Chose a subset of the eigenvalues (eg 95% of the total variation).

  • These eigenvectors are the eigenfaces or basis faces


Eigenfaces image space to face space

Eigenfaces: image space to face space

Generative process:

When we see an image of a face we can transform it to face space

Subset of eigenvectors


Reconstruction

Reconstruction

  • The more eigenfaces you have the better the reconstruction, but you can have high quality reconstruction even with a small number of eigenface

8270 50

3020 10


Recognition in face space

Recognition in face space

Recognition could be done by calculating the

Euclidean distance d between our face and all the other stored faces in face space:

The closest face in face space is the chosen match


Summary

Summary

  • Statistical approach to visual recognition

  • Also used for object recognition

  • Reference: M. Turk and A. Pentland (1991). Eigenfaces for recognition, Journal of Cognitive Neuroscience, 3(1): 71–86.


Active appearance models

Active Appearance Models

  • Appearance models capture variability of objects

    • In terms of shape (landmark points) and texture (image intensities)

  • Ingredients

    • Statistical analyses of an annotated training set

  • Result

    • A generative model synthesising complete images of objects

  • Registration

    • Adjusting the model to fit an image

  • If this adjustment done automatically and fast, we have an ActiveAppearance Model (AAM)[Edwards, Taylor & Cootes, AFGR 1998], [Cootes, Edwards & Taylor, ECCV 1998]


Model building

Model building

  • Training set

    • 35 face images annotated with 58 landmarks

  • Active Appearance Model

    • RGB texture model sampled at 30.000 positions

    • 28 model parameters span 95% variation

Training image

Annotation

Model mesh

Shape-compensation


Procrustes analysis

Procrustes analysis

One shape

Principal Component Analysis

Shape parameterisation


Modelling shape ii

Modelling Shape II


Pc1 vs pc2

PC1 vs PC2


Modelling shape iii

Modelling Shape III

  • The three first shape modes shown with a static texture

Mode 1 – 38%

Mode 2 – 13%

Mode 3 – 9%


Modelling texture i

Modelling texture I

  • Grey-level images

  • Colorimages


Modelling texture ii

Modelling Texture II

  • The three first texture modes shown with the mean static shape

Mode 1 – 21%

Mode 2 – 10%

Mode 3 – 8%


Aamexplorer demo

AAMExplorer Demo


Model optimisation i

Model optimisation I

  • AAMs use a simple and efficient iterative scheme

    • Synthesize

    • Calculate difference between synthetic and real image

    • Estimate parameter update

    • Update

    • Iterate until convergence


Model optimisation iv

Model optimisation IV

  • We already know the optimal solution (annotations)

    • Updates are based on differences in a shape-normalised frame(hence “similar” optimisation problems)

    • Limited support

    • Works well for “mild” texture changes

    • Extensions for moresevere texture variationhave been proposed


Example aam search

Example AAM search

reconstructed face

unknown face

detected facial features


Tracking example

Tracking example


Applications of face image analysis

Applications of face image analysis

  • Biometric security

    • Access systems

  • Lip-reading

    • Assisted speech recognition

    • Automated lip-syncing in cartoons

  • Eye-tracking

    • Human-computer interaction

    • Attention analysis (maps, instruments, road signs, road cues)

  • Virtual characters

    • Hugo!

  • Medical applications

    • Improve understanding of syndromic facial dysmorphologies, e.g. the Noonan syndrome


  • Virtual personal makeover

    Virtual personal makeover


    Eye t racking

    Eye tracking

    • Down-scaling of eye-tracking systems to consumer hardware, i.e. low-priced web-cameras


    Suggested reading

    “A Brief Introduction to Statistical Shape Analysis”Mikkel B. Stegmann, David Delgado Gomez http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=403

    Suggested reading


    Efficient computation of eigenvectors

    Efficient Computation of Eigenvectors

    If B is MxN and M<<N then A=BTBis NxN >> MxM

    • M  number of images, N  number of pixels

    • use BBTinstead, eigenvector of BBTis easily

      converted to that of BTB

      (BBT) y = e y

      => BT(BBT) y = e (BTy)

      => (BTB)(BTy) = e (BTy)

      => BTyis the eigenvector ofBTB


  • Login