# Combining Photometric and Geometric Constraints - PowerPoint PPT Presentation

1 / 104

Combining Photometric and Geometric Constraints. Yael Moses IDC, Herzliya. Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion. Problem 1:. Recover the 3D shape of a general smooth surface from a set of calibrated images. Problem 2:.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Combining Photometric and Geometric Constraints

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

### Combining Photometric and Geometric Constraints

Yael Moses

IDC, Herzliya

Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion

Y. Moses

Problem 1:

• Recover the 3D shape of a general smooth surface from a set of calibrated images

Y. Moses

Problem 2:

Recover the 3D shape of a smooth bilaterally symmetric object from a single image.

Y. Moses

### Shape Recovery

• Geometry: Stereo

• Photometry:

• Photometric stereo

Main problems:

Calibrations and Correspondence

Y. Moses

### 3D Shape Recovery

Photometry:

• Photometric stereo

Geometry:

• Stereo

• Structure from motion

Y. Moses

### Geometric Stereo

• 2 different images

• Known camera parameters

• Known correspondence

+

+

Y. Moses

### Photometric Stereo

• 3D shape recovery:

surface normals from two or more images taken from the same viewpoint

Y. Moses

Three images

Solution:

Matrix notation

Y. Moses

### Photometric Stereo

Main Limitation:

Correspondence is obtained by a fixed viewpoint

• 3D shape recovery (surface normals)

Two or more images taken from the same viewpoint

Y. Moses

### Overview

• Combining photometric and geometric stereo:

• Symmetric surface, single image

• Non symmetric: 3 images

• Mono-Geometric stereo

• Mono-Photometric stereo

• Experimental results.

Y. Moses

The input

• Smooth featureless surface

• Taken under different viewpoints

• Illuminated by different light sources

• The Problem:

• Recover the 3D shape from a set of calibrated images

Y. Moses

n

n

*

• Perspective projection

### Assumptions

• Three or more images

• Given correspondence the normals can be computed (e.g., Lambertian, distant point light source …)

*

Y. Moses

### Our method

Combines photometric and geometric stereo

We make use of:

• Given Correspondence:

• Can compute a normal

• Can compute the 3D point

Y. Moses

Basic Method

Given

Correspondence

Y. Moses

First Order Surface Approximation

Y. Moses

First Order Surface Approximation

Y. Moses

P() = (1 - )O1+ P,

N(P() - P) = 0

First Order Surface Approximation

Y. Moses

First Order Surface Approximation

Y. Moses

New Correspondence

Y. Moses

New Surface Approximation

Y. Moses

Dense Correspondence

Y. Moses

Y. Moses

Y. Moses

### Basic method: First Order

• Given correspondence pi and L

Pand n

• Given P andn

T

• Given P, T andMi

a new correspondence qi

Y. Moses

### Extensions

• Using more than three images

• Propagation:

• Using multi-neighbours

• Smart propagation

• Second error approximation

• Error correction:

• Based on local continuity

• Other assumptions on the surface

Y. Moses

Y. Moses

### Smart Propagation

Y. Moses

Second Order: a Sphere

(P-P())(N+N)=0

N

P()

P

N+N

N

Y. Moses

Second Order Approximation

Y. Moses

Second Order Approximation

Y. Moses

### Using more than three images

• Reduce noise of the photometric stereo

• Noise

• Violation of assumptions on the surface

Y. Moses

Y. Moses

### Error correction

The compatibility of the local 3D shape can be used to correct errors of:

• Correspondence

• Camera parameters

• Illumination parameters

Y. Moses

### Score

• Continuity:

• Shape

• Normals

• Albedo

• The consistency of 3D points locations and the computed normals:

• General case: full triangulation

• Local constraints

Y. Moses

### Extensions

• Using more than three images

• Propagation:

• Using multi-neighbours

• Smart propagation

• Second error approximation

• Error correction:

• Based on local continuity

• Other assumptions on the surface

Y. Moses

### Real Images

• Camera calibration

• Light calibration

• Direction

• Intensity

• Ambient

Y. Moses

Y. Moses

5pp

5nn

5pn

3pp

3nn

Y. Moses

Y. Moses

Y. Moses

Y. Moses

Y. Moses

Y. Moses

### Detected Correspondence

Y. Moses

Error correction + multi-neighbord

Multi-neighbors

Basic scheme (3 images)

Error correction no multi-neighbors

Y. Moses

Synthetic Images

### New Images

Y. Moses

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec a

Y. Moses

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec b

Y. Moses

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec c

Y. Moses

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec d

Ground truth

Basic scheme

Multi-neighbors approx.

Error correction

Y. Moses

### Combining Photometry and Geometry

Yields a dense correspondence and

dense shape recovery of the object in a single path

Y. Moses

### Assumptions

• Bilaterally Symmetric object

• Lambertian surface with constant albedo

• Orthographic projection

• Known “epipolar geometry”

Y. Moses

### Geometric Stereo

• 2 different images

• Known camera parameters

• Known viewpoints

• Known correspondence

3D shape recovery

Y. Moses

Z

### Computing the Depth from Disparity

P

OrthographicProjection

qr

Z

pr

pl

ql

Y. Moses

### Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses

### Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses

### Geometry

• Weak perspective projection:

Around X

Around Z

Around Y

Y. Moses

### Geometry

• Projection of Ry:

• is the only pose parameter

Around Y

Image point

Object point

Y. Moses

Correspondence

Assume YxZ is the symmetry plane.

image

x

object

z

Y. Moses

### Mono-Geometric Stereo

• 3D reconstruction:

given correspondence and ,

image

x object

known

z

unknown

Y. Moses

### Viewpoint Invariant

• Given the correspondence and unknown 

Invariant

Y. Moses

### Photometric Stereo

• 2 images

• Lambertian reflectance

• Known illuminations

• Known correspondence

(same viewpoint)

3D shape recovery

Y. Moses

### Symmetry and Photometric Stereo

Non-frontal illumination of

a symmetric object

Two different images of the same object

Y. Moses

### Notation: Photometry

• Corresponding object points:

• Illumination:

Y. Moses

### Mono-Photometric Stereo

• 3D reconstruction given correspondence and E (up to a twofold ambiguity):

known

unknown

Y. Moses

### Invariance to Illumination

• Given correspondence and E unknown

• Invariant:

Y. Moses

### Mono-Photometric Stereo

• 3D reconstruction E unknown but correspondence is given

• Frontal viewpoint with

non-frontal illumination.

• Use image first derivatives.

Y. Moses

### Mono-Photometric Stereo Using image derivatives

• 3 global unknowns: E

• For each pair:

• 5 unknowns zxzy zxx zxy zyy

• 6 equations

• 3 pairs are sufficient

Y. Moses

Y. Moses

### Correspondence

• No correspondence => no stereo.

• Hard to define correspondence in images of smooth surfaces.

• Almost any correspondence is legal when:

• Only geometric constraints are considered.

• Only photometric constraints are considered.

Y. Moses

### Combining Photometry and Geometry

• Yields a dense correspondence

(dense shape recovery of the object).

• Enables recovering of the global parameters.

Y. Moses

### Self-Correspondence

• A self-correspondence function:

Y. Moses

### Dense Correspondence using Propagation

Assume correspondence between a pair of points, p0land p0r.

Y. Moses

Y. Moses

image

object

x

z

Y. Moses

### First derivatives of the Correspondence

• Assume known 

• Assume known E

Y. Moses

### Computing and

• Object coordinates:

Given computing and is trivial

• Moving from object to image coordinates depends on the viewing parameter 

Y. Moses

• Derivatives with respect to the object coordinates:

• Derivatives with respect to the image coordinates:

Y. Moses

E

image

object

x

z

Y. Moses

### General Idea

• Given a corresponding pair and E

n=(zx,zy,-1)T

• Given  and n

cxand cy

• Given cxand cy

 a new corresponding pair

Y. Moses

Y. Moses

### Finding Global Parameters

• Assume E and  are unknown.

• Assume a pair of corresponding points is given.

• Two possibilities:

• Search for Eand directly.

• ComputeE andfrom the image second derivatives.

Y. Moses

### Integration Constraint:Circular Tour

• Find and verify correct correspondence

• Recover global parameters, E and 

Y. Moses

### Finding Global Parameters

Consider image second derivatives

• Due to foreshortening effect:

and

• We can relate image and object derivatives by

Y. Moses

### Testing E and : Image second derivatives

For each corresponding pair:

and

Plus 4 linear equations in 3 unknown.

Where

Y. Moses

### Counting

• 5 unknowns for each pair:

zx zy,zxx zxy zyy

• 4 global unknowns: E, 

• For each pair: 6 equations.

• For n pairs: 5n+4 unknowns

6n equations.

4 pairs are sufficient

Y. Moses

Ground Truth

Recovered Shape

Y. Moses

Y. Moses

### Degenerate Case

• Close to frontal view: problems with geometric-stereo.

reconstruction problem

• Close to frontal illumination: problems with photometric-stereo.

correspondence problem

Y. Moses

### Future work

• Perspective photometric stereo

• Use as a first approximation to global optimization methods

• Test on other reflection models

• Recovering of the global parameters:

• Light

• Cameras

• Detect the first pair of correspondence

Y. Moses

### Future Work

• Extend to general 3 images under 3 viewpoints and 3 illuminations.

• Extend to non-lambertian surfaces.

Y. Moses

Y. Moses

image

object

x

z

Y. Moses

Y. Moses

Y. Moses

### Searching for E

• Illumination must satisfy:

• E is further constrained by the image second derivatives.

Y. Moses

Where

### Image second derivatives:

4 linear equations in 3 unknown

Y. Moses

### Image second derivatives

For each corresponding pair and E: 4 linear equations in 3 unknown.

Where

Y. Moses

### Counting

• 5 unknowns for each pair: zx,zy,zxx,zxy,zyy

• 3 global unknowns: E

• For each pair: 6 equations.

• For n pairs: 5n+3 unknowns

6n equations.

3 pairs are sufficient

Y. Moses

Y. Moses

### Variations

• Known/unknown distant light source

• Known/unknown viewpoint

• Symmetric/non-symmetric image

• Frontal/non-frontal viewpoint

• Frontal/non-frontal illumination

Y. Moses

### Correspondence

• Epipolar geometry is the only geometric constraint on the correspondence.

• Weak photometric constraint on the correspondence.

Y. Moses

E

n2

n1

E

I=

*

E

*

P

5

Y. Moses

E

### Photometric Stereo

• First proposed by Woodham, 1980.

• Assume that we have two images ..

Y. Moses