- 48 Views
- Uploaded on
- Presentation posted in: General

Combining Photometric and Geometric Constraints

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Yael Moses

IDC, Herzliya

Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion

Y. Moses

Problem 1:

- Recover the 3D shape of a general smooth surface from a set of calibrated images

Y. Moses

Problem 2:

Recover the 3D shape of a smooth bilaterally symmetric object from a single image.

Y. Moses

- Geometry: Stereo
- Photometry:
- Shape from shading
- Photometric stereo
Main problems:

Calibrations and Correspondence

Y. Moses

Photometry:

- Shape from shading
- Photometric stereo

Geometry:

- Stereo
- Structure from motion

Y. Moses

- 2 different images
- Known camera parameters
- Known correspondence

+

+

Y. Moses

- 3D shape recovery:
surface normals from two or more images taken from the same viewpoint

Y. Moses

Three images

Solution:

Matrix notation

Y. Moses

Main Limitation:

Correspondence is obtained by a fixed viewpoint

- 3D shape recovery (surface normals)
Two or more images taken from the same viewpoint

Y. Moses

- Combining photometric and geometric stereo:
- Symmetric surface, single image
- Non symmetric: 3 images

- Mono-Geometric stereo
- Mono-Photometric stereo
- Experimental results.

Y. Moses

The input

- Smooth featureless surface
- Taken under different viewpoints
- Illuminated by different light sources

- The Problem:
- Recover the 3D shape from a set of calibrated images

Y. Moses

n

n

*

- Perspective projection

- Three or more images

- Given correspondence the normals can be computed (e.g., Lambertian, distant point light source …)

*

Y. Moses

Combines photometric and geometric stereo

We make use of:

- Given Correspondence:
- Can compute a normal
- Can compute the 3D point

Y. Moses

Basic Method

Given

Correspondence

Y. Moses

First Order Surface Approximation

Y. Moses

First Order Surface Approximation

Y. Moses

P() = (1 - )O1+ P,

N(P() - P) = 0

First Order Surface Approximation

Y. Moses

First Order Surface Approximation

Y. Moses

New Correspondence

Y. Moses

New Surface Approximation

Y. Moses

Dense Correspondence

Y. Moses

Y. Moses

Y. Moses

- Given correspondence pi and L
Pand n

- Given P andn
T

- Given P, T andMi
a new correspondence qi

Y. Moses

- Using more than three images
- Propagation:
- Using multi-neighbours
- Smart propagation

- Second error approximation
- Error correction:
- Based on local continuity
- Other assumptions on the surface

Y. Moses

Y. Moses

Y. Moses

Second Order: a Sphere

(P-P())(N+N)=0

N

P()

P

N+N

N

Y. Moses

Second Order Approximation

Y. Moses

Second Order Approximation

Y. Moses

- Reduce noise of the photometric stereo
- Avoid shadowed pixels
- Detect “bad pixels”
- Noise
- Shadows
- Violation of assumptions on the surface

Y. Moses

Y. Moses

The compatibility of the local 3D shape can be used to correct errors of:

- Correspondence
- Camera parameters
- Illumination parameters

Y. Moses

- Continuity:
- Shape
- Normals
- Albedo

- The consistency of 3D points locations and the computed normals:
- General case: full triangulation
- Local constraints

Y. Moses

- Using more than three images
- Propagation:
- Using multi-neighbours
- Smart propagation

- Second error approximation
- Error correction:
- Based on local continuity
- Other assumptions on the surface

Y. Moses

- Camera calibration
- Light calibration
- Direction
- Intensity
- Ambient

Y. Moses

Y. Moses

5pp

5nn

5pn

3pp

3nn

Y. Moses

Y. Moses

Y. Moses

Y. Moses

Y. Moses

Y. Moses

Y. Moses

Error correction + multi-neighbord

Multi-neighbors

Basic scheme (3 images)

Error correction no multi-neighbors

Y. Moses

Synthetic Images

Y. Moses

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec a

Y. Moses

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec b

Y. Moses

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec c

Y. Moses

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec d

Ground truth

Basic scheme

Multi-neighbors approx.

Error correction

Y. Moses

Yields a dense correspondence and

dense shape recovery of the object in a single path

Y. Moses

- Bilaterally Symmetric object
- Lambertian surface with constant albedo
- Orthographic projection
- Neither occlusions nor shadows
- Known “epipolar geometry”

Y. Moses

- 2 different images
- Known camera parameters
- Known viewpoints
- Known correspondence

3D shape recovery

Y. Moses

Z

P

OrthographicProjection

qr

Z

pr

pl

ql

Y. Moses

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses

- Weak perspective projection:

Around X

Around Z

Around Y

Y. Moses

- Projection of Ry:

- is the only pose parameter

Around Y

Image point

Object point

Y. Moses

Correspondence

Assume YxZ is the symmetry plane.

image

x

object

z

Y. Moses

- 3D reconstruction:
given correspondence and ,

image

x object

known

z

unknown

Y. Moses

- Given the correspondence and unknown

Invariant

Y. Moses

- 2 images
- Lambertian reflectance
- Known illuminations
- Known correspondence
(same viewpoint)

3D shape recovery

Y. Moses

Non-frontal illumination of

a symmetric object

Two different images of the same object

Y. Moses

- Corresponding object points:
- Illumination:

Y. Moses

- 3D reconstruction given correspondence and E (up to a twofold ambiguity):

known

unknown

Y. Moses

- Given correspondence and E unknown
- Invariant:

Y. Moses

- 3D reconstruction E unknown but correspondence is given
- Frontal viewpoint with
non-frontal illumination.

- Use image first derivatives.

Y. Moses

- 3 global unknowns: E
- For each pair:
- 5 unknowns zxzy zxx zxy zyy
- 6 equations

- 3 pairs are sufficient

Y. Moses

Y. Moses

- No correspondence => no stereo.
- Hard to define correspondence in images of smooth surfaces.
- Almost any correspondence is legal when:
- Only geometric constraints are considered.
- Only photometric constraints are considered.

Y. Moses

- Yields a dense correspondence
(dense shape recovery of the object).

- Enables recovering of the global parameters.

Y. Moses

- A self-correspondence function:

Y. Moses

Assume correspondence between a pair of points, p0land p0r.

Y. Moses

Y. Moses

image

object

x

z

Y. Moses

- Assume known
- Assume known E

Y. Moses

- Object coordinates:
Given computing and is trivial

- Moving from object to image coordinates depends on the viewing parameter

Y. Moses

- Derivatives with respect to the object coordinates:
- Derivatives with respect to the image coordinates:

Y. Moses

E

image

object

x

z

Y. Moses

- Given a corresponding pair and E
n=(zx,zy,-1)T

- Given and n
cxand cy

- Given cxand cy
a new corresponding pair

Y. Moses

Y. Moses

- Assume E and are unknown.
- Assume a pair of corresponding points is given.
- Two possibilities:
- Search for Eand directly.
- ComputeE andfrom the image second derivatives.

Y. Moses

- All roads lead to Rome …
- Find and verify correct correspondence
- Recover global parameters, E and

Y. Moses

Consider image second derivatives

- Due to foreshortening effect:
and

- We can relate image and object derivatives by

Y. Moses

For each corresponding pair:

and

Plus 4 linear equations in 3 unknown.

Where

Y. Moses

- 5 unknowns for each pair:
zx zy,zxx zxy zyy

- 4 global unknowns: E,
- For each pair: 6 equations.
- For n pairs: 5n+4 unknowns
6n equations.

4 pairs are sufficient

Y. Moses

Ground Truth

Recovered Shape

Y. Moses

Y. Moses

- Close to frontal view: problems with geometric-stereo.
reconstruction problem

- Close to frontal illumination: problems with photometric-stereo.
correspondence problem

Y. Moses

- Perspective photometric stereo
- Use as a first approximation to global optimization methods
- Test on other reflection models
- Recovering of the global parameters:
- Light
- Cameras

- Detect the first pair of correspondence

Y. Moses

- Extend to general 3 images under 3 viewpoints and 3 illuminations.
- Extend to non-lambertian surfaces.

Y. Moses

Y. Moses

image

object

x

z

Y. Moses

Y. Moses

Y. Moses

- Illumination must satisfy:
- E is further constrained by the image second derivatives.

Y. Moses

Where

4 linear equations in 3 unknown

Y. Moses

For each corresponding pair and E: 4 linear equations in 3 unknown.

Where

Y. Moses

- 5 unknowns for each pair: zx,zy,zxx,zxy,zyy
- 3 global unknowns: E
- For each pair: 6 equations.
- For n pairs: 5n+3 unknowns
6n equations.

3 pairs are sufficient

Y. Moses

Y. Moses

- Known/unknown distant light source
- Known/unknown viewpoint
- Symmetric/non-symmetric image
- Frontal/non-frontal viewpoint
- Frontal/non-frontal illumination

Y. Moses

- Epipolar geometry is the only geometric constraint on the correspondence.
- Weak photometric constraint on the correspondence.

Y. Moses

E

n2

n1

E

I=

Basic radiometric

*

E

*

P

5

Y. Moses

E

- First proposed by Woodham, 1980.
- Assume that we have two images ..

Y. Moses