1 / 104

# Combining Photometric and Geometric Constraints - PowerPoint PPT Presentation

Combining Photometric and Geometric Constraints. Yael Moses IDC, Herzliya. Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion. Problem 1:. Recover the 3D shape of a general smooth surface from a set of calibrated images. Problem 2:.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about ' Combining Photometric and Geometric Constraints' - hadley-fowler

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Yael Moses

IDC, Herzliya

Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion

Y. Moses

• Recover the 3D shape of a general smooth surface from a set of calibrated images

Y. Moses

Recover the 3D shape of a smooth bilaterally symmetric object from a single image.

Y. Moses

• Geometry: Stereo

• Photometry:

• Photometric stereo

Main problems:

Calibrations and Correspondence

Y. Moses

Photometry:

• Photometric stereo

Geometry:

• Stereo

• Structure from motion

Y. Moses

• 2 different images

• Known camera parameters

• Known correspondence

+

+

Y. Moses

• 3D shape recovery:

surface normals from two or more images taken from the same viewpoint

Y. Moses

Photometric Stereo

Solution:

Matrix notation

Y. Moses

Main Limitation:

Correspondence is obtained by a fixed viewpoint

• 3D shape recovery (surface normals)

Two or more images taken from the same viewpoint

Y. Moses

• Combining photometric and geometric stereo:

• Symmetric surface, single image

• Non symmetric: 3 images

• Mono-Geometric stereo

• Mono-Photometric stereo

• Experimental results.

Y. Moses

• Smooth featureless surface

• Taken under different viewpoints

• Illuminated by different light sources

• The Problem:

• Recover the 3D shape from a set of calibrated images

Y. Moses

n

*

• Perspective projection

Assumptions

• Three or more images

• Given correspondence the normals can be computed (e.g., Lambertian, distant point light source …)

*

Y. Moses

Combines photometric and geometric stereo

We make use of:

• Given Correspondence:

• Can compute a normal

• Can compute the 3D point

Y. Moses

Given

Correspondence

Y. Moses

P() = (1 - )O1+ P,

N(P() - P) = 0

First Order Surface Approximation

Y. Moses

Y. Moses

Y. Moses

Y. Moses

Y. Moses

Y. Moses

• Given correspondence pi and L

Pand n

• Given P andn

T

• Given P, T andMi

 a new correspondence qi

Y. Moses

• Using more than three images

• Propagation:

• Using multi-neighbours

• Smart propagation

• Second error approximation

• Error correction:

• Based on local continuity

• Other assumptions on the surface

Y. Moses

Y. Moses

Y. Moses

(P-P())(N+N)=0

N

P()

P

N+N

N

Y. Moses

Y. Moses

Y. Moses

• Reduce noise of the photometric stereo

• Noise

• Violation of assumptions on the surface

Y. Moses

Y. Moses

The compatibility of the local 3D shape can be used to correct errors of:

• Correspondence

• Camera parameters

• Illumination parameters

Y. Moses

• Continuity:

• Shape

• Normals

• Albedo

• The consistency of 3D points locations and the computed normals:

• General case: full triangulation

• Local constraints

Y. Moses

• Using more than three images

• Propagation:

• Using multi-neighbours

• Smart propagation

• Second error approximation

• Error correction:

• Based on local continuity

• Other assumptions on the surface

Y. Moses

• Camera calibration

• Light calibration

• Direction

• Intensity

• Ambient

Y. Moses

5nn

5pn

3pp

3nn

Y. Moses

Y. Moses

Multi-neighbors

Basic scheme (3 images)

Error correction no multi-neighbors

Y. Moses

New Images

Y. Moses

Basic scheme

Multi-neighbors Error correction

Sec a

Y. Moses

Basic scheme

Multi-neighbors Error correction

Sec b

Y. Moses

Basic scheme

Multi-neighbors Error correction

Sec c

Y. Moses

Basic scheme

Multi-neighbors Error correction

Sec d

Ground truth

Basic scheme

Multi-neighbors approx.

Error correction

Y. Moses

Combining Photometry and Geometry

Yields a dense correspondence and

dense shape recovery of the object in a single path

Y. Moses

• Bilaterally Symmetric object

• Lambertian surface with constant albedo

• Orthographic projection

• Known “epipolar geometry”

Y. Moses

• 2 different images

• Known camera parameters

• Known viewpoints

• Known correspondence

3D shape recovery

Y. Moses

Computing the Depth from Disparity

P

OrthographicProjection

qr

Z

pr

pl

ql

Y. Moses

Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses

Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses

• Weak perspective projection:

Around X

Around Z

Around Y

Y. Moses

• Projection of Ry:

• is the only pose parameter

Around Y

Image point

Object point

Y. Moses

Assume YxZ is the symmetry plane.

image

x

object

z

Y. Moses

• 3D reconstruction:

given correspondence and ,

image

x object

known

z

unknown

Y. Moses

• Given the correspondence and unknown 

Invariant

Y. Moses

• 2 images

• Lambertian reflectance

• Known illuminations

• Known correspondence

(same viewpoint)

3D shape recovery

Y. Moses

Symmetry and Photometric Stereo

Non-frontal illumination of

a symmetric object

Two different images of the same object

Y. Moses

• Corresponding object points:

• Illumination:

Y. Moses

• 3D reconstruction given correspondence and E (up to a twofold ambiguity):

known

unknown

Y. Moses

• Given correspondence and E unknown

• Invariant:

Y. Moses

• 3D reconstruction E unknown but correspondence is given

• Frontal viewpoint with

non-frontal illumination.

• Use image first derivatives.

Y. Moses

Mono-Photometric Stereo Using image derivatives

• 3 global unknowns: E

• For each pair:

• 5 unknowns zxzy zxx zxy zyy

• 6 equations

• 3 pairs are sufficient

Y. Moses

Mono-Photometric StereoUnknown Illumination

Y. Moses

• No correspondence => no stereo.

• Hard to define correspondence in images of smooth surfaces.

• Almost any correspondence is legal when:

• Only geometric constraints are considered.

• Only photometric constraints are considered.

Y. Moses

Combining Photometry and Geometry

• Yields a dense correspondence

(dense shape recovery of the object).

• Enables recovering of the global parameters.

Y. Moses

• A self-correspondence function:

Y. Moses

Assume correspondence between a pair of points, p0land p0r.

Y. Moses

object

x

z

Y. Moses

• Assume known 

• Assume known E

Y. Moses

• Object coordinates:

Given computing and is trivial

• Moving from object to image coordinates depends on the viewing parameter 

Y. Moses

Y. Moses

image

object

x

z

Y. Moses

• Given a corresponding pair and E

n=(zx,zy,-1)T

• Given  and n

cxand cy

• Given cxand cy

 a new corresponding pair

Y. Moses

Results on Real Images: Given global parameters

Y. Moses

• Assume E and  are unknown.

• Assume a pair of corresponding points is given.

• Two possibilities:

• Search for Eand directly.

• ComputeE andfrom the image second derivatives.

Y. Moses

Integration Constraint:Circular Tour

• Find and verify correct correspondence

• Recover global parameters, E and 

Y. Moses

Consider image second derivatives

• Due to foreshortening effect:

and

• We can relate image and object derivatives by

Y. Moses

Testing E and : Image second derivatives

For each corresponding pair:

and

Plus 4 linear equations in 3 unknown.

Where

Y. Moses

• 5 unknowns for each pair:

zx zy,zxx zxy zyy

• 4 global unknowns: E, 

• For each pair: 6 equations.

• For n pairs: 5n+4 unknowns

6n equations.

4 pairs are sufficient

Y. Moses

Ground Truth

Recovered Shape

Y. Moses

Y. Moses

• Close to frontal view: problems with geometric-stereo.

reconstruction problem

• Close to frontal illumination: problems with photometric-stereo.

correspondence problem

Y. Moses

• Perspective photometric stereo

• Use as a first approximation to global optimization methods

• Test on other reflection models

• Recovering of the global parameters:

• Light

• Cameras

• Detect the first pair of correspondence

Y. Moses

• Extend to general 3 images under 3 viewpoints and 3 illuminations.

• Extend to non-lambertian surfaces.

Y. Moses

Y. Moses

object

x

z

Y. Moses

Y. Moses

Y. Moses

• Illumination must satisfy:

• E is further constrained by the image second derivatives.

Y. Moses

Image second derivatives:

4 linear equations in 3 unknown

Y. Moses

For each corresponding pair and E: 4 linear equations in 3 unknown.

Where

Y. Moses

• 5 unknowns for each pair: zx,zy,zxx,zxy,zyy

• 3 global unknowns: E

• For each pair: 6 equations.

• For n pairs: 5n+3 unknowns

6n equations.

3 pairs are sufficient

Y. Moses

Y. Moses

• Known/unknown distant light source

• Known/unknown viewpoint

• Symmetric/non-symmetric image

• Frontal/non-frontal viewpoint

• Frontal/non-frontal illumination

Y. Moses

• Epipolar geometry is the only geometric constraint on the correspondence.

• Weak photometric constraint on the correspondence.

Y. Moses

n2

n1

E

Lambertian Surface

I=

*

E

*

P

5

Y. Moses

Photometric Stereo

• First proposed by Woodham, 1980.

• Assume that we have two images ..

Y. Moses