Combining Photometric and Geometric Constraints

1 / 104

# Combining Photometric and Geometric Constraints - PowerPoint PPT Presentation

Combining Photometric and Geometric Constraints. Yael Moses IDC, Herzliya. Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion. Problem 1:. Recover the 3D shape of a general smooth surface from a set of calibrated images. Problem 2:.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about ' Combining Photometric and Geometric Constraints' - hadley-fowler

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Combining Photometric and Geometric Constraints

Yael Moses

IDC, Herzliya

Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion

Y. Moses

Problem 1:

• Recover the 3D shape of a general smooth surface from a set of calibrated images

Y. Moses

Problem 2:

Recover the 3D shape of a smooth bilaterally symmetric object from a single image.

Y. Moses

Shape Recovery
• Geometry: Stereo
• Photometry:
• Photometric stereo

Main problems:

Calibrations and Correspondence

Y. Moses

3D Shape Recovery

Photometry:

• Photometric stereo

Geometry:

• Stereo
• Structure from motion

Y. Moses

Geometric Stereo
• 2 different images
• Known camera parameters
• Known correspondence

+

+

Y. Moses

Photometric Stereo
• 3D shape recovery:

surface normals from two or more images taken from the same viewpoint

Y. Moses

Three images

Photometric Stereo

Solution:

Matrix notation

Y. Moses

Photometric Stereo

Main Limitation:

Correspondence is obtained by a fixed viewpoint

• 3D shape recovery (surface normals)

Two or more images taken from the same viewpoint

Y. Moses

Overview
• Combining photometric and geometric stereo:
• Symmetric surface, single image
• Non symmetric: 3 images
• Mono-Geometric stereo
• Mono-Photometric stereo
• Experimental results.

Y. Moses

The input

• Smooth featureless surface
• Taken under different viewpoints
• Illuminated by different light sources
• The Problem:
• Recover the 3D shape from a set of calibrated images

Y. Moses

n

n

*

• Perspective projection
Assumptions
• Three or more images
• Given correspondence the normals can be computed (e.g., Lambertian, distant point light source …)

*

Y. Moses

Our method

Combines photometric and geometric stereo

We make use of:

• Given Correspondence:
• Can compute a normal
• Can compute the 3D point

Y. Moses

Basic Method

Given

Correspondence

Y. Moses

P() = (1 - )O1+ P,

N(P() - P) = 0

First Order Surface Approximation

Y. Moses

Basic method: First Order
• Given correspondence pi and L

Pand n

• Given P andn

T

• Given P, T andMi

 a new correspondence qi

Y. Moses

Extensions
• Using more than three images
• Propagation:
• Using multi-neighbours
• Smart propagation
• Second error approximation
• Error correction:
• Based on local continuity
• Other assumptions on the surface

Y. Moses

Second Order: a Sphere

(P-P())(N+N)=0

N

P()

P

N+N

N

Y. Moses

Using more than three images
• Reduce noise of the photometric stereo
• Noise
• Violation of assumptions on the surface

Y. Moses

Error correction

The compatibility of the local 3D shape can be used to correct errors of:

• Correspondence
• Camera parameters
• Illumination parameters

Y. Moses

Score
• Continuity:
• Shape
• Normals
• Albedo
• The consistency of 3D points locations and the computed normals:
• General case: full triangulation
• Local constraints

Y. Moses

Extensions
• Using more than three images
• Propagation:
• Using multi-neighbours
• Smart propagation
• Second error approximation
• Error correction:
• Based on local continuity
• Other assumptions on the surface

Y. Moses

Real Images
• Camera calibration
• Light calibration
• Direction
• Intensity
• Ambient

Y. Moses

5pp

5nn

5pn

3pp

3nn

Y. Moses

Error correction + multi-neighbord

Multi-neighbors

Basic scheme (3 images)

Error correction no multi-neighbors

Y. Moses

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec a

Y. Moses

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec b

Y. Moses

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec c

Y. Moses

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec d

Ground truth

Basic scheme

Multi-neighbors approx.

Error correction

Y. Moses

Combining Photometry and Geometry

Yields a dense correspondence and

dense shape recovery of the object in a single path

Y. Moses

Assumptions
• Bilaterally Symmetric object
• Lambertian surface with constant albedo
• Orthographic projection
• Known “epipolar geometry”

Y. Moses

Geometric Stereo
• 2 different images
• Known camera parameters
• Known viewpoints
• Known correspondence

3D shape recovery

Y. Moses

Z

Computing the Depth from Disparity

P

OrthographicProjection

qr

Z

pr

pl

ql

Y. Moses

Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses

Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses

Geometry
• Weak perspective projection:

Around X

Around Z

Around Y

Y. Moses

Geometry
• Projection of Ry:
• is the only pose parameter

Around Y

Image point

Object point

Y. Moses

Correspondence

Assume YxZ is the symmetry plane.

image

x

object

z

Y. Moses

Mono-Geometric Stereo
• 3D reconstruction:

given correspondence and ,

image

x object

known

z

unknown

Y. Moses

Viewpoint Invariant
• Given the correspondence and unknown 

Invariant

Y. Moses

Photometric Stereo
• 2 images
• Lambertian reflectance
• Known illuminations
• Known correspondence

(same viewpoint)

3D shape recovery

Y. Moses

Symmetry and Photometric Stereo

Non-frontal illumination of

a symmetric object

Two different images of the same object

Y. Moses

Notation: Photometry
• Corresponding object points:
• Illumination:

Y. Moses

Mono-Photometric Stereo
• 3D reconstruction given correspondence and E (up to a twofold ambiguity):

known

unknown

Y. Moses

Invariance to Illumination
• Given correspondence and E unknown
• Invariant:

Y. Moses

Mono-Photometric Stereo
• 3D reconstruction E unknown but correspondence is given
• Frontal viewpoint with

non-frontal illumination.

• Use image first derivatives.

Y. Moses

Mono-Photometric Stereo Using image derivatives
• 3 global unknowns: E
• For each pair:
• 5 unknowns zxzy zxx zxy zyy
• 6 equations
• 3 pairs are sufficient

Y. Moses

Correspondence
• No correspondence => no stereo.
• Hard to define correspondence in images of smooth surfaces.
• Almost any correspondence is legal when:
• Only geometric constraints are considered.
• Only photometric constraints are considered.

Y. Moses

Combining Photometry and Geometry
• Yields a dense correspondence

(dense shape recovery of the object).

• Enables recovering of the global parameters.

Y. Moses

Self-Correspondence
• A self-correspondence function:

Y. Moses

Dense Correspondence using Propagation

Assume correspondence between a pair of points, p0land p0r.

Y. Moses

image

object

x

z

Y. Moses

First derivatives of the Correspondence
• Assume known 
• Assume known E

Y. Moses

Computing and
• Object coordinates:

Given computing and is trivial

• Moving from object to image coordinates depends on the viewing parameter 

Y. Moses

Derivatives with respect to the object coordinates:
• Derivatives with respect to the image coordinates:

Y. Moses

E

image

object

x

z

Y. Moses

General Idea
• Given a corresponding pair and E

n=(zx,zy,-1)T

• Given  and n

cxand cy

• Given cxand cy

 a new corresponding pair

Y. Moses

Finding Global Parameters
• Assume E and  are unknown.
• Assume a pair of corresponding points is given.
• Two possibilities:
• Search for Eand directly.
• ComputeE andfrom the image second derivatives.

Y. Moses

Integration Constraint:Circular Tour
• Find and verify correct correspondence
• Recover global parameters, E and 

Y. Moses

Finding Global Parameters

Consider image second derivatives

• Due to foreshortening effect:

and

• We can relate image and object derivatives by

Y. Moses

Testing E and : Image second derivatives

For each corresponding pair:

and

Plus 4 linear equations in 3 unknown.

Where

Y. Moses

Counting
• 5 unknowns for each pair:

zx zy,zxx zxy zyy

• 4 global unknowns: E, 
• For each pair: 6 equations.
• For n pairs: 5n+4 unknowns

6n equations.

4 pairs are sufficient

Y. Moses

Results on Simulated Data

Ground Truth

Recovered Shape

Y. Moses

Degenerate Case
• Close to frontal view: problems with geometric-stereo.

reconstruction problem

• Close to frontal illumination: problems with photometric-stereo.

correspondence problem

Y. Moses

Future work
• Perspective photometric stereo
• Use as a first approximation to global optimization methods
• Test on other reflection models
• Recovering of the global parameters:
• Light
• Cameras
• Detect the first pair of correspondence

Y. Moses

Future Work
• Extend to general 3 images under 3 viewpoints and 3 illuminations.
• Extend to non-lambertian surfaces.

Y. Moses

Thanks

Y. Moses

image

object

x

z

Y. Moses

Searching for E
• Illumination must satisfy:
• E is further constrained by the image second derivatives.

Y. Moses

Where

Image second derivatives:

4 linear equations in 3 unknown

Y. Moses

Image second derivatives

For each corresponding pair and E: 4 linear equations in 3 unknown.

Where

Y. Moses

Counting
• 5 unknowns for each pair: zx,zy,zxx,zxy,zyy
• 3 global unknowns: E
• For each pair: 6 equations.
• For n pairs: 5n+3 unknowns

6n equations.

3 pairs are sufficient

Y. Moses

Variations
• Known/unknown distant light source
• Known/unknown viewpoint
• Symmetric/non-symmetric image
• Frontal/non-frontal viewpoint
• Frontal/non-frontal illumination

Y. Moses

Correspondence
• Epipolar geometry is the only geometric constraint on the correspondence.
• Weak photometric constraint on the correspondence.

Y. Moses

E

n2

n1

E

Lambertian Surface

I=

*

E

*

P

5

Y. Moses

E

Photometric Stereo
• First proposed by Woodham, 1980.
• Assume that we have two images ..

Y. Moses