Combining photometric and geometric constraints
Sponsored Links
This presentation is the property of its rightful owner.
1 / 104

Combining Photometric and Geometric Constraints PowerPoint PPT Presentation


  • 48 Views
  • Uploaded on
  • Presentation posted in: General

Combining Photometric and Geometric Constraints. Yael Moses IDC, Herzliya. Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion. Problem 1:. Recover the 3D shape of a general smooth surface from a set of calibrated images. Problem 2:.

Download Presentation

Combining Photometric and Geometric Constraints

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Combining Photometric and Geometric Constraints

Yael Moses

IDC, Herzliya

Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion

Y. Moses


Problem 1:

  • Recover the 3D shape of a general smooth surface from a set of calibrated images

Y. Moses


Problem 2:

Recover the 3D shape of a smooth bilaterally symmetric object from a single image.

Y. Moses


Shape Recovery

  • Geometry: Stereo

  • Photometry:

    • Shape from shading

    • Photometric stereo

      Main problems:

      Calibrations and Correspondence

Y. Moses


3D Shape Recovery

Photometry:

  • Shape from shading

  • Photometric stereo

Geometry:

  • Stereo

  • Structure from motion

Y. Moses


Geometric Stereo

  • 2 different images

  • Known camera parameters

  • Known correspondence

+

+

Y. Moses


Photometric Stereo

  • 3D shape recovery:

    surface normals from two or more images taken from the same viewpoint

Y. Moses


Three images

Photometric Stereo

Solution:

Matrix notation

Y. Moses


Photometric Stereo

Main Limitation:

Correspondence is obtained by a fixed viewpoint

  • 3D shape recovery (surface normals)

    Two or more images taken from the same viewpoint

Y. Moses


Overview

  • Combining photometric and geometric stereo:

    • Symmetric surface, single image

    • Non symmetric: 3 images

  • Mono-Geometric stereo

  • Mono-Photometric stereo

  • Experimental results.

Y. Moses


The input

  • Smooth featureless surface

  • Taken under different viewpoints

  • Illuminated by different light sources

  • The Problem:

  • Recover the 3D shape from a set of calibrated images

Y. Moses


n

n

*

  • Perspective projection

Assumptions

  • Three or more images

  • Given correspondence the normals can be computed (e.g., Lambertian, distant point light source …)

*

Y. Moses


Our method

Combines photometric and geometric stereo

We make use of:

  • Given Correspondence:

    • Can compute a normal

    • Can compute the 3D point

Y. Moses


Basic Method

Given

Correspondence

Y. Moses


First Order Surface Approximation

Y. Moses


First Order Surface Approximation

Y. Moses


P() = (1 - )O1+ P,

N(P() - P) = 0

First Order Surface Approximation

Y. Moses


First Order Surface Approximation

Y. Moses


New Correspondence

Y. Moses


New Surface Approximation

Y. Moses


Dense Correspondence

Y. Moses


Basic Propagation

Y. Moses


Basic Propagation

Y. Moses


Basic method: First Order

  • Given correspondence pi and L

    Pand n

  • Given P andn

    T

  • Given P, T andMi

    a new correspondence qi

Y. Moses


Extensions

  • Using more than three images

  • Propagation:

    • Using multi-neighbours

    • Smart propagation

  • Second error approximation

  • Error correction:

    • Based on local continuity

    • Other assumptions on the surface

Y. Moses


Multi-neighbors Propagation

Y. Moses


Smart Propagation

Y. Moses


Second Order: a Sphere

(P-P())(N+N)=0

N

P()

P

N+N

N

Y. Moses


Second Order Approximation

Y. Moses


Second Order Approximation

Y. Moses


Using more than three images

  • Reduce noise of the photometric stereo

  • Avoid shadowed pixels

  • Detect “bad pixels”

    • Noise

    • Shadows

    • Violation of assumptions on the surface

Y. Moses


Smart Propagation

Y. Moses


Error correction

The compatibility of the local 3D shape can be used to correct errors of:

  • Correspondence

  • Camera parameters

  • Illumination parameters

Y. Moses


Score

  • Continuity:

    • Shape

    • Normals

    • Albedo

  • The consistency of 3D points locations and the computed normals:

    • General case: full triangulation

    • Local constraints

Y. Moses


Extensions

  • Using more than three images

  • Propagation:

    • Using multi-neighbours

    • Smart propagation

  • Second error approximation

  • Error correction:

    • Based on local continuity

    • Other assumptions on the surface

Y. Moses


Real Images

  • Camera calibration

  • Light calibration

    • Direction

    • Intensity

    • Ambient

Y. Moses


Error correction + multi-neighbor5 Images

Y. Moses


5pp

5nn

5pn

3pp

3nn

Y. Moses


Y. Moses


Y. Moses


Y. Moses


Y. Moses


Y. Moses


Detected Correspondence

Y. Moses


Error correction + multi-neighbord

Multi-neighbors

Basic scheme (3 images)

Error correction no multi-neighbors

Y. Moses


Synthetic Images

New Images

Y. Moses


Ground truth

Basic scheme

Multi-neighbors Error correction

Sec a

Y. Moses


Ground truth

Basic scheme

Multi-neighbors Error correction

Sec b

Y. Moses


Ground truth

Basic scheme

Multi-neighbors Error correction

Sec c

Y. Moses


Ground truth

Basic scheme

Multi-neighbors Error correction

Sec d

Ground truth

Basic scheme

Multi-neighbors approx.

Error correction

Y. Moses


Combining Photometry and Geometry

Yields a dense correspondence and

dense shape recovery of the object in a single path

Y. Moses


Assumptions

  • Bilaterally Symmetric object

  • Lambertian surface with constant albedo

  • Orthographic projection

  • Neither occlusions nor shadows

  • Known “epipolar geometry”

Y. Moses


Geometric Stereo

  • 2 different images

  • Known camera parameters

  • Known viewpoints

  • Known correspondence

3D shape recovery

Y. Moses


Z

Computing the Depth from Disparity

P

OrthographicProjection

qr

Z

pr

pl

ql

Y. Moses


Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses


Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses


Geometry

  • Weak perspective projection:

Around X

Around Z

Around Y

Y. Moses


Geometry

  • Projection of Ry:

  • is the only pose parameter

Around Y

Image point

Object point

Y. Moses


Correspondence

Assume YxZ is the symmetry plane.

image

x

object

z

Y. Moses


Mono-Geometric Stereo

  • 3D reconstruction:

    given correspondence and ,

image

x object

known

z

unknown

Y. Moses


Viewpoint Invariant

  • Given the correspondence and unknown 

Invariant

Y. Moses


Photometric Stereo

  • 2 images

  • Lambertian reflectance

  • Known illuminations

  • Known correspondence

    (same viewpoint)

3D shape recovery

Y. Moses


Symmetry and Photometric Stereo

Non-frontal illumination of

a symmetric object

Two different images of the same object

Y. Moses


Notation: Photometry

  • Corresponding object points:

  • Illumination:

Y. Moses


Mono-Photometric Stereo

  • 3D reconstruction given correspondence and E (up to a twofold ambiguity):

known

unknown

Y. Moses


Invariance to Illumination

  • Given correspondence and E unknown

  • Invariant:

Y. Moses


Mono-Photometric Stereo

  • 3D reconstruction E unknown but correspondence is given

  • Frontal viewpoint with

    non-frontal illumination.

  • Use image first derivatives.

Y. Moses


Mono-Photometric Stereo Using image derivatives

  • 3 global unknowns: E

  • For each pair:

    • 5 unknowns zxzy zxx zxy zyy

    • 6 equations

  • 3 pairs are sufficient

Y. Moses


Mono-Photometric StereoUnknown Illumination

Y. Moses


Correspondence

  • No correspondence => no stereo.

  • Hard to define correspondence in images of smooth surfaces.

  • Almost any correspondence is legal when:

    • Only geometric constraints are considered.

    • Only photometric constraints are considered.

Y. Moses


Combining Photometry and Geometry

  • Yields a dense correspondence

    (dense shape recovery of the object).

  • Enables recovering of the global parameters.

Y. Moses


Self-Correspondence

  • A self-correspondence function:

Y. Moses


Dense Correspondence using Propagation

Assume correspondence between a pair of points, p0land p0r.

Y. Moses


Dense Correspondence using Propagation

Y. Moses


image

object

x

z

Y. Moses


First derivatives of the Correspondence

  • Assume known 

  • Assume known E

Y. Moses


Computing and

  • Object coordinates:

    Given computing and is trivial

  • Moving from object to image coordinates depends on the viewing parameter 

Y. Moses


  • Derivatives with respect to the object coordinates:

  • Derivatives with respect to the image coordinates:

Y. Moses


E

image

object

x

z

Y. Moses


General Idea

  • Given a corresponding pair and E

    n=(zx,zy,-1)T

  • Given  and n

    cxand cy

  • Given cxand cy

     a new corresponding pair

Y. Moses


Results on Real Images: Given global parameters

Y. Moses


Finding Global Parameters

  • Assume E and  are unknown.

  • Assume a pair of corresponding points is given.

  • Two possibilities:

    • Search for Eand directly.

    • ComputeE andfrom the image second derivatives.

Y. Moses


Integration Constraint:Circular Tour

  • All roads lead to Rome …

  • Find and verify correct correspondence

  • Recover global parameters, E and 

Y. Moses


Finding Global Parameters

Consider image second derivatives

  • Due to foreshortening effect:

    and

  • We can relate image and object derivatives by

Y. Moses


Testing E and : Image second derivatives

For each corresponding pair:

and

Plus 4 linear equations in 3 unknown.

Where

Y. Moses


Counting

  • 5 unknowns for each pair:

    zx zy,zxx zxy zyy

  • 4 global unknowns: E, 

  • For each pair: 6 equations.

  • For n pairs: 5n+4 unknowns

    6n equations.

    4 pairs are sufficient

Y. Moses


Results on Simulated Data

Ground Truth

Recovered Shape

Y. Moses


Recovering the Global Parameters

Y. Moses


Degenerate Case

  • Close to frontal view: problems with geometric-stereo.

    reconstruction problem

  • Close to frontal illumination: problems with photometric-stereo.

    correspondence problem

Y. Moses


Future work

  • Perspective photometric stereo

  • Use as a first approximation to global optimization methods

  • Test on other reflection models

  • Recovering of the global parameters:

    • Light

    • Cameras

  • Detect the first pair of correspondence

Y. Moses


Future Work

  • Extend to general 3 images under 3 viewpoints and 3 illuminations.

  • Extend to non-lambertian surfaces.

Y. Moses


Thanks

Y. Moses


image

object

x

z

Y. Moses


Integration Constraint

Y. Moses


Integration Constraint

Y. Moses


Searching for E

  • Illumination must satisfy:

  • E is further constrained by the image second derivatives.

Y. Moses


Where

Image second derivatives:

4 linear equations in 3 unknown

Y. Moses


Image second derivatives

For each corresponding pair and E: 4 linear equations in 3 unknown.

Where

Y. Moses


Counting

  • 5 unknowns for each pair: zx,zy,zxx,zxy,zyy

  • 3 global unknowns: E

  • For each pair: 6 equations.

  • For n pairs: 5n+3 unknowns

    6n equations.

    3 pairs are sufficient

Y. Moses


Correspondence

Y. Moses


Variations

  • Known/unknown distant light source

  • Known/unknown viewpoint

  • Symmetric/non-symmetric image

    • Frontal/non-frontal viewpoint

    • Frontal/non-frontal illumination

Y. Moses


Correspondence

  • Epipolar geometry is the only geometric constraint on the correspondence.

  • Weak photometric constraint on the correspondence.

Y. Moses


E

n2

n1

E

Lambertian Surface

I=

Basic radiometric

*

E

*

P

5

Y. Moses


E

Photometric Stereo

  • First proposed by Woodham, 1980.

  • Assume that we have two images ..

Y. Moses


  • Login