Combining photometric and geometric constraints
This presentation is the property of its rightful owner.
Sponsored Links
1 / 104

Combining Photometric and Geometric Constraints PowerPoint PPT Presentation


  • 44 Views
  • Uploaded on
  • Presentation posted in: General

Combining Photometric and Geometric Constraints. Yael Moses IDC, Herzliya. Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion. Problem 1:. Recover the 3D shape of a general smooth surface from a set of calibrated images. Problem 2:.

Download Presentation

Combining Photometric and Geometric Constraints

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Combining photometric and geometric constraints

Combining Photometric and Geometric Constraints

Yael Moses

IDC, Herzliya

Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion

Y. Moses


Combining photometric and geometric constraints

Problem 1:

  • Recover the 3D shape of a general smooth surface from a set of calibrated images

Y. Moses


Combining photometric and geometric constraints

Problem 2:

Recover the 3D shape of a smooth bilaterally symmetric object from a single image.

Y. Moses


Shape recovery

Shape Recovery

  • Geometry: Stereo

  • Photometry:

    • Shape from shading

    • Photometric stereo

      Main problems:

      Calibrations and Correspondence

Y. Moses


3d shape recovery

3D Shape Recovery

Photometry:

  • Shape from shading

  • Photometric stereo

Geometry:

  • Stereo

  • Structure from motion

Y. Moses


Geometric stereo

Geometric Stereo

  • 2 different images

  • Known camera parameters

  • Known correspondence

+

+

Y. Moses


Photometric stereo

Photometric Stereo

  • 3D shape recovery:

    surface normals from two or more images taken from the same viewpoint

Y. Moses


Photometric stereo1

Three images

Photometric Stereo

Solution:

Matrix notation

Y. Moses


Photometric stereo2

Photometric Stereo

Main Limitation:

Correspondence is obtained by a fixed viewpoint

  • 3D shape recovery (surface normals)

    Two or more images taken from the same viewpoint

Y. Moses


Overview

Overview

  • Combining photometric and geometric stereo:

    • Symmetric surface, single image

    • Non symmetric: 3 images

  • Mono-Geometric stereo

  • Mono-Photometric stereo

  • Experimental results.

Y. Moses


Combining photometric and geometric constraints

The input

  • Smooth featureless surface

  • Taken under different viewpoints

  • Illuminated by different light sources

  • The Problem:

  • Recover the 3D shape from a set of calibrated images

Y. Moses


Assumptions

n

n

*

  • Perspective projection

Assumptions

  • Three or more images

  • Given correspondence the normals can be computed (e.g., Lambertian, distant point light source …)

*

Y. Moses


Our method

Our method

Combines photometric and geometric stereo

We make use of:

  • Given Correspondence:

    • Can compute a normal

    • Can compute the 3D point

Y. Moses


Combining photometric and geometric constraints

Basic Method

Given

Correspondence

Y. Moses


Combining photometric and geometric constraints

First Order Surface Approximation

Y. Moses


Combining photometric and geometric constraints

First Order Surface Approximation

Y. Moses


Combining photometric and geometric constraints

P() = (1 - )O1+ P,

N(P() - P) = 0

First Order Surface Approximation

Y. Moses


Combining photometric and geometric constraints

First Order Surface Approximation

Y. Moses


Combining photometric and geometric constraints

New Correspondence

Y. Moses


Combining photometric and geometric constraints

New Surface Approximation

Y. Moses


Combining photometric and geometric constraints

Dense Correspondence

Y. Moses


Basic propagation

Basic Propagation

Y. Moses


Basic propagation1

Basic Propagation

Y. Moses


Basic method first order

Basic method: First Order

  • Given correspondence pi and L

    Pand n

  • Given P andn

    T

  • Given P, T andMi

    a new correspondence qi

Y. Moses


Extensions

Extensions

  • Using more than three images

  • Propagation:

    • Using multi-neighbours

    • Smart propagation

  • Second error approximation

  • Error correction:

    • Based on local continuity

    • Other assumptions on the surface

Y. Moses


Multi neighbors propagation

Multi-neighbors Propagation

Y. Moses


Smart propagation

Smart Propagation

Y. Moses


Combining photometric and geometric constraints

Second Order: a Sphere

(P-P())(N+N)=0

N

P()

P

N+N

N

Y. Moses


Combining photometric and geometric constraints

Second Order Approximation

Y. Moses


Combining photometric and geometric constraints

Second Order Approximation

Y. Moses


Using more than three images

Using more than three images

  • Reduce noise of the photometric stereo

  • Avoid shadowed pixels

  • Detect “bad pixels”

    • Noise

    • Shadows

    • Violation of assumptions on the surface

Y. Moses


Smart propagation1

Smart Propagation

Y. Moses


Error correction

Error correction

The compatibility of the local 3D shape can be used to correct errors of:

  • Correspondence

  • Camera parameters

  • Illumination parameters

Y. Moses


Score

Score

  • Continuity:

    • Shape

    • Normals

    • Albedo

  • The consistency of 3D points locations and the computed normals:

    • General case: full triangulation

    • Local constraints

Y. Moses


Extensions1

Extensions

  • Using more than three images

  • Propagation:

    • Using multi-neighbours

    • Smart propagation

  • Second error approximation

  • Error correction:

    • Based on local continuity

    • Other assumptions on the surface

Y. Moses


Real images

Real Images

  • Camera calibration

  • Light calibration

    • Direction

    • Intensity

    • Ambient

Y. Moses


Error correction multi neighbor 5 images

Error correction + multi-neighbor5 Images

Y. Moses


Combining photometric and geometric constraints

5pp

5nn

5pn

3pp

3nn

Y. Moses


Combining photometric and geometric constraints

Y. Moses


Combining photometric and geometric constraints

Y. Moses


Combining photometric and geometric constraints

Y. Moses


Combining photometric and geometric constraints

Y. Moses


Combining photometric and geometric constraints

Y. Moses


Detected correspondence

Detected Correspondence

Y. Moses


Combining photometric and geometric constraints

Error correction + multi-neighbord

Multi-neighbors

Basic scheme (3 images)

Error correction no multi-neighbors

Y. Moses


New images

Synthetic Images

New Images

Y. Moses


Combining photometric and geometric constraints

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec a

Y. Moses


Combining photometric and geometric constraints

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec b

Y. Moses


Combining photometric and geometric constraints

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec c

Y. Moses


Combining photometric and geometric constraints

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec d

Ground truth

Basic scheme

Multi-neighbors approx.

Error correction

Y. Moses


Combining photometry and geometry

Combining Photometry and Geometry

Yields a dense correspondence and

dense shape recovery of the object in a single path

Y. Moses


Assumptions1

Assumptions

  • Bilaterally Symmetric object

  • Lambertian surface with constant albedo

  • Orthographic projection

  • Neither occlusions nor shadows

  • Known “epipolar geometry”

Y. Moses


Geometric stereo1

Geometric Stereo

  • 2 different images

  • Known camera parameters

  • Known viewpoints

  • Known correspondence

3D shape recovery

Y. Moses


Computing the depth from disparity

Z

Computing the Depth from Disparity

P

OrthographicProjection

qr

Z

pr

pl

ql

Y. Moses


Symmetry and geometric stereo

Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses


Symmetry and geometric stereo1

Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses


Geometry

Geometry

  • Weak perspective projection:

Around X

Around Z

Around Y

Y. Moses


Geometry1

Geometry

  • Projection of Ry:

  • is the only pose parameter

Around Y

Image point

Object point

Y. Moses


Combining photometric and geometric constraints

Correspondence

Assume YxZ is the symmetry plane.

image

x

object

z

Y. Moses


Mono geometric stereo

Mono-Geometric Stereo

  • 3D reconstruction:

    given correspondence and ,

image

x object

known

z

unknown

Y. Moses


Viewpoint invariant

Viewpoint Invariant

  • Given the correspondence and unknown 

Invariant

Y. Moses


Photometric stereo3

Photometric Stereo

  • 2 images

  • Lambertian reflectance

  • Known illuminations

  • Known correspondence

    (same viewpoint)

3D shape recovery

Y. Moses


Symmetry and photometric stereo

Symmetry and Photometric Stereo

Non-frontal illumination of

a symmetric object

Two different images of the same object

Y. Moses


Notation photometry

Notation: Photometry

  • Corresponding object points:

  • Illumination:

Y. Moses


Mono photometric stereo

Mono-Photometric Stereo

  • 3D reconstruction given correspondence and E (up to a twofold ambiguity):

known

unknown

Y. Moses


Invariance to illumination

Invariance to Illumination

  • Given correspondence and E unknown

  • Invariant:

Y. Moses


Mono photometric stereo1

Mono-Photometric Stereo

  • 3D reconstruction E unknown but correspondence is given

  • Frontal viewpoint with

    non-frontal illumination.

  • Use image first derivatives.

Y. Moses


Mono photometric stereo using image derivatives

Mono-Photometric Stereo Using image derivatives

  • 3 global unknowns: E

  • For each pair:

    • 5 unknowns zxzy zxx zxy zyy

    • 6 equations

  • 3 pairs are sufficient

Y. Moses


Mono photometric stereo unknown illumination

Mono-Photometric StereoUnknown Illumination

Y. Moses


Correspondence

Correspondence

  • No correspondence => no stereo.

  • Hard to define correspondence in images of smooth surfaces.

  • Almost any correspondence is legal when:

    • Only geometric constraints are considered.

    • Only photometric constraints are considered.

Y. Moses


Combining photometry and geometry1

Combining Photometry and Geometry

  • Yields a dense correspondence

    (dense shape recovery of the object).

  • Enables recovering of the global parameters.

Y. Moses


Self correspondence

Self-Correspondence

  • A self-correspondence function:

Y. Moses


Dense correspondence using propagation

Dense Correspondence using Propagation

Assume correspondence between a pair of points, p0land p0r.

Y. Moses


Dense correspondence using propagation1

Dense Correspondence using Propagation

Y. Moses


Combining photometric and geometric constraints

image

object

x

z

Y. Moses


First derivatives of the correspondence

First derivatives of the Correspondence

  • Assume known 

  • Assume known E

Y. Moses


Computing and

Computing and

  • Object coordinates:

    Given computing and is trivial

  • Moving from object to image coordinates depends on the viewing parameter 

Y. Moses


Combining photometric and geometric constraints

  • Derivatives with respect to the object coordinates:

  • Derivatives with respect to the image coordinates:

Y. Moses


Combining photometric and geometric constraints

E

image

object

x

z

Y. Moses


General idea

General Idea

  • Given a corresponding pair and E

    n=(zx,zy,-1)T

  • Given  and n

    cxand cy

  • Given cxand cy

     a new corresponding pair

Y. Moses


Results on real images given global parameters

Results on Real Images: Given global parameters

Y. Moses


Finding global parameters

Finding Global Parameters

  • Assume E and  are unknown.

  • Assume a pair of corresponding points is given.

  • Two possibilities:

    • Search for Eand directly.

    • ComputeE andfrom the image second derivatives.

Y. Moses


Integration constraint circular tour

Integration Constraint:Circular Tour

  • All roads lead to Rome …

  • Find and verify correct correspondence

  • Recover global parameters, E and 

Y. Moses


Finding global parameters1

Finding Global Parameters

Consider image second derivatives

  • Due to foreshortening effect:

    and

  • We can relate image and object derivatives by

Y. Moses


Testing e and image second derivatives

Testing E and : Image second derivatives

For each corresponding pair:

and

Plus 4 linear equations in 3 unknown.

Where

Y. Moses


Counting

Counting

  • 5 unknowns for each pair:

    zx zy,zxx zxy zyy

  • 4 global unknowns: E, 

  • For each pair: 6 equations.

  • For n pairs: 5n+4 unknowns

    6n equations.

    4 pairs are sufficient

Y. Moses


Results on simulated data

Results on Simulated Data

Ground Truth

Recovered Shape

Y. Moses


Recovering the global parameters

Recovering the Global Parameters

Y. Moses


Degenerate case

Degenerate Case

  • Close to frontal view: problems with geometric-stereo.

    reconstruction problem

  • Close to frontal illumination: problems with photometric-stereo.

    correspondence problem

Y. Moses


Future work

Future work

  • Perspective photometric stereo

  • Use as a first approximation to global optimization methods

  • Test on other reflection models

  • Recovering of the global parameters:

    • Light

    • Cameras

  • Detect the first pair of correspondence

Y. Moses


Future work1

Future Work

  • Extend to general 3 images under 3 viewpoints and 3 illuminations.

  • Extend to non-lambertian surfaces.

Y. Moses


Thanks

Thanks

Y. Moses


Combining photometric and geometric constraints

image

object

x

z

Y. Moses


Integration constraint

Integration Constraint

Y. Moses


Integration constraint1

Integration Constraint

Y. Moses


Searching for e

Searching for E

  • Illumination must satisfy:

  • E is further constrained by the image second derivatives.

Y. Moses


Image second derivatives

Where

Image second derivatives:

4 linear equations in 3 unknown

Y. Moses


Image second derivatives1

Image second derivatives

For each corresponding pair and E: 4 linear equations in 3 unknown.

Where

Y. Moses


Counting1

Counting

  • 5 unknowns for each pair: zx,zy,zxx,zxy,zyy

  • 3 global unknowns: E

  • For each pair: 6 equations.

  • For n pairs: 5n+3 unknowns

    6n equations.

    3 pairs are sufficient

Y. Moses


Correspondence1

Correspondence

Y. Moses


Variations

Variations

  • Known/unknown distant light source

  • Known/unknown viewpoint

  • Symmetric/non-symmetric image

    • Frontal/non-frontal viewpoint

    • Frontal/non-frontal illumination

Y. Moses


Correspondence2

Correspondence

  • Epipolar geometry is the only geometric constraint on the correspondence.

  • Weak photometric constraint on the correspondence.

Y. Moses


Lambertian surface

E

n2

n1

E

Lambertian Surface

I=

Basic radiometric

*

E

*

P

5

Y. Moses


Photometric stereo4

E

Photometric Stereo

  • First proposed by Woodham, 1980.

  • Assume that we have two images ..

Y. Moses


  • Login