Combining photometric and geometric constraints
Download
1 / 104

Combining Photometric and Geometric Constraints - PowerPoint PPT Presentation


  • 62 Views
  • Uploaded on

Combining Photometric and Geometric Constraints. Yael Moses IDC, Herzliya. Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion. Problem 1:. Recover the 3D shape of a general smooth surface from a set of calibrated images. Problem 2:.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Combining Photometric and Geometric Constraints' - hadley-fowler


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Combining photometric and geometric constraints
Combining Photometric and Geometric Constraints

Yael Moses

IDC, Herzliya

Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion

Y. Moses


Problem 1:

  • Recover the 3D shape of a general smooth surface from a set of calibrated images

Y. Moses


Problem 2:

Recover the 3D shape of a smooth bilaterally symmetric object from a single image.

Y. Moses


Shape recovery
Shape Recovery

  • Geometry: Stereo

  • Photometry:

    • Shape from shading

    • Photometric stereo

      Main problems:

      Calibrations and Correspondence

Y. Moses


3d shape recovery
3D Shape Recovery

Photometry:

  • Shape from shading

  • Photometric stereo

Geometry:

  • Stereo

  • Structure from motion

Y. Moses


Geometric stereo
Geometric Stereo

  • 2 different images

  • Known camera parameters

  • Known correspondence

+

+

Y. Moses


Photometric stereo
Photometric Stereo

  • 3D shape recovery:

    surface normals from two or more images taken from the same viewpoint

Y. Moses


Photometric stereo1

Three images

Photometric Stereo

Solution:

Matrix notation

Y. Moses


Photometric stereo2
Photometric Stereo

Main Limitation:

Correspondence is obtained by a fixed viewpoint

  • 3D shape recovery (surface normals)

    Two or more images taken from the same viewpoint

Y. Moses


Overview
Overview

  • Combining photometric and geometric stereo:

    • Symmetric surface, single image

    • Non symmetric: 3 images

  • Mono-Geometric stereo

  • Mono-Photometric stereo

  • Experimental results.

Y. Moses


The input

  • Smooth featureless surface

  • Taken under different viewpoints

  • Illuminated by different light sources

  • The Problem:

  • Recover the 3D shape from a set of calibrated images

Y. Moses


Assumptions

n

n

*

  • Perspective projection

Assumptions

  • Three or more images

  • Given correspondence the normals can be computed (e.g., Lambertian, distant point light source …)

*

Y. Moses


Our method
Our method

Combines photometric and geometric stereo

We make use of:

  • Given Correspondence:

    • Can compute a normal

    • Can compute the 3D point

Y. Moses


Basic Method

Given

Correspondence

Y. Moses




P() = (1 - )O1+ P,

N(P() - P) = 0

First Order Surface Approximation

Y. Moses








Basic method first order
Basic method: First Order

  • Given correspondence pi and L

    Pand n

  • Given P andn

    T

  • Given P, T andMi

     a new correspondence qi

Y. Moses


Extensions
Extensions

  • Using more than three images

  • Propagation:

    • Using multi-neighbours

    • Smart propagation

  • Second error approximation

  • Error correction:

    • Based on local continuity

    • Other assumptions on the surface

Y. Moses




Second Order: a Sphere

(P-P())(N+N)=0

N

P()

P

N+N

N

Y. Moses




Using more than three images
Using more than three images

  • Reduce noise of the photometric stereo

  • Avoid shadowed pixels

  • Detect “bad pixels”

    • Noise

    • Shadows

    • Violation of assumptions on the surface

Y. Moses



Error correction
Error correction

The compatibility of the local 3D shape can be used to correct errors of:

  • Correspondence

  • Camera parameters

  • Illumination parameters

Y. Moses


Score
Score

  • Continuity:

    • Shape

    • Normals

    • Albedo

  • The consistency of 3D points locations and the computed normals:

    • General case: full triangulation

    • Local constraints

Y. Moses


Extensions1
Extensions

  • Using more than three images

  • Propagation:

    • Using multi-neighbours

    • Smart propagation

  • Second error approximation

  • Error correction:

    • Based on local continuity

    • Other assumptions on the surface

Y. Moses


Real images
Real Images

  • Camera calibration

  • Light calibration

    • Direction

    • Intensity

    • Ambient

Y. Moses



5pp

5nn

5pn

3pp

3nn

Y. Moses








Error correction + multi-neighbord

Multi-neighbors

Basic scheme (3 images)

Error correction no multi-neighbors

Y. Moses


New images

Synthetic Images

New Images

Y. Moses


Ground truth

Basic scheme

Multi-neighbors Error correction

Sec a

Y. Moses


Ground truth

Basic scheme

Multi-neighbors Error correction

Sec b

Y. Moses


Ground truth

Basic scheme

Multi-neighbors Error correction

Sec c

Y. Moses


Ground truth

Basic scheme

Multi-neighbors Error correction

Sec d

Ground truth

Basic scheme

Multi-neighbors approx.

Error correction

Y. Moses


Combining photometry and geometry
Combining Photometry and Geometry

Yields a dense correspondence and

dense shape recovery of the object in a single path

Y. Moses


Assumptions1
Assumptions

  • Bilaterally Symmetric object

  • Lambertian surface with constant albedo

  • Orthographic projection

  • Neither occlusions nor shadows

  • Known “epipolar geometry”

Y. Moses


Geometric stereo1
Geometric Stereo

  • 2 different images

  • Known camera parameters

  • Known viewpoints

  • Known correspondence

3D shape recovery

Y. Moses


Computing the depth from disparity

Z

Computing the Depth from Disparity

P

OrthographicProjection

qr

Z

pr

pl

ql

Y. Moses


Symmetry and geometric stereo
Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses


Symmetry and geometric stereo1
Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses


Geometry
Geometry

  • Weak perspective projection:

Around X

Around Z

Around Y

Y. Moses


Geometry1
Geometry

  • Projection of Ry:

  • is the only pose parameter

Around Y

Image point

Object point

Y. Moses


Correspondence

Assume YxZ is the symmetry plane.

image

x

object

z

Y. Moses


Mono geometric stereo
Mono-Geometric Stereo

  • 3D reconstruction:

    given correspondence and ,

image

x object

known

z

unknown

Y. Moses


Viewpoint invariant
Viewpoint Invariant

  • Given the correspondence and unknown 

Invariant

Y. Moses


Photometric stereo3
Photometric Stereo

  • 2 images

  • Lambertian reflectance

  • Known illuminations

  • Known correspondence

    (same viewpoint)

3D shape recovery

Y. Moses


Symmetry and photometric stereo
Symmetry and Photometric Stereo

Non-frontal illumination of

a symmetric object

Two different images of the same object

Y. Moses


Notation photometry
Notation: Photometry

  • Corresponding object points:

  • Illumination:

Y. Moses


Mono photometric stereo
Mono-Photometric Stereo

  • 3D reconstruction given correspondence and E (up to a twofold ambiguity):

known

unknown

Y. Moses


Invariance to illumination
Invariance to Illumination

  • Given correspondence and E unknown

  • Invariant:

Y. Moses


Mono photometric stereo1
Mono-Photometric Stereo

  • 3D reconstruction E unknown but correspondence is given

  • Frontal viewpoint with

    non-frontal illumination.

  • Use image first derivatives.

Y. Moses


Mono photometric stereo using image derivatives
Mono-Photometric Stereo Using image derivatives

  • 3 global unknowns: E

  • For each pair:

    • 5 unknowns zxzy zxx zxy zyy

    • 6 equations

  • 3 pairs are sufficient

Y. Moses


Mono photometric stereo unknown illumination
Mono-Photometric StereoUnknown Illumination

Y. Moses


Correspondence
Correspondence

  • No correspondence => no stereo.

  • Hard to define correspondence in images of smooth surfaces.

  • Almost any correspondence is legal when:

    • Only geometric constraints are considered.

    • Only photometric constraints are considered.

Y. Moses


Combining photometry and geometry1
Combining Photometry and Geometry

  • Yields a dense correspondence

    (dense shape recovery of the object).

  • Enables recovering of the global parameters.

Y. Moses


Self correspondence
Self-Correspondence

  • A self-correspondence function:

Y. Moses


Dense correspondence using propagation
Dense Correspondence using Propagation

Assume correspondence between a pair of points, p0land p0r.

Y. Moses



image

object

x

z

Y. Moses


First derivatives of the correspondence
First derivatives of the Correspondence

  • Assume known 

  • Assume known E

Y. Moses


Computing and
Computing and

  • Object coordinates:

    Given computing and is trivial

  • Moving from object to image coordinates depends on the viewing parameter 

Y. Moses


Y. Moses


E

image

object

x

z

Y. Moses


General idea
General Idea

  • Given a corresponding pair and E

    n=(zx,zy,-1)T

  • Given  and n

    cxand cy

  • Given cxand cy

     a new corresponding pair

Y. Moses


Results on real images given global parameters
Results on Real Images: Given global parameters

Y. Moses


Finding global parameters
Finding Global Parameters

  • Assume E and  are unknown.

  • Assume a pair of corresponding points is given.

  • Two possibilities:

    • Search for Eand directly.

    • ComputeE andfrom the image second derivatives.

Y. Moses


Integration constraint circular tour
Integration Constraint:Circular Tour

  • All roads lead to Rome …

  • Find and verify correct correspondence

  • Recover global parameters, E and 

Y. Moses


Finding global parameters1
Finding Global Parameters

Consider image second derivatives

  • Due to foreshortening effect:

    and

  • We can relate image and object derivatives by

Y. Moses


Testing e and image second derivatives
Testing E and : Image second derivatives

For each corresponding pair:

and

Plus 4 linear equations in 3 unknown.

Where

Y. Moses


Counting
Counting

  • 5 unknowns for each pair:

    zx zy,zxx zxy zyy

  • 4 global unknowns: E, 

  • For each pair: 6 equations.

  • For n pairs: 5n+4 unknowns

    6n equations.

    4 pairs are sufficient

Y. Moses


Results on simulated data
Results on Simulated Data

Ground Truth

Recovered Shape

Y. Moses



Degenerate case
Degenerate Case

  • Close to frontal view: problems with geometric-stereo.

    reconstruction problem

  • Close to frontal illumination: problems with photometric-stereo.

    correspondence problem

Y. Moses


Future work
Future work

  • Perspective photometric stereo

  • Use as a first approximation to global optimization methods

  • Test on other reflection models

  • Recovering of the global parameters:

    • Light

    • Cameras

  • Detect the first pair of correspondence

Y. Moses


Future work1
Future Work

  • Extend to general 3 images under 3 viewpoints and 3 illuminations.

  • Extend to non-lambertian surfaces.

Y. Moses


Thanks
Thanks

Y. Moses


image

object

x

z

Y. Moses




Searching for e
Searching for E

  • Illumination must satisfy:

  • E is further constrained by the image second derivatives.

Y. Moses


Image second derivatives

Where

Image second derivatives:

4 linear equations in 3 unknown

Y. Moses


Image second derivatives1
Image second derivatives

For each corresponding pair and E: 4 linear equations in 3 unknown.

Where

Y. Moses


Counting1
Counting

  • 5 unknowns for each pair: zx,zy,zxx,zxy,zyy

  • 3 global unknowns: E

  • For each pair: 6 equations.

  • For n pairs: 5n+3 unknowns

    6n equations.

    3 pairs are sufficient

Y. Moses


Correspondence1
Correspondence

Y. Moses


Variations
Variations

  • Known/unknown distant light source

  • Known/unknown viewpoint

  • Symmetric/non-symmetric image

    • Frontal/non-frontal viewpoint

    • Frontal/non-frontal illumination

Y. Moses


Correspondence2
Correspondence

  • Epipolar geometry is the only geometric constraint on the correspondence.

  • Weak photometric constraint on the correspondence.

Y. Moses


Lambertian surface

E

n2

n1

E

Lambertian Surface

I=

Basic radiometric

*

E

*

P

5

Y. Moses


Photometric stereo4

E

Photometric Stereo

  • First proposed by Woodham, 1980.

  • Assume that we have two images ..

Y. Moses


ad