combining photometric and geometric constraints
Download
Skip this Video
Download Presentation
Combining Photometric and Geometric Constraints

Loading in 2 Seconds...

play fullscreen
1 / 104

Combining Photometric and Geometric Constraints - PowerPoint PPT Presentation


  • 64 Views
  • Uploaded on

Combining Photometric and Geometric Constraints. Yael Moses IDC, Herzliya. Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion. Problem 1:. Recover the 3D shape of a general smooth surface from a set of calibrated images. Problem 2:.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Combining Photometric and Geometric Constraints' - hadley-fowler


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
combining photometric and geometric constraints
Combining Photometric and Geometric Constraints

Yael Moses

IDC, Herzliya

Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion

Y. Moses

slide2

Problem 1:

  • Recover the 3D shape of a general smooth surface from a set of calibrated images

Y. Moses

slide3

Problem 2:

Recover the 3D shape of a smooth bilaterally symmetric object from a single image.

Y. Moses

shape recovery
Shape Recovery
  • Geometry: Stereo
  • Photometry:
    • Shape from shading
    • Photometric stereo

Main problems:

Calibrations and Correspondence

Y. Moses

3d shape recovery
3D Shape Recovery

Photometry:

  • Shape from shading
  • Photometric stereo

Geometry:

  • Stereo
  • Structure from motion

Y. Moses

geometric stereo
Geometric Stereo
  • 2 different images
  • Known camera parameters
  • Known correspondence

+

+

Y. Moses

photometric stereo
Photometric Stereo
  • 3D shape recovery:

surface normals from two or more images taken from the same viewpoint

Y. Moses

photometric stereo1

Three images

Photometric Stereo

Solution:

Matrix notation

Y. Moses

photometric stereo2
Photometric Stereo

Main Limitation:

Correspondence is obtained by a fixed viewpoint

  • 3D shape recovery (surface normals)

Two or more images taken from the same viewpoint

Y. Moses

overview
Overview
  • Combining photometric and geometric stereo:
    • Symmetric surface, single image
    • Non symmetric: 3 images
  • Mono-Geometric stereo
  • Mono-Photometric stereo
  • Experimental results.

Y. Moses

slide11

The input

  • Smooth featureless surface
  • Taken under different viewpoints
  • Illuminated by different light sources
  • The Problem:
  • Recover the 3D shape from a set of calibrated images

Y. Moses

assumptions

n

n

*

  • Perspective projection
Assumptions
  • Three or more images
  • Given correspondence the normals can be computed (e.g., Lambertian, distant point light source …)

*

Y. Moses

our method
Our method

Combines photometric and geometric stereo

We make use of:

  • Given Correspondence:
    • Can compute a normal
    • Can compute the 3D point

Y. Moses

slide14

Basic Method

Given

Correspondence

Y. Moses

slide17

P() = (1 - )O1+ P,

N(P() - P) = 0

First Order Surface Approximation

Y. Moses

basic method first order
Basic method: First Order
  • Given correspondence pi and L

Pand n

  • Given P andn

T

  • Given P, T andMi

 a new correspondence qi

Y. Moses

extensions
Extensions
  • Using more than three images
  • Propagation:
    • Using multi-neighbours
    • Smart propagation
  • Second error approximation
  • Error correction:
    • Based on local continuity
    • Other assumptions on the surface

Y. Moses

slide28

Second Order: a Sphere

(P-P())(N+N)=0

N

P()

P

N+N

N

Y. Moses

using more than three images
Using more than three images
  • Reduce noise of the photometric stereo
  • Avoid shadowed pixels
  • Detect “bad pixels”
    • Noise
    • Shadows
    • Violation of assumptions on the surface

Y. Moses

error correction
Error correction

The compatibility of the local 3D shape can be used to correct errors of:

  • Correspondence
  • Camera parameters
  • Illumination parameters

Y. Moses

score
Score
  • Continuity:
    • Shape
    • Normals
    • Albedo
  • The consistency of 3D points locations and the computed normals:
    • General case: full triangulation
    • Local constraints

Y. Moses

extensions1
Extensions
  • Using more than three images
  • Propagation:
    • Using multi-neighbours
    • Smart propagation
  • Second error approximation
  • Error correction:
    • Based on local continuity
    • Other assumptions on the surface

Y. Moses

real images
Real Images
  • Camera calibration
  • Light calibration
    • Direction
    • Intensity
    • Ambient

Y. Moses

slide38

5pp

5nn

5pn

3pp

3nn

Y. Moses

slide45

Error correction + multi-neighbord

Multi-neighbors

Basic scheme (3 images)

Error correction no multi-neighbors

Y. Moses

slide47

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec a

Y. Moses

slide48

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec b

Y. Moses

slide49

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec c

Y. Moses

slide50

Ground truth

Basic scheme

Multi-neighbors Error correction

Sec d

Ground truth

Basic scheme

Multi-neighbors approx.

Error correction

Y. Moses

combining photometry and geometry
Combining Photometry and Geometry

Yields a dense correspondence and

dense shape recovery of the object in a single path

Y. Moses

assumptions1
Assumptions
  • Bilaterally Symmetric object
  • Lambertian surface with constant albedo
  • Orthographic projection
  • Neither occlusions nor shadows
  • Known “epipolar geometry”

Y. Moses

geometric stereo1
Geometric Stereo
  • 2 different images
  • Known camera parameters
  • Known viewpoints
  • Known correspondence

3D shape recovery

Y. Moses

computing the depth from disparity

Z

Computing the Depth from Disparity

P

OrthographicProjection

qr

Z

pr

pl

ql

Y. Moses

symmetry and geometric stereo
Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses

symmetry and geometric stereo1
Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses

geometry
Geometry
  • Weak perspective projection:

Around X

Around Z

Around Y

Y. Moses

geometry1
Geometry
  • Projection of Ry:
  • is the only pose parameter

Around Y

Image point

Object point

Y. Moses

slide59

Correspondence

Assume YxZ is the symmetry plane.

image

x

object

z

Y. Moses

mono geometric stereo
Mono-Geometric Stereo
  • 3D reconstruction:

given correspondence and ,

image

x object

known

z

unknown

Y. Moses

viewpoint invariant
Viewpoint Invariant
  • Given the correspondence and unknown 

Invariant

Y. Moses

photometric stereo3
Photometric Stereo
  • 2 images
  • Lambertian reflectance
  • Known illuminations
  • Known correspondence

(same viewpoint)

3D shape recovery

Y. Moses

symmetry and photometric stereo
Symmetry and Photometric Stereo

Non-frontal illumination of

a symmetric object

Two different images of the same object

Y. Moses

notation photometry
Notation: Photometry
  • Corresponding object points:
  • Illumination:

Y. Moses

mono photometric stereo
Mono-Photometric Stereo
  • 3D reconstruction given correspondence and E (up to a twofold ambiguity):

known

unknown

Y. Moses

invariance to illumination
Invariance to Illumination
  • Given correspondence and E unknown
  • Invariant:

Y. Moses

mono photometric stereo1
Mono-Photometric Stereo
  • 3D reconstruction E unknown but correspondence is given
  • Frontal viewpoint with

non-frontal illumination.

  • Use image first derivatives.

Y. Moses

mono photometric stereo using image derivatives
Mono-Photometric Stereo Using image derivatives
  • 3 global unknowns: E
  • For each pair:
    • 5 unknowns zxzy zxx zxy zyy
    • 6 equations
  • 3 pairs are sufficient

Y. Moses

correspondence
Correspondence
  • No correspondence => no stereo.
  • Hard to define correspondence in images of smooth surfaces.
  • Almost any correspondence is legal when:
    • Only geometric constraints are considered.
    • Only photometric constraints are considered.

Y. Moses

combining photometry and geometry1
Combining Photometry and Geometry
  • Yields a dense correspondence

(dense shape recovery of the object).

  • Enables recovering of the global parameters.

Y. Moses

self correspondence
Self-Correspondence
  • A self-correspondence function:

Y. Moses

dense correspondence using propagation
Dense Correspondence using Propagation

Assume correspondence between a pair of points, p0land p0r.

Y. Moses

slide75

image

object

x

z

Y. Moses

first derivatives of the correspondence
First derivatives of the Correspondence
  • Assume known 
  • Assume known E

Y. Moses

computing and
Computing and
  • Object coordinates:

Given computing and is trivial

  • Moving from object to image coordinates depends on the viewing parameter 

Y. Moses

slide78
Derivatives with respect to the object coordinates:
  • Derivatives with respect to the image coordinates:

Y. Moses

slide79

E

image

object

x

z

Y. Moses

general idea
General Idea
  • Given a corresponding pair and E

n=(zx,zy,-1)T

  • Given  and n

cxand cy

  • Given cxand cy

 a new corresponding pair

Y. Moses

finding global parameters
Finding Global Parameters
  • Assume E and  are unknown.
  • Assume a pair of corresponding points is given.
  • Two possibilities:
    • Search for Eand directly.
    • ComputeE andfrom the image second derivatives.

Y. Moses

integration constraint circular tour
Integration Constraint:Circular Tour
  • All roads lead to Rome …
  • Find and verify correct correspondence
  • Recover global parameters, E and 

Y. Moses

finding global parameters1
Finding Global Parameters

Consider image second derivatives

  • Due to foreshortening effect:

and

  • We can relate image and object derivatives by

Y. Moses

testing e and image second derivatives
Testing E and : Image second derivatives

For each corresponding pair:

and

Plus 4 linear equations in 3 unknown.

Where

Y. Moses

counting
Counting
  • 5 unknowns for each pair:

zx zy,zxx zxy zyy

  • 4 global unknowns: E, 
  • For each pair: 6 equations.
  • For n pairs: 5n+4 unknowns

6n equations.

4 pairs are sufficient

Y. Moses

results on simulated data
Results on Simulated Data

Ground Truth

Recovered Shape

Y. Moses

degenerate case
Degenerate Case
  • Close to frontal view: problems with geometric-stereo.

reconstruction problem

  • Close to frontal illumination: problems with photometric-stereo.

correspondence problem

Y. Moses

future work
Future work
  • Perspective photometric stereo
  • Use as a first approximation to global optimization methods
  • Test on other reflection models
  • Recovering of the global parameters:
    • Light
    • Cameras
  • Detect the first pair of correspondence

Y. Moses

future work1
Future Work
  • Extend to general 3 images under 3 viewpoints and 3 illuminations.
  • Extend to non-lambertian surfaces.

Y. Moses

thanks
Thanks

Y. Moses

slide93

image

object

x

z

Y. Moses

searching for e
Searching for E
  • Illumination must satisfy:
  • E is further constrained by the image second derivatives.

Y. Moses

image second derivatives

Where

Image second derivatives:

4 linear equations in 3 unknown

Y. Moses

image second derivatives1
Image second derivatives

For each corresponding pair and E: 4 linear equations in 3 unknown.

Where

Y. Moses

counting1
Counting
  • 5 unknowns for each pair: zx,zy,zxx,zxy,zyy
  • 3 global unknowns: E
  • For each pair: 6 equations.
  • For n pairs: 5n+3 unknowns

6n equations.

3 pairs are sufficient

Y. Moses

variations
Variations
  • Known/unknown distant light source
  • Known/unknown viewpoint
  • Symmetric/non-symmetric image
    • Frontal/non-frontal viewpoint
    • Frontal/non-frontal illumination

Y. Moses

correspondence2
Correspondence
  • Epipolar geometry is the only geometric constraint on the correspondence.
  • Weak photometric constraint on the correspondence.

Y. Moses

lambertian surface

E

n2

n1

E

Lambertian Surface

I=

Basic radiometric

*

E

*

P

5

Y. Moses

photometric stereo4

E

Photometric Stereo
  • First proposed by Woodham, 1980.
  • Assume that we have two images ..

Y. Moses

ad