slide1
Download
Skip this Video
Download Presentation
Optimal Illumination for Image and Video Relighting

Loading in 2 Seconds...

play fullscreen
1 / 165

Optimal Illumination for Image and Video Relighting - PowerPoint PPT Presentation


  • 95 Views
  • Uploaded on

Optimal Illumination for Image and Video Relighting. Francesc Moreno-Noguer Shree K. Nayar Peter N. Belhumeur. Department of Computer Science – Columbia University. European Conference on CVMP November 2005, London, UK. Objective. Objective. Objective. Objective.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Optimal Illumination for Image and Video Relighting' - benito


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Optimal Illumination for Image and Video Relighting

Francesc Moreno-Noguer Shree K. Nayar Peter N. Belhumeur

Department of Computer Science – Columbia University

European Conference on CVMP

November 2005, London, UK

relighting static objects26
Relighting Static Objects

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

?

relighting static objects27
Relighting Static Objects

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

relighting moving objects26
Relighting Moving Objects

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

?

relighting moving objects27
Relighting Moving Objects

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

relighting moving objects28
Relighting Moving Objects

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

x

relighting moving objects52
Relighting Moving Objects

=Ialign

= L

?

Lnew

Inew

sources of error in video relighting
Sources of error in video relighting

Sub-basis error

Ground Truth

sources of error in video relighting1

Rendered

3 basis

Error

3 basis

Sources of error in video relighting

Sub-basis error

Ground Truth

sources of error in video relighting2
Sources of error in video relighting

Sub-basis error

Ground Truth

Rendered

3 basis

Error

3 basis

sources of error in video relighting3

Rendered

6 basis

Error

6 basis

Sources of error in video relighting

Sub-basis error

Ground Truth

Rendered

3 basis

Error

3 basis

sources of error in video relighting4

Rendered

6 basis

Error

6 basis

Sources of error in video relighting

Sub-basis error

Ground Truth

Rendered

3 basis

Error

3 basis

sources of error in video relighting5
Sources of error in video relighting

Alignment error

Reference frame

sources of error in video relighting6
Sources of error in video relighting

Alignment error

Reference frame

Frame t+1

sources of error in video relighting7

Alignment errors

Sources of error in video relighting

Alignment error

Reference frame

Frame t+1

sources of error in video relighting8

Alignment errors

Sources of error in video relighting

Alignment error

Reference frame

Frame t+1

Frame t+2

sources of error in video relighting9

Alignment errors

Sources of error in video relighting

Alignment error

Reference frame

Frame t+1

Frame t+2

sources of error in video relighting10

Alignment errors

Sources of error in video relighting

Alignment error

Reference frame

Frame t+1

Frame t+2

Frame t+3

sources of error in video relighting11

Alignment errors

Sources of error in video relighting

Alignment error

Reference frame

Frame t+1

Frame t+2

Frame t+3

sources of error in video relighting12

Alignment errors

Sources of error in video relighting

Alignment error

Reference frame

Frame t+1

Frame t+2

Frame t+3

Frame t+4

sources of error in video relighting13

Alignment errors

Sources of error in video relighting

Alignment error

Reference frame

Frame t+1

Frame t+2

Frame t+3

Frame t+4

sources of error in video relighting14

Alignment errors

Sources of error in video relighting

Alignment error

Reference frame

Frame t+1

Frame t+2

Frame t+3

Frame t+4

Frame t+5

sources of error in video relighting15

Alignment errors

Sources of error in video relighting

Alignment error

Reference frame

Frame t+1

Frame t+2

Frame t+3

Frame t+4

Frame t+5

sources of error in video relighting18
Sources of error in video relighting

Orientation error

Reference frame

sources of error in video relighting19
Sources of error in video relighting

Orientation error

Reference frame

Perfectly

aligned frame

sources of error in video relighting20
Sources of error in video relighting

Orientation error

Reference frame

Perfectly

aligned frame

Error after

perfect

alignment

sources of error in video relighting21
Sources of error in video relighting

Sub-basis error

Alignment error

Orientation error

criterion for selecting the olb
Criterion for selecting the OLB

Minimize the sub-basis error

BUT

Keep the number of reference images to a minimum

object independent lighting bases
Object independent lighting bases

Spherical Harmonic basis (SHLB)

object independent lighting bases1
Object independent lighting bases

Spherical Harmonic basis (SHLB)

Fourier basis (FLB)

object independent lighting bases2
Object independent lighting bases

Spherical Harmonic basis (SHLB)

Fourier basis (FLB)

Haar basis (HaLB)

object dependent lighting basis

...

A=

Object dependent lighting basis

Optimal Lighting Basis (OLB)

  • Computed as a simple calibration procedure before acquisition
  • Initial acquisition of the images of the still object under single light sources
  • Compute SVD over the matrix of images
object dependent lighting basis4
Object dependent lighting basis

A=UDW

T

...

A=

...

U=

...

W =

T

object dependent lighting basis5
Object dependent lighting basis

A=UDW

T

...

A=

...

U=

...

W =

T

...

Lopt=

synthetic results
Synthetic results

Optimal Lighting Basis (OLB)

Spherical Hamonic (SHLB)

synthetic results1
Synthetic results

Optimal Lighting Basis (OLB)

Fourier (FLB)

Haar (HaLB)

Optimal Lighting Basis (OLB)

Spherical Hamonic (SHLB)

synthetic results static objects1
Synthetic results – Static objects

Ground Truth

Fourier 16 basis

Error

Fourier 16 basis

synthetic results static objects2
Synthetic results – Static objects

Ground Truth

Fourier 16 basis

OLB 16 basis

Error

Fourier 16 basis

Error

OLB 16 basis

synthetic results static objects4
Synthetic results – Static objects

Ground Truth

Haar 3 basis

Error

Haar 3 basis

synthetic results static objects5
Synthetic results – Static objects

Ground Truth

Haar 3 basis

OLB 3 basis

Error

Haar 3 basis

Error

OLB 3 basis

synthetic results static objects7
Synthetic results – Static objects

Ground Truth

Sph. Harm.

7 basis

Error

Sph. Harm. 7 basis

synthetic results static objects8
Synthetic results – Static objects

Ground Truth

Sph. Harm.

7 basis

OLB 7 basis

Error

Sph. Harm. 7 basis

Error

OLB 7 basis

synthetic results static objects10
Synthetic results – Static objects

Ground Truth

Sph. Harm. 3 basis

Error

Sph. Harm. 3 basis

synthetic results static objects11
Synthetic results – Static objects

Ground Truth

Sph. Harm. 3 basis

OLB 3 basis

Error

Sph. Harm. 3 basis

Error

OLB 3 basis

synthetic results static objects14
Synthetic results – Static objects

Dragon Example:

8 OLB = 8 x 1.6 SHLB 13 SHLB

synthetic results static objects15
Synthetic results – Static objects

Dragon Example:

8 OLB = 8 x 1.6 SHLB 13 SHLB

8 OLB = 8 x 1.9 SHLB 15 FLB

synthetic results static objects16
Synthetic results – Static objects

Dragon Example:

8 OLB = 8 x 1.6 SHLB 13 SHLB

8 OLB = 8 x 1.9 SHLB 15 FLB

8 OLB = 8 x 3.2 SHLB 25 HaLB

synthetic results moving objects
Synthetic results – Moving objects

OLB vs. SHLB

Frame #1

Frame #50

synthetic results moving objects1
Synthetic results – Moving objects

OLB vs. SHLB

Frame #1

Frame #50

9 OLB

synthetic results moving objects2
Synthetic results – Moving objects

OLB vs. SHLB

Frame #1

Frame #50

9 OLB 16 SHLB

synthetic results moving objects3
Synthetic results – Moving objects

OLB vs. Fourier & Haar

Frame #1

Frame #50

synthetic results moving objects4
Synthetic results – Moving objects

OLB vs. Fourier & Haar

Frame #1

Frame #50

3 OLB

synthetic results moving objects5
Synthetic results – Moving objects

OLB vs. Fourier & Haar

Frame #1

Frame #50

3 OLB5 FLB

synthetic results moving objects6
Synthetic results – Moving objects

OLB vs. Fourier & Haar

Frame #1

Frame #50

3 OLB5 FLB

16 HaLB

real experiments procedure

1.1. Static scene, single lights

OLB computation

1.2.SVD  OLB

Video adquisition

2. Moving scene, OLB

3.1. Alignment

Postprocessing

3.2.Relighting

Real Experiments - Procedure
conclusions
Conclusions
  • Proposed an “object-dependent” lighting basis appropriate for relighting tasks.
  • Minimizes the number of reference images needed for relighting.
  • Effective method for video relighting.
future work

Frame t

Frame t+1

Frame t+2

Frame t+3

Frame t+4

Frame t+5

More accurate results

Future work
  • Use faster cameras.
  • Use “motion-dependent” relighting bases.

Slow motion

future work1

Frame t

Frame t+1

Frame t+2

Frame t+3

Frame t+4

Frame t+5

Reduce alignment & orientation errors

More accurate results

Future work
  • Use faster cameras.
  • Use “motion-dependent” relighting bases.

Slow motion

Fast motion

acknowledgments
Acknowledgments
  • Ko Nishino for his useful comments.
  • Anne Fleming for her help in the edition of the videos.
  • All the people from the “Laboratory for the Study of Visual Appearance” and “Columbia Automated Vision Environment\'s” at Columbia University.
  • Cyberware for the 3D models.
  • Funding Sponsors:
    • Spanish Ministry of Science & Technology
    • National Science Foundation
ad