Csce 641 computer graphics image based modeling
This presentation is the property of its rightful owner.
Sponsored Links
1 / 52

CSCE 641 Computer Graphics: Image-based Modeling PowerPoint PPT Presentation


  • 96 Views
  • Uploaded on
  • Presentation posted in: General

CSCE 641 Computer Graphics: Image-based Modeling. Jinxiang Chai. Image-based modeling. Estimating 3D structure Estimating motion, e.g., camera motion Estimating lighting Estimating surface model. Traditional modeling and rendering. Geometry Reflectance Light source Camera model.

Download Presentation

CSCE 641 Computer Graphics: Image-based Modeling

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Csce 641 computer graphics image based modeling

CSCE 641 Computer Graphics: Image-based Modeling

Jinxiang Chai


Image based modeling

Image-based modeling

  • Estimating 3D structure

  • Estimating motion, e.g., camera motion

  • Estimating lighting

  • Estimating surface model


Traditional modeling and rendering

Traditional modeling and rendering

Geometry Reflectance Light source Camera model

rendering

modeling

User inputTexture map survey data

Images

For photorealism:

- Modeling is hard

- Rendering is slow


Csce 641 computer graphics image based modeling

Can we model and render this?

What do we want to do for this model?


Image based modeling and rendering

Image based modeling and rendering

Image-based modeling

Image-based rendering

Imagesuser input range scans

Model

Images


Spectrum of ibmr

Spectrum of IBMR

Model

Panoroma

Image-based rendering

Image based modeling

Images + Depth

Geometry+ Images

Camera + geometry

Imagesuser input range scans

Images

Geometry+ Materials

Light field

Kinematics

Dynamics

Etc.


Spectrum of ibmr1

Spectrum of IBMR

Model

Panoroma

Image-based rendering

Image based modeling

Images + Depth

Geometry+ Images

Camera + geometry

Imagesuser input range scans

Images

Geometry+ Materials

Light field

Kinematics

Dynamics

Etc.


Spectrum of ibmr2

Spectrum of IBMR

Model

Panoroma

Image-based rendering

Image based modeling

Images + Depth

Geometry+ Images

Camera + geometry

Imagesuser input range scans

Images

Geometry+ Materials

Light field

Kinematics

Dynamics

Etc.


Stereo reconstruction

Stereo reconstruction

  • Given two or more images of the same scene or object, compute a representation of its shape

  • What are some possible applications?

known

camera

viewpoints


3d modeling

3D modeling

  • From one stereo pair to a 3D head model

  • [Frederic Deverney, INRIA]


3d modeling1

3D modeling

The Digital Michelangelo Project, Levoy et al.


Optical mocap

Optical mocap

Vicon mocap system


Z keying mix live and synthetic

Z-keying: mix live and synthetic

  • Takeo Kanade, CMU (Stereo Machine)


Virtualized reality tm

Virtualized RealityTM

  • [Takeo Kanade et al., CMU]

    • collect video from 50+ stream

    • reconstruct 3D model sequences

    • steerable version used forSuperBowl XXV “eye vision”

  • http://www.cs.cmu.edu/afs/cs/project/VirtualizedR/www/VirtualizedR.html


View interpolation

View interpolation

  • inputdepth image novel view

  • [Szeliski & Kang ‘95]


View morphing

View morphing

  • Morph between pair of images using epipolar geometry [Seitz & Dyer, SIGGRAPH’96]


Image warping

Image warping


Video view interpolation

Video view interpolation


Performance interface

Performance Interface

  • Microsoft Natal project


Additional applications

Additional applications?

  • Real-time people tracking (systems from Pt. Gray Research and SRI)

  • “Gaze” correction for video conferencing [Ott,Lewis,Cox InterChi’93]

  • Other ideas?


Stereo matching

Stereo matching

  • Given two or more images of the same scene or object, compute a representation of its shape

  • What are some possible representations for shapes?

    • depth maps

    • volumetric models

    • 3D surface models

    • planar (or offset) layers


Outline

Outline

  • Stereo matching

  • - Traditional stereo

  • - Multi-baseline stereo

  • - Active stereo

  • Volumetric stereo

  • - Visual hull

  • - Voxel coloring

  • - Space carving


Papers

Papers

  • Stereo matching

    • Masatoshi Okutomi and Takeo Kanade. A multiple-baseline stereo. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 15(4), 1993, pp. 353--363.

    • D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.International Journal of Computer Vision, 47(1/2/3):7-42, April-June 2002.

  • Visual-hull reconstruction

    • Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32.

    • Matusik, Buehler, Raskar, McMillan, and Gortler , “Image-Based Visual Hulls”, Proc. SIGGRAPH 2000, pp. 369-374.

  • Photo-hull reconstruction

    • Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Intl. Journal of Computer Vision (IJCV), 1999, 35(2), pp. 151-173.

    • Kutulakos & Seitz, “A Theory of Shape by Space Carving”, International Journal of Computer Vision, 2000, 38(3), pp. 199-218.


Stereo

Stereo

scene point

image plane

optical center


Stereo1

Stereo

  • Basic Principle: Triangulation

    • Gives reconstruction as intersection of two rays

  • Requires

    • calibration

    • point correspondence


Camera calibration

Camera calibration

  • From world coordinate to image coordinate

Perspective projection

View transformation

Viewport projection

u

sx

a

u0

v

0

-sy

v0

1

0

0

1

2D projections

3D points

Camera parameters


Stereo correspondence

epipolar plane

epipolar line

Stereo correspondence

  • Determine Pixel Correspondence

    • Pairs of points that correspond to same scene point

epipolar line

  • Epipolar Constraint

    • Reduces correspondence problem to 1D search along conjugateepipolar lines

    • Java demo: http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html


Stereo image rectification

Stereo image rectification


Stereo image rectification1

Stereo image rectification

  • reproject image planes onto a common

  • plane parallel to the line between optical centers

  • pixel motion is horizontal after this transformation

  • two homographies (3x3 transform), one for each input image reprojection

  • C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.


Rectification

Rectification

Original image pairs

Rectified image pairs


Stereo matching algorithms

Stereo matching algorithms

  • Match Pixels in Conjugate Epipolar Lines

    • Assume brightness constancy

    • This is a tough problem

    • Numerous approaches

      • A good survey and evaluation: http://www.middlebury.edu/stereo/


Your basic stereo algorithm

For each epipolar line

For each pixel in the left image

  • Improvement: match windows

    • This should look familiar..

    • Can use Lukas-Kanade or discrete search (latter more common)

Your basic stereo algorithm

  • compare with every pixel on same epipolar line in right image

  • pick pixel with minimum matching cost


Window size

W = 3

W = 20

Window size

  • Smaller window

    -

  • Larger window

    -

  • Effect of window size


Stereo results

Stereo results

  • Data from University of Tsukuba

  • Similar results on other images without ground truth

Scene

Ground truth


Results with window search

Results with window search

Window-based matching

(best window size)

Ground truth


Better methods exist

Better methods exist...

  • State of the art method

    • Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,

    • International Conference on Computer Vision, September 1999.

Ground truth


Stereo reconstruction pipeline

Stereo reconstruction pipeline

  • Steps

    • Calibrate cameras

    • Rectify images

    • Compute disparity

    • Estimate depth


Stereo reconstruction pipeline1

Stereo reconstruction pipeline

  • Steps

    • Calibrate cameras

    • Rectify images

    • Compute disparity

    • Estimate depth

  • Camera calibration errors

  • Poor image resolution

  • Occlusions

  • Violations of brightness constancy (specular reflections)

  • Large motions

  • Low-contrast image regions

  • What will cause errors?


Outline1

Outline

  • Stereo matching

  • - Traditional stereo

  • - Multi-baseline stereo

  • - Active stereo

  • Volumetric stereo

  • - Visual hull

  • - Voxel coloring

  • - Space carving


Depth from disparity

disparity map

3D rendering

[Szeliski & Kang ‘95]

Depth from disparity

input image (1 of 2)

X

z

x

x’

f

f

baseline

C

C’


Choosing the stereo baseline

Choosing the stereo baseline

  • What’s the optimal baseline?

    • Too small: large depth error

    • Too large: difficult search problem

all of these

points project

to the same

pair of pixels

width of

a pixel

Large Baseline

Small Baseline


The effect of baseline on depth estimation

The effect of baseline on depth estimation


Csce 641 computer graphics image based modeling

1/z

width of

a pixel

width of

a pixel

pixel matching score

1/z


Multi baseline stereo

Multi-baseline stereo

  • Basic Approach

    • Choose a reference view

    • Use your favorite stereo algorithm BUT

      • replace two-view SSD with SSD over all baselines

  • Limitations

    • Must choose a reference view (bad)

    • Visibility!

  • CMU’s 3D Room Video


Outline2

Outline

  • Stereo matching

  • - Traditional stereo

  • - Multi-baseline stereo

  • - Active stereo

  • Volumetric stereo

  • - Visual hull

  • - Voxel coloring

  • - Space carving


Active stereo with structured light

camera 1

camera 1

projector

projector

camera 2

Active stereo with structured light

  • Project “structured” light patterns onto the object

    • simplifies the correspondence problem

Li Zhang’s one-shot stereo


Active stereo with structured light1

Active stereo with structured light


Laser scanning

Laser scanning

  • Optical triangulation

    • Project a single stripe of laser light

    • Scan it across the surface of the object

    • This is a very precise version of structured light scanning

  • Digital Michelangelo Project

  • http://graphics.stanford.edu/projects/mich/


Laser scanned models

Laser scanned models

The Digital Michelangelo Project, Levoy et al.


Laser scanned models1

Laser scanned models

The Digital Michelangelo Project, Levoy et al.


Desktop scanner

Desktop scanner

  • Convenient to use

  • Good quality

  • Relatively low-cost

  • - next engine (about 2k)


  • Login