1 / 55

Unwrap Mosaics: A new representation for video editing

Unwrap Mosaics: A new representation for video editing. Alex Rav-Acha et al. In SIGGRAPH 2008 발 표 이성호 2009 년 1 월 22 일. Abstract. A new representation for video Modeling the image-formation process From an object’s texture map to the image Modulated by an object-space occlusion mask

kreeli
Download Presentation

Unwrap Mosaics: A new representation for video editing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Unwrap Mosaics: A new representation for video editing Alex Rav-Achaet al. In SIGGRAPH 2008 발표 이성호 2009년 1월 22일

  2. Abstract • A new representation for video • Modeling the image-formation process • From an object’s texture map to the image • Modulated by an object-space occlusion mask • Recover “unwrap mosaic” • Editing • Re-composition • Into the original space • Resizing objects • Repainting textures • Copying/cutting/pasting objects • Attaching effects layers to deforming objects

  3. Introduction • Unwrap mosaics • A wide range of Editing • Deforming surface • Self-occlusion • Provides references for feature-point tracking

  4. Reconstructiong 3D model • Not easy • Software packages • [2d3 Ltd. 2008; Thorm¨ahlenand Broszio2008] • Extensions to nonrigidscenes • [Bregler et al. 2000; Brand 2001; Torresani et al. 2008] • Less reliable under occlusion

  5. Dense reconstruction from video • Restricted to rigid scenes • [Seitz et al. 2006] • Using interactive tools • [Debevec et al. 1996; van den Hengelet al. 2007] • Also for rigid scenes • Triangulation of the sparse points from nonrigidstructure • Surprisingly troublesome

  6. Unwrap mosaics • Ro recover the object’s texture map • Rather than its 3D shape • Directly from video • recovered texture map will be a • 2D-to-2D mapping • Sequence of binary masks modeling occlusion • Unwrap mosaic

  7. A video will typically be represented • by an assembly of several unwrap mosaics • one per object, • and one for the background. • Edits • performed on mosaic itself • Without converting to 3D • Main contribution • Recover the unwrap mosaic from images • Energy minimization procedure

  8. Figure 2: Reconstruction overview. Steps in constructing an unwrap mosaic representation of a video sequence. Steps 1 to 3a form an initial estimate of the model parameters, and step 3 is an energy minimization procedure which refines the model to subpixel accuracy.

  9. Segmentation • Segment the sequence into • independently moving objects • “video cut and paste” [Li et al. 2005] • Allow for user interaction

  10. Tracking • Recover the texturemap of • a deforming 3D object • from a sequence of 2D images • texture map may be assumed to be constant • although the model is changing its shape • Interestpoint detection and tracking • [Sand and Teller 2006, for example]

  11. Embedding • view the sparse point tracks • as a high dimensional projection of the 2D surface parameters

  12. Mosaic stitching • a map from the tracked points in each image • to the (u; v) parameter space. • A variation of [Agarwala et al. 2004] • emerges naturally from the energy formulation.

  13. Track refinement • the mosaic is good enough • to create a reference template to match • against the original frames • reduces any drift that may have been present after the • original tracking phase

  14. Using the model for video editing • Edit the texture map • For example by drawing on it • Warp it via the recovered mapping • Combine with the other layers of the original sequence • Re-rendered mosaic will not exactly match the original sequence • warped by the 2D–2D mapping • masked by the occlusion masks • alpha-blended with the original image • Remove layers • The removed area is filled in • because the mapping is defined even in occluded areas.

  15. Limitations • Textured surfaces are required for point tracking • Low texture • One dimensional textures • motion blur • The assumption of a smoothly varying smooth 3D surface • objects with significant protrusions • the dinosaur in figure 11 • The assumption of smoothly varying lighting • strong shadows will disrupt tracking • limited to disc-topology objects • rotating cylinder will be reconstructed as a long tapestry • see figure 11

  16. The unwrap mosaic model • Image generation model • how an image sequence is constructed • from a collection of unwrap mosaics • a fitting problem • Energy minimization • via nonlinear optimization • Initial estimate for the • 2D-2D mapping • Texture map • Can be obtained from sparse 2D tracking data.

  17. Point-spread functions

  18. Discrete energy formulation

  19. Data cost

  20. Constraints

  21. Mapping smoothness

  22. Visibility smoothness

  23. Minimizing the energy

  24. Minimizing over C: stitching

  25. Reparametrization and embedding

  26. Minimizing over w: dense mapping • Simply use MATLAB’s griddata to interpolate • No guarantee of minimizing the original energy

  27. Minimizing over w and b: dense mapping with occlusion

  28. Lighting

  29. User interaction and hinting • Segmentatinon • the user selects objects in a small number of frames, • and the segmentation is propagated using optical flow or sparse tracks. • mosaic coverage

  30. Tuning parameters • The robust kernel width τ • is set to match an estimate of image noise. • Set to 5/255 gray-levels • Scale parameter τ3 • 40 pixels • Except the face, which had many outlier tracks • spatial smoothness λwl • controls the amount of deformation of the mapping • Constant for all • but the “boy” sequence

  31. Results • Synthetic sequence

  32. Synthetic sequence • There is no concept of a “ground truth” • visually evaluate the recovered mosaic (figure 7) • About 30% of mosaic pixels visible • in any one frame • Self occlusion near the nose

  33. Face sequence • Few high-contrast points coupled • with strong lighting

  34. Giraffe sequence: foreground • A logo • is placed on the foreground giraffe’s back and head • With optical flow, • the annotation drifts by about 10 pixels in 30 frames, while the • Unwrap mosaic • shows no visible drift.

  35. Boy sequence • both sides of his head and torso are shown • variable focus • motion blur • considerable foreground occlusion • Ear is doubled • could be fixed • by editing as in section 4

More Related