Light Field Video Stabilization

1. Light Field Video Stabilization Insight: Only Need Relative Transformation Rf, tf This work is sup- ported in part by Adobe System Inc. & NSF IIS-0845916. Relative Transformation , Zf-1 pf-1 pf+1 , Zf+1 , Zf pf Brandon M. Smith, Li Zhang Computer Sciences, UW-Madison Hailin Jin, AseemAgarwala Adobe Systems, Inc. • Stabilization without structure from motion R,t • Use fewer cameras (two instead of five) Original Camera Canny Edge Maps • Can handle challenging cases sensor stabilization • Motion deblurring with camera arrays UW-Madison Graphics & Vision Group Input Images Depth Maps reproj(pf-1, Zf-1,Rf-1,tf-1) reproj(pf+1, Zf+1,Rf+1,tf+1) reproj(pf, Zf,Rf,tf) Panasonic HD Stereo Camcorder Viewplus Profusion 25C • Better handle image periphery problems Virtual Camera Path Virtual Camera Path 4. Run spacetime optimization to find {R,t}f=1,…,F • Evaluate a range of camera baselines Existing applications qf-1 New View Synthesis qf qf+1 Actual Camera Path Actual Camera Path • New-view synthesis [Levoy & Hanrahan SIGGRAPH ‘96, Gortler et al. SIGGRAPH ‘96] Algorithm Outline Motivation Videos shot with a hand-held conventional video camera often appear very shaky. This shakiness is often the most distracting aspect of amateur videos that easily distinguishes them from more professional work. Spacetime optimization: Maximize smoothness of virtual video as function of {Rf, tf}f=1…F Synthesize a video along a virtual smooth camera path • Synthetic aperture[Wilburn et al., SIGGRAPH 2005] Virtual Camera Vibration Reduction (VR) Optical Flow [Bruhn et al. 2005] • Noise Removal [Zhang et al., CVPR 2009] Synthesized Image optical stabilization F-1 qf,j- -qf-1,jprev+ qf+1,jnext 2 | | 1 ( ( ) ) å å F-1 • {R,t}f=1,…,F = wf,j E • +Ereg 2 Advantage: Do not need to compute 3D input camera path Compute depth map for each time instant[Smith et al. 2009] f=2 j New application f Îf 2 | | å å 1 ( ) ) ( qf,j- -qf-1,jprev+ qf+1,jnext Video Stabilization = E • {R,t}f=1,…,F wf,j • +Ereg • Video Stabilization 2 f=2 j Îf Professional Solutions Consumer Solutions 2. Compute optical flow for each time instant[Bruhn et al. 2005] 3. Detect Canny edges, use flow to match edges over time | | • 2D-transformation based methods • Burt & Anandan, Image Stabilization by Registration to a Reference Mosaic. DARPA Image Understanding Workshop, 1994 • Hansen et al., Real-Time Scene Stabilization and Mosaic Construction. DARPA Image Understanding Workshop, 1994 • Lee et al., Video Stabilization Using Robust Feature Trajectories. ICCV 2009 5. New view synthesis • Cumbersome State-of-the-art New Approach • Rotational camera motion • Small baseline … camera crane steadicam camera dolly New View Synthesis Use special hardware to avoid camera shake • Expensive • Distant scenes • Limited DOF Why Does a Camera Array Help? How to Avoid Structure from Motion? Liu et al., SIGGRAPH 2009 Results Works well if structure from motion is successful • Background has enough visual features • Small dynamic targets Stabilization as image based rendering[Buehler et al. CVPR 2001] How to Define the Smoothness of a Video? Matching Salient Features • More input views at each time instant • Easier to work with dynamic scenes • Better handling of parallax Summary of Contributions Future Work | | • Use an array for stabilization • Nearby, dynamic targets • Increase algorithm efficiency • Large scene depth variation • Violent camera shake