Spacetime Fusion for Enhanced Video Quality

1. Using Photographs to Enhance Videos of a Static Scene Pravin Bhat1, C. Lawrence Zitnick2, Noah Snavely1, Aseem Agarwala3, Maneesh Agrawala4, Michael Cohen1,2, Brian Curless1, Sing Bing Kang2 University of Washington1, Microsoft Research Redmond2 University of California3, Adobe Systems4 EGSR 2007

2. An overview of Spacetime Fusion

3. Motivation • Low quality video Input Video

4. Motivation • Low quality video • Reconstructed video Input Video Reconstructed Video

5. Motivation • Low quality video • Reconstructed video • Reconstructed from photos • Good spatial reconstruction • Bad temporal reconstruction Input Video Reconstructed Video

6. Motivation • Spacetime Fusion result Input Video Spacetime Fusion Result

7. Motivation • Spacetime Fusion result • Spatial properties of reconstruction • Temporal properties of input video Input Video Spacetime Fusion Result

8. Spacetime Fusion • Define a 3D gradient field

9. Spacetime Fusion • Define a 3D gradient field • Spatial gradients from reconstruction • Temporal gradients from input video

10. Spacetime Fusion • Define a 3D gradient field • Spatial gradients from reconstruction • Temporal gradients from input video • Key Idea • Temporal gradients defined betweenmotion compensated temporal neighbors

11. Spacetime Fusion • Define a 3D gradient field • Spatial gradients from reconstruction • Temporal gradients from input video • Key Idea • Temporal gradients defined betweenmotion compensated temporal neighbors Video frame: t - 1 Video frame: t

12. Spacetime Fusion • Define a 3D gradient field • Spatial gradients from reconstruction • Temporal gradients from input video • Key Idea • Temporal gradients defined betweenmotion compensated temporal neighbors Gt Video frame: t - 1 Video frame: t • Gt(x, y, t) = V(x, y, t) - V(x, y, t - 1)

13. Spacetime Fusion • Define a 3D gradient field • Spatial gradients from reconstruction • Temporal gradients from input video • Key Idea • Temporal gradients defined betweenmotion compensated temporal neighbors Gt Video frame: t - 1 Video frame: t • Gt(x, y, t) = V(x, y, t) - V(x - u, y - v, t - 1)

14. Spacetime Fusion • Define a 3D gradient field • Spatial gradients from reconstruction • Temporal gradients from input video • Key Idea • Temporal gradients defined betweenmotion compensated temporal neighbors • Increases compatibility betweentemporal gradients and spatial gradients

15. Spacetime Fusion • Define a 3D gradient field • Spatial gradients from reconstruction • Temporal gradients from input video • Key Idea • Temporal gradients defined betweenmotion compensated temporal neighbors • Increases compatibility betweentemporal gradients and spatial gradients • Integrate the 3D gradient field

16. Spacetime Fusion • Integrating the gradient field Solve linear system: Av = b

17. Spacetime Fusion • Integrating the gradient field Solve linear system: Av = b Constraints: vx, y, t – vx-1, y, t = Gx(x, y, t) vx, y, t – vx, y-1, t = Gy(x, y, t) vx, y, t – vx-u, y-v, t = Gt(x, y, t)

18. Applications

19. Enhanced Exposure

20. Edit Propagation Input Video

21. Edit Propagation User Edits

22. Edit Propagation User Edits

23. Edit Propagation User Edits

24. Edit Propagation User Edits

25. Edit Propagation User Edits

26. Edit Propagation User Edits

27. Edit Propagation Edited Video

28. Super-Resolution

29. Conclusion • Spacetime fusion

30. Conclusion • Spacetime fusion • Combines spatial and temporal gradients from two different sources

31. Conclusion • Spacetime fusion • Combines spatial and temporal gradients from two different sources • Requires motion vectors for temporal source • stereo (static scenes) • flow (dynamic scenes)

32. Conclusion • Spacetime fusion • Combines spatial and temporal gradients from two different sources • Requires motion vectors for temporal source • stereo (static scenes) • flow (dynamic scenes) • Major applications • Enforcing temporal coherence • Transferring lighting information