Author ： Tzu- Chieh Yen, Chia -Ming Tsai, Chia-Wen Lin, Senior Member, IEEE

1. Author：Tzu-Chieh Yen, Chia-Ming Tsai, Chia-Wen Lin, Senior Member, IEEE Source：IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 8, Source：AUGUST 2011 Speaker：Jhen-Yu Yang Advisor：Guo-Shiang Lin Maintaining Temporal Coherence in Video Retargeting Using Mosaic-Guided Scaling

2. Outline • Introduction • Methods • Results • Conclusion

3. Introduction • 基於內容為主的影像畫面濃縮，目的是希望畫面經過濃縮之後能盡量維持畫面中人眼感興趣的區域，並且縮小或裁切掉人眼較不感興趣的區域，使得濃縮後的影像能呈現原始影像所要分享的訊息。 • 對於濃縮後的視訊如何確保畫面之間在空間軸上以及時軸上的一致性將會是影響視訊品質的關鍵。 • 目前現存的方法，對於同時包含有背景鏡頭移動與物體移動的影片很難在畫面濃縮後保持在時軸上的一致性。

4. Introduction • 利用全景接圖導引的方式去決定每一個相對應區域的縮放比例，以確保畫面之間在時軸上的一致性。 • 提出的方法與步驟 • 先將影片中屬於同一個場景的每一張畫面接合產生一張全景圖。 • 並根據全景圖產生一張全域縮放圖。 • 接著每一張畫面參考全域縮放圖上的資訊與提出的空間軸上限制作最佳化處理。 • 最後得到每一張畫面個別的局部縮放圖和濃縮後的畫面。

5. Outline • Introduction • Methods • Results • Conclusion

6. Method • Overall view

7. Method • Mosaic-guided scaling method • Frame-level energy maps • Represent the visual importance of a pixel in each video frame • Initial local scaling maps • Determine how to scale a pixel in frame resizing • Shot-level panoramic mosaic

8. Method • Frame-level energy map

9. Method • Energy fusion function • Combine the gradient energy and the PQSM-based saliency map

10. Method • Initial local scaling maps • Uses an energy-based frame resizing approach

11. Method • Shot-level panoramic mosaic • A panoramic mosaic is generated by using three steps • Feature points detection • Camera motion estimation • Frame registration

12. Method • Mosaic-Guided video retargeting • Global scaling map • Global map constraint • Spatial coherence constraints • Iterative optimization procedure • Frame resizing based on local maps

13. Method • Global scaling map • To synchronize the scaling of a corresponding pixel in different frames of the shot

14. Method • The maximum scaling factor

15. Method • Global map constraint • Thefirst-round local scaling maps areextracted by  • The first-round local scaling maps are set as

16. Method • To ensure the temporal coherence • Use the first-round maps to constrain the iterated local scaling maps • by introducing the following distortion cost

17. Method • Spatial coherence constraints • Regions-of-Interest (ROI) Deformation Constraint • Spatial Smoothness Constraint

18. Method • Regions-of-Interest (ROI) Deformation Constraint • The scaling factors of the pixels of visually important foregrounds/backgrounds should be made consistent

19. Method • To maintain the consistency of foreground object size • the following spatial scaling inconsistency should be minimized

20. Method • Spatial Smoothness Constraint • Limit the sum of the differences between the scaling factors of every two vertically adjacent pixels/patches on a line not to exceed a threshold as follows

21. Method • Iterative optimization procedure

22. Method • Frame resizing based on local scaling maps • Pixel fusion based image downscaling

23. Method • Pixel value of the resized frame is computed by

24. Outline • Introduction • Methods • Results • Conclusion

25. Results • Compared with other methods

26. Outline • Introduction • Methods • Results • Discussion

27. Conclusion • 實驗的結果顯示我們提出的視訊濃縮方法可以有效維持濃縮後的視訊畫面在時軸上的一致性，甚至對於包含有背景鏡頭移動與物體移動的影片也能有不錯的效果。

28. The End Thanks