Static Sprite Generation

1. Static Sprite Generation Prof︰ David, Lin Student︰ Jang-Ta, Jiang 2006.02.16

2. Outline • Introduction • Sprite-Generation Architecture• GME、Segmentation、Extraction、Blending • Application in Frame-Based Video Coding • Experimental Results • Conclusions

3. Reference • [1] Y. Lu, W. Gao, and F. Wu, "Sprite generation for frame-based video coding," in Proc. IEEE International Conference on Image Processing (ICIP), 1, pp. 473--476, 2001. • [2] Y. Lu, W. Gao, and F. Wu, " High efficient sprite coding with directional spatial prediction , " ICIP (1) 2002: 201-204 • [3] Y. Lu, W. Gao, and F. Wu,"Efficient background video coding with static sprite generation and arbitrary-shape spatial prediction techniques, " IEEE Trans. Circuits Syst. Video Techn. 13(5): 394-405 (2003)

4. Introduction (1/4) • A sprite is an image composed of pixels belonging to a video object visible throughout a video segment • For instance, sprite generated from a panning sequence will contain all the visible pixels of the background object throughout the sequence • Portions of this background may not be visible in certain frames due to the occlusion of the foreground objects or the camera motion. • Thus, the sprite contains all parts of the background that were at least visible once

5. Introduction (2/4) • The sprite encoding syntax can be utilized for the transmission of any still image to the decoder since a sprite is essentially just a still image • Static sprites are those that are directly copied (including appropriate warping and cropping) to generate a particular rendition of the sprite at a particular time instant • Improves the coding efficiency for video sequences with lots of revisiting backgrounds

6. Introduction (3/4) • The main idea of static sprite coding technique is to generate the reconstructed VOPs by directly warping the quantized sprite using specified motion parameters • Residual error between the original VOP and the warped sprite is not added to the warped sprite

7. Introduction (4/4)

8. Motivation • The motion of foreground video objects not only disturbs the accuracy of motion estimation but also blurs the generated sprite image • We presents a novel technique for the generation of background sprite image with improved Global Motion Estimation (GME) and automatic segmentation • The proposed technique is used to construct high quality sprite directly from raw video sequence

9. Outline • Introduction • Sprite-Generation Architecture• GME、Segmentation、Extraction、Blending • Application in Frame-Based Video Coding • Experimental Results • Conclusions

10. Sprite-Generation Architecture • The first frame is blended into the blank sprite to obtain the initial sprite.

11. Sprite-Generation Architecture (cont.) • First, the global motion between the current frame and the sprite is estimated. • Secondly, the reliability map is extracted based on the segmentation information. • Third, both the current frame and the extracted reliability map are warped toward the sprite image. • Finally, the warped frame is blended with the sprite using three different strategies in terms of the warped reliability map.

12. Global Motion Estimation (GME) • GME aims at finding the global motion parameters of the current frame relative to the sprite image. • A traditional hierarchical GME based on gradient decent algorithm is employed • However, the difference is that a hybrid scheme jointing the short-term and long-term motion estimation is developed • Traditionally, GME either performs the short-term motion estimation and then concatenates the parameters or performs the long-term motion estimation directly

13. Global Motion Estimation (GME) • Instead of estimating the global motion of the current image directly from the previous sprite, the described algorithm first warps the previous sprite and then calculates the global motion referencing the warped sprite image at the top level of the hierarchical GME • The proposed hybrid GME scheme can tackle the long-term motion influence with the assistance of the previously estimated motion, and meanwhile avoid error accumulation by directly using the original sprite as reference image in the final step of GME

14. Intermediate global motion Initial global motion

15. Rough Image Segmentation 1. Motion occlusion zones detection • detected from the difference image • the zones can be detected using a threshold • morphological filtering is developed

16. Rough Image Segmentation 2. Segmentation model initialization • Canny operator is employed to detect all the edge pixels in the current frame • the initial segmentation model is extracted by selecting foreground edge pixels in term of the motion occlusion zones

17. Rough Image Segmentation 3. Moving object extraction • utilizing a filling-in technique to the initial segmentation model • the foreground objects are segmented from the mask

18. Pixel Reliability Extraction • Segment the current frame into three regions • Pixels not belonging to the background consist of the undefined region (UN) • Pixels belonging to the background are segmented into reliable region (RR) and unreliable region (UR) • UR is defined by extracting some pixels along the borders of the background object,i.e.

19. Pixel Reliability Extraction (a) (b) (c) (a) extracted reliability map (b) original segmentation map (c) reliability map warped toward the sprite

20. Reliability-Based Image Blending • Reliable pixels in the input frame can be averagely blended with reliable pixels in the sprite, or replace unreliable pixels in the sprite and then mark them as reliable. • Unreliable pixels in the input frame cannot be used to update reliable pixels in the sprite, but it can be averagely blended with the unreliable pixel, or fill the empty region in the sprite and then set it as unreliable. • Undefined pixels in the input frame do not contribute to the sprite updating.

21. Reliability-Based Image Blending • After processing all frames of the video sequence, the reliable and unreliable regions in the sprite are merged together to obtain the final opaque areas of the sprite image Two advantages︰ • prevent the unreliable pixels from constantly contributing to the sprite updating • unreliable region tackles the aperture problem of the sprite generation that might happen in the place where the reliable pixel never appears

22. Some Background Sprite ForemanCoastguard

23. Outline • Introduction • Sprite-Generation Architecture• GME、Segmentation、Extraction、Blending • Application in Frame-Based Video Coding • Experimental Results • Conclusions

24. Application in Frame-Based Video Coding • Sprite coding is usually used in object-based video coding, whose implementation premise is that precise segmentation mask should be available in advance • A frame-based video coding scheme incorporating sprite coding techniques is developed • Rough segmentation is enough for the purpose of sprite generation

25. Application in Frame-Based Video Coding The coding mode can be selected automatically in two step • Firstly, the selection is decided between SPRITE and INTRA/INTER mode • If SPRITE mode is selected, error signals are encoded using MPEG-4 INTER coding method; otherwise the traditional block-based motion compensation is performed and then INTRA or INTER mode is secondly selected

26. Application in Frame-Based Video Coding • The raw MBs in INTRA mode and the error signal MBs in INTER mode are encoded using MPEG-4 coding scheme, respectively • The proposed coding method is more efficient compared with traditional frame-based techniques, because the sprite motion model can effectively describe the motion of camera, and therefore the bits can be greatly saved

27. Outline • Introduction • Sprite-Generation Architecture• GME、Segmentation、Extraction、Blending • Application in Frame-Based Video Coding • Experimental Results • Conclusions

28. Experimental Results (1/5) (1) Original frame (2) auxiliary segmentation (3) rough segmentation

29. Experimental Results (2/5) Stefan sequence, 300 frames in CIF at frame rate 30fps (auxiliary segmentation) (a) reliability-based blending (average PSNR= 23.1dB)(b) average blending(average PSNR= 22.4dB)

30. Experimental Results (3/5) (c) (d) (e) (c) original image (d) image reconstructed from the sprite in (a) (e) image reconstructed from the sprite in (b)

31. Experimental Results (4/5) Stefan sequence, 300 frames in CIF at frame rate 30fps (rough segmentation) (f) reliability-based blending (average PSNR= 22.9dB)(g) average blending(average PSNR= 22.0dB)

32. Experimental Results (5/5) (h) (i) (j) (h) original image (i) image reconstructed from the sprite in (f) (j) image reconstructed from the sprite in (g)

33. Outline • Introduction • Sprite-Generation Architecture• GME、Segmentation、Extraction、Blending • Application in Frame-Based Video Coding • Experimental Results • Conclusions

34. Conclusions • Considering that accurate segmentation is very hard to obtain, we proposes to utilize the reliability-based blending in the sprite generation • Experiments have proven that the reliability-based blending scheme can effectively eliminate the blurs caused by moving foreground objects due to the inaccurate segmentation • The proposed technique can produce the sprite directly from the raw video sequence by using the rough image segmentation and the reliability-based blending scheme

35. Thank You