Video Retrieval

1. Video Retrieval InsightVideo: Toward Hierarchical Video Content Organization for Efficient Browsing, Summarization and Retrieval IEEE TRANSATION ON MUTILMDEIDA, VOL. 7, NO. 4, AUGUST 2005

2. System Flowchart

3. Hierarchical Video Content Organization • Generally, videos can be represented using a hierarchy of five levels (video, scene, group, shot, key-frame) • Since similar scenes may appear repeatedly in a video, redundant scene information should be reduced by clustering beyond the scene level. • video, clustered scene, scene, group, shot, key-frame

5. Hierarchical Video Content Organization • Construct the video content hierarchy in three steps: • Group detection • video shots are first grouped into semantically richer groups • Scene detection • similar neighboring groups are merged into scenes • Scene clustering • eliminate repeated scenes in the video and reduce the redundant information

6. Group Detection

7. Group Detection • A video group is an intermediate entity between the physical shots and semantic scenes. • The shots in one group usually share similar background or have a high correlation in time series. • A given shot is compared with the shots that precede and succeed it (no more than two shots) to determine the correlation between them

8. Group Detection Procedure

9. Group Detection Procedure • Input: Video shots • Output: Video groups • Procedure: • 1) Given any shot , if larger than • a) If larger than , claim a new group starts at shot • b) Otherwise, go to step1 to process other shots

10. Group Detection Procedure • 2) Otherwise: • a) If both and are smaller than , claim a new group starts at shot • b) Otherwise, go to step1 to process other shots • 3) Iteratively execute step 1 and 2 until all shots are parsed successfully

11. Group Classification • Using the shot grouping strategy, two kinds of shots are absorbed into a given group • temporally related :shots related in temporal series, where similar shots are shown back and forth. • spatially related :shots similar in visual perception, where all shots in the group are similar in visual features

13. Group Classification • Given any group , assign it to one of two categories: • temporally related group • spatially related group • Input: • Video group • Shots in • Output: • Cluster shots in

14. Group Classification Procedure(1) • Procedure • 1) Initially, set , cluster has no members. • 2) Select the shot ( ) in with the smallest shot number as the seed for cluster , and subtract from . If there are no more shots contained in , go to step 5. • 3) Calculate the similarity between and other shot in , If is larger than threshold , absorb shot in cluster . Subtract from .

15. Group Classification Procedure(2) • 4) Iteratively execute step 3, until there are no more shots that can be absorbed in current cluster . and go to step 2. • 5) If is larger than 1, we claim is a temporally related group, otherwise, it is a spatially related group.

16. Representative Shot Selection • Given clusters in , In all, representative shots will be selected for each . • Only one shot in • Two shots in • more key-frames and larger time duration • More than two shots

17. Scene Detection

18. Group Merging for Scene Detection • One scene may be grouped into several groups • Groups in the same scene usually have higher correlation with each other when compared with other groups in different scenes.

19. Group Merging for Scene Detection • Input: Video groups (Gi , i = 1,…,M ) • Output: Video scenes ( SEj , j = 1, …, N ) • Procedure: • Scenes containing only two shots are eliminated • less semantic information

20. SelectRepGroup() • only one groups • only two groups • ratio between the sum of key-frame numbers and shot numbers in each group • choose the one with the highest ratio and longer time duration as the representative group • More than two groups

21. Scene Clustering

22. Video Scene Clustering • Many similar scenes would appear several times in the video. • pairwise cluster scheme (PCS): seedless • Input: • Video scenes • All member groups • Output: • Clustered scene structure

23. Video Scene Clustering • Procedure: • 1)The group similarity matrix • 2) Find the largest value in matrix , and merge the corresponding scenes into a new scene, and use SelectRepGroup() to find the representative group (scene centroid) for a newly generated scene. • 3) If obtained the desired number of clusters, go to the end; else, go to step 4. • 4) Based on the group similarity matrix and the updated centroid of the newly generated scene, update the scene similarity matrix ,then go to step 2.

24. Video Similarity Assessment Frame Level Similarity Evaluation Shot-Level Similarity Evaluation Group-Level Similarity Evaluation Scene-Level Similarity Evaluation Video Level Similarity Evaluation

25. Frame Level Similarity Evaluation

26. Shot-Level Similarity Evaluation • Average Color Histogram Matching: • Shot-Length Matching: • Camera Motion Matching: • Key-Frame Matching: • Shot Level Similarity:

27. Shot-Level Similarity Evaluation • Average Color Histogram Matching: • Shot-Length Matching:

28. Shot-Level Similarity Evaluation • Camera Motion Matching

29. Shot-Level Similarity Evaluation • Key-Frame Matching

30. Group-Level Similarity Evaluation

31. Scene-Level Similarity Evaluation • The similarity between two scenes is the similarity between their representative groups.

32. Video Level Similarity Evaluation • The similarity between two videos is the most similar scenes similarity between them