Robust video fingerprinting system

1. Robust video fingerprinting system Daniel Pereira d.pereira@skillupjapan.co.jp Luis Loyola loyola@skillupjapan.co.jp

2. Robust video fingerprinting system • Summary • Purpose of the system • What is video fingerprinting • Practical problems to solve • Proposed solution • Results analysis 2

3. Robust video fingerprinting system • Purpose of the system • SkillUpJapan distributes digital contents • FujiTV, TV Tokyo, SkyPerfecTV, Warner Brothers Japan, … • Our platform, Uliza, is an extensible digital content management system • Piracy and DRM are of importance to digital contents rights holders 3

4. Robust video fingerprinting system • Video fingerprinting • A way to effectively tie a video, or a segment of it, to a unique hash value • Information needs to be stored and searched efficiently • Avoid to store original contents provided by clients • Contents should not be recreated from said fingerprint 4

5. Robust video fingerprinting system • Key technical aspects about video • Measured characteristics • Luma and chroma (brightness and color components) • Edge detection, gradient orientation • Time variance • A movie is, after all, a sequence of images that change over time at a defined rate • Amount of data per frame 5

6. Robust video fingerprinting system • Efficiency metrics • Uniqueness • Accurately find videos we search; not return videos that are not what we search • Database • Efficiently index the results in a database • Solution must be fast • Find the clip among many other videos in fastest time 6

7. Robust video fingerprinting system • Some practical problems to solve • Current solutions have relatively accurate algorithms, however the process is computationally expensive • Partitioning of frames, complex algorithm • Database storage is not taken seriously • It is unaffordable to store information for every frame, or large arrays of information for each fingerprint • Slow search times when database grows 7

8. Robust video fingerprinting system • Proposed solution • Lowers the needed resources and process time, while improving upon results (Luma and time based indexes) • Addresses algorithmic complexity by using simple methods (Euclidean distance of vectors and Tanimoto correlation) • Stores information in an efficient way, allowing for quick retrievals, with use of Look-up Tables 8

9. Robust video fingerprinting system Luma Luma Threshold • Proposed solution (video properties): • Average the Luma value of each frame • Luma values show prolonged, relatively constant, values that can be indexed to an interval of time Time Time 9

10. Robust video fingerprinting system • Proposed solution (video properties): • Average Luma calculated according to Luma …… Time 10

11. Robust video fingerprinting system Luma 140 120 85 • Proposed solution (database): • Using those indexes to store only segments we can save lots of space • Each segment of several seconds has a value of 2 bytes • Luma values range from 0 to 255 • Look-up table for segments 16.0 85 Luma Time 19.4 110 2.0 120 2.0 10.3 16.0 19.4 21.4 time 21.4 120 10.3 140 11

12. Robust video fingerprinting system Fingerprints on database Fingerprint A Comparisons • Proposed solution (database): Luma 9 85 19 125 Luma 9 85 Luma Time Time Time 2 120 12 110 4 10 32 13 2 120 12 110 5 140 … … … … 15 160 5 140 … … … Fingerprint A Fingerprint 1 Fingerprint 3 Time 12

13. B A C Robust video fingerprinting system • Proposed solution (algorithm): • Tanimoto • Tanimoto makes a correlation between C and the remaining elements outside C • Euclidean vector distance 13

14. Robust video fingerprinting system • Proposed solution (algorithm): • Hierarchical approach • Look-up Table of segments • Compares the time indexes • & 4. Tanimoto Correlation and Vector Distance of Luma • Look-up Tables discard perceptually different movies efficiently • Comparison of time indexes also behaves efficiently • The number of movies that are ultimately analyzed with Tanimoto Correlation and Euclidean Vector Distance is very low 14

15. Robust video fingerprinting system • Evaluation of algorithm: • 220 movies were analyzed with each other • Quality varies from FullHD to SD • Duration ranges from 15 second commercials to full length movies • Frame-rate of movies varies from 15fps to 30fps • Comparison against C.G.O. (Centroids of Gradient Orientation) [1] • Tests were conducted by searching scenes of 10 seconds • Evaluation compares algorithm, database size and robustness of solutions [1] Sunil Lee and Chang D. Yoo, “Robust Video Fingerprinting for Content-Based Video Identification”, IEEE Trans. Circuits and Systems for Video Technology, vol. 18, no. 7, pp983-988, July, 2008 15

16. Robust video fingerprinting system • Obtained results (database size): 16

17. Robust video fingerprinting system • Obtained results (run-time): 17

18. Robust video fingerprinting system • Obtained results (robustness): 18

19. Robust video fingerprinting system • Summary • State of the art solutions need to better address practical issues • The proposed algorithm can improve upon state of the art algorithms on storage and speed of analysis • Evaluation shows that the proposed solution also provides higher robustness 20

20. Robust video fingerprinting system Questions? Daniel Pereira d.pereira@skillupjapan.co.jp Luis Loyola loyola@skillupjapan.co.jp 21

21. Robust video fingerprinting system Daniel Pereira d.pereira@skillupjapan.co.jp Luis Loyola loyola@skillupjapan.co.jp