Video Synopsis

1. Video Synopsis Yael Pritch Alex Rav-Acha Shmuel Peleg The Hebrew University of Jerusalem

2. Detective Series: “Elementary”

3. Video Surveillance Problem Cologne Train Bombs, 31-7-06 Terrorists, London tube, 7-7-05 • It took weeks to find these events in video archives. • Cost of a lost information or a delay may be very high.

4. Challenges in Video Surveillance • Millions of surveillance cameras are installed, capturing data 24/365 • Number of cameras and their resolution increases rapidly • Not enough people to watch captured data • Human Attention is Lost after ~20 Minutes • Result: Recorded Video is Lost Video • Less than 1% of surveillance video is examined

5. Handling Surveillance Video • Object Detection and Tracking • Background Subtraction • Object Recognition • Individual people • Activity Recognition • Left luggage; Fight • A lot of progress done. More work remains.

6. Handling Surveillance VideoVideo Synopsis • Object Detection and Tracking • Background Subtraction (Assume Done) • Object Recognition (Do not use) • Individual people • Activity Recognition (Do not use) • Left luggage; Fight • A lot of progress done. More work remains. • Let People do the Recognition

8. Video Synopsis Original video Video Synopsis • A fast way to browse & index video archives. • Summarize a full day of video in a few minutes. • Events from different times appear simultaneously. • Human inspection of synopsis!!!

9. Synopsis of Surveillance VideosHuman Inspection of Search Results • Serve queries regarding each camera: • Generate a 3 minutes video showing most activities in the last 24 hours • Generate the shortest video showing all activities in the last 24 hours • Each presented activity points back to original time in the original video • Orthogonal to Video Analytics

10. Non-Chronological Time Dynamic Mosaicing Video Synopsis The Hebrew University of Jerusalem Salvador Dali

11. Dynamic Mosaics Non Chronological Time

12. HandheldStereo Mosaic

13. u strips Original frames t Mosaic Image

14. ub ua u Frame tk Space-Time Slice Visibility region Frame tl  t Mosaic Image

15. Creating Dynamic Panoramic Movies u First Mosaic - Appearance First Slice t play Last Slice Last Mosaic - Disappearance

16. t u Dynamic Panorama: Iguazu Falls

17. From Video In to Video Out Constructing an aligned Space-Time Volume u t dt a α v b Alignment: Parallax, Dynamic Scenes, etc.

18. Aligned ST Volume: View from Top u u k k k+1 k+1 t t Stationary Camera Panning Camera

19. Generate Output Video Sweeping a “Time Front” surface Interpolation Time is not chronological any more

20. Generate Output Video Sweeping a “Time Front” surface Interpolation Time is not chronological any more

21. u t Mapping each TF to a new frame using spatio-temporal interpolation x Evolving Time Front u t

22. Example: Demolition

23. u t

24. Example: Racing

25. v t

26. Dynamic Panorama: Thessaloniki

27. Creating Panorama: 4D min-cut Aligned space-time volume t x

28. Mosaic Stitching Examples

29. Mosaic Stitching Examples

30. Video Synopsis and Indexing Making a Long Video Short • 11 million cameras in 2008 • Expected 30 million in 2013 • Recording 24 hours a day, every day

31. Explosive growth in cameras… 24m 11m 2009 2014 31

32. Handling the Video Overflow • Not enough people to watch captured data • Guards are watching 1% of video • Automatic Video Analytics covers less than 5% • Only when events can be accurately defined & detected • Most video is never watched or examined!!!

33. A Recent Example

34. Related Work (Video Summary) • Key frames C. Kim and J. Hwang. An integrated scheme for object-based video abstraction. In ACM Multimedia, pages 303–311, New York, 2000. • Collection of short video sequences A. M. Smith and T. Kanade. Video skimming and characterization through the combination of image and language understanding. In CAIVD, pages 61–70, 1998. • Adaptive Fast Forward N. Petrovic, N. Jojic, and T. Huang. Adaptive video fast forward. Multimedia Tools and Applications, 26(3):327–344, August 2005. Entire frames are used as the fundamental building blocks • Mosaic images together with some meta-data for video indexing M. Irani, P. Anandan, J. Bergen, R. Kumar, and S. Hsu. Efficient representations of video sequences and their applications. Signal Processing: Image Communication, 8(4):327–351, 1996. • Space Time Video montage H. Kang, Y. Matsushita, X. Tang, and X. Chen. Space-time video montage. In CVPR’06, pages 1331–1338, New-York, June 2006.

35. Object Based Video Summary • We proposed Objects / Events based summary as opposed to Frames based summary • Enables to shorten a very long video into a short time • No fast forward of objects (preserve dynamics) • Causality is not necessarily kept

36. Video Synopsis • Browse Hours in Minutes • Index back to Original Video Video Synopsis: 1 minute Original video: 24 hours

37. Video Synopsis Shift Objects in Time Synopsis VideoS(x,y,t) Input Video I(x,y,t) t

38. How does Video Synopsis work? Original: 9 hours 09:03 Objects Extracted to Database 10:00 Video Synopsis: 30 seconds 14:38 11:08 18:45 21:50 38 38

39. How Does Video Synopsis works Original: 9 hours Video Synopsis: 30 seconds

40. Steps in Video Synopsis • Detect and track objects, store in database. • Select relevant objects from database • Display selected objects in a very short “Video Synopsis” • In “Video Synopsis”, objects from different times can appear simultaneously • Index from selected objects into original video • Cluster similar objects

41. Object “Packing” Input Video • Compute object trajectories • Pack objects in shorter time (minimize overlap) • Overlay objects on top of time-laps background t Synopsis Video x 42

42. Example: Monitoring a Coffee Station t x

43. t x

44. Original Movie Stroboscopic Movie

45. Panoramic Synopsis Original Panoramic synopsis is possible when the camera is rotating. Panoramic Video Synopsis

46. Endless video – Challenges • Endless video – finite storage (“forget” events) • Background changes during long time periods • Stitching object on a background from a different time • Fast response to user queries

47. Online Monitoring Online Monitoring (real time) Compute background (background model) Find Activity Tubes and insert to database Handle a queue of objects Query Service Collect tubes with desired properties (time…) Generate Time Lapse Background Pack tubes into desired length of synopsis Stitching of objects to background 2 Phase approach

48. Online Monitoring Online Monitoring (real time) Compute background (background model) Find Activity Tubes and insert to database Handle a queue of objects Query Service Collect tubes with desired properties (time…) Generate Time Lapse Background Pack tubes into desired length of synopsis Stitching of objects to background 2 Phase approach

49. Extract TubesObject Detection and Tracking • We used a simplification of Background-Cut* • combining background subtraction with min-cut • Connect space time tubes component • Morphological operations * J. Sun, W. Zhang, X. Tang, and H. Shum. Background cut. In ECCV, pages 628–641, 2006

50. Extract Tubes