DM: Distributed Meetings, a meeting capture and broadcasting system

1. DM: Distributed Meetings, a meeting capture and broadcasting system

2. Overview • Concept • Hardware, Devices • Sound, Audio • Virtual Director • Whiteboard • Cells, classification and background • Image filters • Key Framing • Conclusions

3. Overview 1) Whiteboard 6) Server 5) „Kiosk“ 3) Ringcamera 4) Overview cam 2) WB camera

4. GUI

5. GUI

6. Schemata

7. Specification • Simple, cheap hardware • Maximal comfort for the participants • No special pens, normal WB

8. Hardware • Overview camera • Ring camera • Whiteboard camera • Server • Kiosk

9. Hardware Overview camera: • 640x480 at 15 fps • 90° HFOV view • 1394 bus to server

10. Hardware Ring camera: • Array of 5 cheappixel cameras (~50$) • Total of 3000x480pixels • 360° view • 8 microphones • 1394 bus to server

11. Hardware Whiteboard camera: • Still, consumer-level 4MP camera:CanonG2 • One shot every 5 seconds • MJPEG format via USB to server

12. Hardware Meeting Room Server: • Intel dual P4 2.2 Ghz Archived Meeting Server: • Intel dual P4 2.2 Ghz

13. Hardware Kiosk: • Simple switchboard to setup, start and stop the DM system • Keycard reader for participants

14. Sound, Audio • SSL: Sound source localization.Goal: which participant is speaking? • Noise filtering: • Background Noise (fans, server, etc) • Reverbrations • Beam forming: • the microphone array virtually targets • helps dereverbrate audio

15. Virtual Director • Closes up to speaker(s) in the „speaker window“ • Zooms 360° view • Uses SSL and visual multi-person tracker as desicion base • Has to make „good desicions“ on what to show. (instantly show speaker, show multiple speakers, not switch too often etc)

16. Whiteboard Decision: • Live camera with low resolution catches movements but misses content • Still camera with high resolution catches WB content but misses movements (X)

17. Whiteboard Requirements: • No special drawing and erasing tools • No keyframe marking button next to WB • Fixed camera • Cheap fully remote controllable camera Canon G2 with SDK with 4 MPixels

18. Whiteboard Arising problems: • Obscuring foreground objects • Optical distortion of WB • Unperfect white of WB • Recognizing strokes

19. Whiteboard Image Sequence analysis • Rectify • Extract WB bgcolor • Cluster cell images • Classify as: {stroke, foreground object or WB} • Filter cell images • Extract key frame images • Color-balance key frame images

20. Whiteboard: 1) Rectifying • The corners of the WB are calibrated once per hand • Anything else than WB is cropped • The WB is bi-linear warped using bi-cubic interpolation

21. Whiteboard: 1) Rectifying • The corners of the WB are calibrated once per hand • Anything else than WB is cropped • The WB is bi-linear warped using bi-cubic interpolation

22. Whiteboard: 2) Extracting BG color • For every images, find bg color of every cell • Parts may be obscured (holes) • Must be accurate for final white-balancing

23. Whiteboard: 2) Extracting BG color 1. Strategy: • Assumption:WB-cells are brightest • Holes are filled with nearest neighbours • May fail, ex: paper in foreground

24. Whiteboard: 2) Extracting BG color 2. Strategy • Histogram of each cell (over time) • Peaks are very likely WB BG

25. Whiteboard: 2) Extracting BG color 2. Strategy • Histogram of each cell (over time) • Peaks are very likely WB BG • Detect „outliers“ with least-median-squares

26. Whiteboard: 2) Extracting BG color 2. Strategy • Histogram of each cell (over time) • Peaks are very likely WB BG • Detect „outliers“ with least-median-squares

27. Whiteboard: 2) Extracting BG color 2. Strategy • Histogram of each cell (over time) • Peaks are very likely WB BG • Detect „outliers“ with least-median-squares • Use neighbours for outliers again

28. Whiteboard: 3) Clustering

29. Whiteboard: 3) Clustering

30. Whiteboard: 3) Clustering

31. Whiteboard: 3) Clustering • mmm

32. Whiteboard: 4) Classifying 3 classes: • White Board (background)  greyish: RGB values ~ equal • Strokes  mostly grey with slight color in it • Foreground objects (obscured)  anything else

33. Whiteboard: 4) Classifying

34. Whiteboard: 4) Classifying

35. Whiteboard: 4) Classifying

36. Whiteboard: 4) Classifying

37. Whiteboard: 4) Classifying • The cell contents are compared to the previously computed backround color: Whiteboard color Whiteboard standard deviation Current cells‘s mean color Current cells‘s standard deviation

38. Whiteboard: 4) Classifying • The cell contents are compared to the previously computed backround color: whiteboard stroke foreground

39. Whiteboard: 5) Filtering • Reclassify isolated foreground cells as strokes • Reclassify strokecells next to foreground cells as foreground cells

40. Whiteboard: 5) Filtering • Reclassify isolated foreground cells as strokes • Reclassify strokecells next to foreground cells as foreground cells

41. Whiteboard: 6) Extracting key frames • Key-frames should contain the „most important“ WB content • The best moment to make a key-frame is right before a major erasure

42. Whiteboard: 6) Extracting key frames • Key-frames should contain the „most important“ WB content • The best moment to make a key-frame is right before a major erasure

43. Whiteboard: 6) Extracting key frames Image reconstruction: • If cell image is WB or stroke, use it • If foreground object neighbours or obscures cell, search the cluster for the most recent valid cell image • If no cell image in the cluster is valid, replace it with WB color

44. Whiteboard: 7) Color-balancing

45. Whiteboard: 7) Color-balancing

46. Whiteboard: • Every stroke cell receives a time-stamp where it is being drawed • In the browser, every not yet drawed stroke cell is madevisible as „ghost

47. Whiteboard: • Every stroke cell receives a time-stamp where it is being drawed • In the browser, every not yet drawed stroke cell is madevisible as „ghost“

48. Whiteboard: • Every stroke cell receives a time-stamp where it is being drawed • In the browser, every not yet drawed stroke cell is madevisible as „ghost“ • By clicking on anystroke cell, thebrowsers jumps tothe correct time

49. Meeting Room Server dataflow

50. Meeting Room Server dataflow