1 / 38

New dimensions of media

New dimensions of media. Michael M. Bronstein. Department of Computer Science Technion – Israel Institute of Technology cs.technion.ac.il/~mbron. MMSN, San Jose 1 November 2007. Dimensions of media. Radio. Black-and-white television. Color television. 3D video. Conventional 2D TV.

rigg
Download Presentation

New dimensions of media

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. New dimensions of media Michael M. Bronstein Department of Computer Science Technion – Israel Institute of Technology cs.technion.ac.il/~mbron MMSN, San Jose 1 November 2007

  2. Dimensions of media Radio Black-and-white television Color television 3D video

  3. Conventional 2D TV You see exactly what the camera shot Video: Bullettime

  4. Free viewpoint TV (FTV) Interactive selection of viewpoints Video: Bullettime

  5. Augmented reality Place 3D generated objects in the scene Video: Japanese TV

  6. Emerging computer vision applications Analysis of 3D non-rigid objects 3DV systems Bronstein et al, 2003 Gesture recognition (user interface) Face recognition (biometrics)

  7. 3D/Stereoscopic TV Three-dimensional depth perception of the scene

  8. Evolution 1851 Brewster streoscope sold in London 1891 Anaglyph is invented 1952 First color 3D movie premiere Today 3D content available User-grade 3D displays 3DTV broadcast 1922 First 3D movie premiere Early 2000s Commercial 3D monitors appear ? 1800 1820 1840 1860 1880 1900 1920 1940 1960 1980 2000 1838 Wheatson explains binocular vision 1839 Talbot invents photographic process 1928 First television broadcast in USA 1995 First IMAX 3D fiction movie

  9. Ecosystem Computer graphics 3D video acquisition 3D scene representation Coding & Transmission Rendering & Display

  10. Our 3D perception • Binocular vision • Each eye sees a slightly different picture • Depth perception (stereopsis) achieved by merging two pictures

  11. Shape from stereo Closer objects have larger parallax • Depth recovered from parallax (disparity) between corresponding points • Correspondence problem • Can be generalized to multiple views Left eye Right eye Parallax

  12. Stereo cameras Multiview camera array (Carnegy Mellon) IMAX 3D camera NASA Mars rover

  13. Structured light Moving projector Camera • Active sensor • 3D shape extracted from deformation of the projected light pattern • Typically slow and unsuitable for fast moving objects

  14. 3D on your desk Can be literally done using just a pencil and a lamp! Jean-Yves Bouguet, 1998

  15. Coded light 0 0 0 0 1 1 1 Camera Projector Light pattern Angle • Sequence of black/white patterns projected onto the object • Patterns form a binary code encoding the angle relative to the projector • Depth recovered by triangulation • Active sensor

  16. Time-of-flight Nanosecond gate Object Object Sensor Transmitted pulse of light Reflected pulse of light • Principle of a laser range finder • Distance to surface measured by timing the travel of a pulse of light • Requires nanosecond gating • Active sensor

  17. 3D scene representation Stereo pair Depth map Layered depth map (2.5D) Mesh IMAGE-BASED MODEL-BASED • No 3D reconstruction • Allows view synthesis • Natural for stereo cameras • Explicit 3D model • Requires rendering • Used for synthetic CG content

  18. Stereo pair Left Right • Left and right views are stored side-by-side • Can use existing video compression and distribution standards • Large redundancy: special compression algorithms are desirable

  19. Stereo compression Motion-compensated prediction (MCP) CONVENTIONAL VIDEO I P P P I P P P I Motion-compensated prediction I P P P I P P P I Disparity-compensated prediction (DCP) STEREOSCOPIC VIDEO I P P P I P P P I Motion-compensated prediction

  20. 2D+Z Conventional video (2D) Depth (Z) • 2D video and corresponding depth map are stored side-by-size • Uses existing video compression and distribution standards • Z-component increases bandwidth only by 5-20% • Used by Philips (WOW vx)

  21. Geometry compression Trajectory compression Motion trajectories Common representation & coding Motion analysis 3D video Mesh Mesh compression • MPEG spirit • Compress mesh and motion separately • Active research field

  22. Correspondence again Bronstein et al, 2007 • Mesh motion = displacement of corresponding points • Non-rigid shapes are the biggest challenge • Intrinsic geometric correspondence • Methods borrowed from data mining (generalized MDS)

  23. Mesh compression Hoppe, 1996 Progressive mesh compression • Both similar and dissimilar to image compression • Laplace-Beltrami (spectral methods) • Successive approximation (progressive meshes) • Connectivity compression

  24. Viewpoint selection • Capture the scene with multiple cameras • closely located • Transition between different views • If the cameras are dense enough, the • transition will be smooth • Used in QuickTime VR • Only existing viewpoints Object

  25. Panorama • Produce a wide-angle or 360 degrees panorama • Change the viewpoint by selecting a subset of the panorama • Used in QuickTime VR • Limited range of viewpoints

  26. Image-based rendering • Rendering based on existing views • instead of geometric model • Light field: 2D collection of images • (4D array) + time • New viewpoint = 2D slice of the 4D • array, produced by interpolation • Requires multiple views (tens) • Problems with shadows, • specularities, etc. Existing views New view Visualization of light field

  27. 3D display How to create three-dimensional depth perception of the scene? Display to each eye the image it would see Head mounted display (VR glasses) IMAGE-BASED MODEL-BASED • Stereo camera = eyes • Multiview: synthesize the views • the eyes would see • Render the views the eyes • would see • Computer graphics methods

  28. 3D on a big screen Block unwanted rays Screen

  29. Anaglyph Screen + = Left Right Anaglyph • Compatible with any color display • Requires color glasses • Unnatural colors Color glasses

  30. Polarized light • Used in IMAX 3D • Requires special projection • Requires polarized glasses Screen Polarized projector Polarized glasses Polarized projector

  31. Shutter glasses • Left and right image displayed interleaved in time (double framerate) • Shutter glasses keep only one of the eyes open at a time • Requires screen and glasses sync

  32. Do we need glasses?

  33. Lenticular display • Lenticular lens sending parts of the image • to different eyes • No glasses needed (autostereoscopic) • Can be attached to a legacy monitor • User-grade commercial products • available from Sharp, Philips, etc. • Viewing angle is a challenge Screen Lenticular lens

  34. Volumetric display • Display a periodically time-varying 2D image on a rotating mirror • Illusion of 3D object due to visual persistence • 360 degrees view Rotating screen Projector Perspecta display

  35. Summary: acquisition PASSIVE ACTIVE Coded light Structured light Stereo Time-of-flight

  36. Summary: representation & compression PIXEL-TYPE DATA GEOMETRIC DATA Conventional (MCP-based) Mesh compression 2D+Z Stereoscopic (DCP-based)

  37. Summary: display HEAD-MOUNTED STEREOSCOPIC GLASSES VR glasses Shutter glasses Color glasses Polarized glasses VOLUMETRIC AUTOSTEREOSCOPIC Lenticular

  38. Conclusion • 3D applications are an active research and development field • Another hype? • Looks like 3D is going to become a commodity • Still many challenges • Wait 5-10 years and see

More Related