multiview video n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Multiview Video PowerPoint Presentation
Download Presentation
Multiview Video

Loading in 2 Seconds...

play fullscreen
1 / 35

Multiview Video - PowerPoint PPT Presentation


  • 214 Views
  • Uploaded on

Multiview Video. Kai-Chao Yang. Outline. History of Video Coding Standards Definition of Multiview Video Applications of Multiview Video Concept of Stereo Video Multiview TV/Video System Multiview Content Capture Correction Coding Display. History of Video Coding Standards. H.261.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Multiview Video' - vesna


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
multiview video

Multiview Video

Kai-Chao Yang

VCLAB, National Tsing Hua University, Taiwan

outline
Outline
  • History of Video Coding Standards
  • Definition of Multiview Video
  • Applications of Multiview Video
  • Concept of Stereo Video
  • Multiview TV/Video System
    • Multiview Content
    • Capture
    • Correction
    • Coding
    • Display

VCLAB, National Tsing Hua University, Taiwan

history of video coding standards
History of Video Coding Standards

H.261

H.263

H.263+

H.263++

ITU-T

VCEG

MPEG-1

MPEG-4 v2/visual

ISO

MPEG

H.262/MPEG-2

H.264/MPEG4 AVC

FRExt

SVC

MVC

JVT

1984

1986

1988

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

VCLAB, National Tsing Hua University, Taiwan

definition of multiview video
Definition of Multiview Video
  • Multiview video
    • Multiple cameras are used to simultaneously acquire various viewpoints of a scene.
  • Multiview video coding
    • Encoding of sequences captured simultaneously from multiple cameras using a single video stream.

VCLAB, National Tsing Hua University, Taiwan

applications of multiview videos
Applications of Multiview Videos
  • Free viewpoint TV
    • The viewers can experience the free viewpoint navigation within the range covered by the cameras
    • http://www.youtube.com/watch?v=0yP_J6M4fiU
  • Three-dimensional TV
  • Immersive teleconference

VCLAB, National Tsing Hua University, Taiwan

examples 1 4
Examples (1/4)
  • Entertainments
    • 3D game, FVV, …
  • Medicine
    • 3D microscope, 3D endoscope, …
  • Education
    • 3D model, 3D classroom, …
  • Surveillance
    • Nurse, object recognition and tracking, …
  • Simulation
    • Flight simulation, …

VCLAB, National Tsing Hua University, Taiwan

examples 2 4
Examples (2/4)
  • 3D microscope and 3D endoscope

http://www.njneurosurgeons.com/news/2010/04/19/new-jerseys-first-3d-endoscopic-surgery-performed-by-ninj-faculty.html

VCLAB, National Tsing Hua University, Taiwan

examples 3 4
Examples (3/4)
  • Pedestrian Tracking

Kyungnam Kim and Larry S. Davis, "Multi-Camera Tracking and Segmentation of Occluded People on Ground Plane using Search-Guided Particle Filtering", European Conference on Computer Vision (ECCV), LNCS, 2006.

VCLAB, National Tsing Hua University, Taiwan

examples 4 4
Examples (4/4)
  • Flight Simulation

VCLAB, National Tsing Hua University, Taiwan

concept of stereo image video
Concept of Stereo Image/Video

3D information: Determination of relative depths in a perceived scene

Stereopsis

Accommodation of the eyeball

Occlusion

Linear perspective

Vertical position

2D knowledge

VCLAB, National Tsing Hua University, Taiwan

concept of stereo video 3d video
Concept of Stereo Video (3D Video)
  • Stereopsis
    • visual perception leading to the sensation of depth from the two slightly different projections of the world onto the retinas of the two eyes

VCLAB, National Tsing Hua University, Taiwan

concept of stereo video
Concept of Stereo Video

VCLAB, National Tsing Hua University, Taiwan

multiview tv system
Multiview TV System

Content

Encoding

Delivery

Decoding

Display

VCLAB, National Tsing Hua University, Taiwan

multiview content
Multiview Content

VCLAB, National Tsing Hua University, Taiwan

multiview content1
Multiview Content
  • Camera capture
    • Stereo cameras
    • Camera array
  • Computer generated
    • Objects is defined
    • Depth is inherent
  • Conversion from 2D video
    • Detecting objects
    • Assigning depth
    • Filling occluded parts

VCLAB, National Tsing Hua University, Taiwan

camera capture
Camera Capture
  • Stanford multi-camera array (128 video cameras)
  • Google street view camera (Image only)
  • Panasonic Full-HD 3D camera

VCLAB, National Tsing Hua University, Taiwan

correction
Correction
  • Rectification of misalignment
  • Normalization of colors

VCLAB, National Tsing Hua University, Taiwan

multiview video coding
Multiview Video Coding

VCLAB, National Tsing Hua University, Taiwan

representation formats full resolution
Representation Formats (Full-Resolution)
  • Full-Resolution Stereo and Multiview
    • Double data rate for stereo videos
    • N-fold data rate for N-view videos
  • Efficient compression is the key issue
    • MVC extension of H.264/AVC

VCLAB, National Tsing Hua University, Taiwan

representation formats stereo interleaving

L

R

L

R

Representation Formats (Stereo Interleaving)
  • Stereo Interleaving
    • A multiplex of the two views into a single sequence
  • Spatial multiplexing
    • The left and right views are sub-sampled and interleaved into a single frame
  • Temporal multiplexing
    • The left and right views are interleaved as alternating frames
  • Advantages
    • Compatible with existing codecs and delivery infrastructure
  • Drawbacks
    • Loss of spatial or temporal resolution

VCLAB, National Tsing Hua University, Taiwan

representation formats depth based
Representation Formats (Depth-based)
  • Depth-based Format
    • 2D video + depth map
  • Advantages
    • Backward compatibility with the older coding standards
    • Supporting both stereo and multiview displays
    • Depth is adjustable
  • Drawbacks
    • Limited depth range
    • Occlusions
    • Other views have to be synthesized

Synthesized view

2D view

VCLAB, National Tsing Hua University, Taiwan

multiview video coding1
Multiview Video Coding
  • Single view
  • Multi-view
    • Independent encoding
      • Low coding complexity but also low coding efficiency
    • Inter-view prediction
      • Exploiting both spatial and temporal redundancy

I

B

B

B

P

B

B

B

P

I

B

B

B

P

B

B

B

P

I

B

B

B

P

B

B

B

P

VCLAB, National Tsing Hua University, Taiwan

multiview video coding2

I

B

P

P

P

P

P

P

P

P

P

P

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

P

B

P

B

B

B

B

B

B

B

B

B

P

P

P

P

P

P

P

P

P

B

B

B

B

B

B

B

B

B

P

B

P

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

I

B

P

I

B

P

I

P

Multiview Video Coding
  • Multiview camera

VCLAB, National Tsing Hua University, Taiwan

view generation
View Generation

View 1

synthesis

View 2

VCLAB, National Tsing Hua University, Taiwan

view generation1
View Generation

View synthesis

Disparity-based

e.g.

view1view3/2 view2

object(0,0) object(3,0) object(6,0)

Depth-based

2D + depth map

http://www.imec.be/ScientificReport/SR2007/html/1384302.html

VCLAB, National Tsing Hua University, Taiwan

26

DEPTH MAPS EXTRACTION FROM MULTI-VIEW VIDEOS

http://www.youtube.com/watch?v=KtRSbey1sKM

view generation2
View Generation

VCLAB, National Tsing Hua University, Taiwan

display 3d space
Display (3D space)

Volumetric video

Holographic video

VCLAB, National Tsing Hua University, Taiwan

http://www.youtube.com/watch?v=kIDgC2no1uo

display
Display
  • Free viewpoint video
    • http://www.youtube.com/watch?v=vyhz8KgW49E
    • http://www.youtube.com/watch?v=GSumx0Zs2XA

VCLAB, National Tsing Hua University, Taiwan

display with 3d glasses
Display with 3D Glasses
  • Stereo video with glasses
    • Left view and right view
      • Passive and active

VCLAB, National Tsing Hua University, Taiwan

display with 3d glasses1
Display with 3D Glasses

Anaglyph, polarized, and shutter glasses

http://www2.ciw.com.cn/h/2562/357866-17902.html

VCLAB, National Tsing Hua University, Taiwan

display1
Display
  • Lenticular panel and Parallax barrier

VCLAB, National Tsing Hua University, Taiwan

display2
Display
  • Stereo view display without glasses

A2

A2

A2

Pixel 2

A2

A1

A2

A2

A1

A2

A1

A2

A1

A2

A1

A2

A1

Pixel 1

A1

A1

A1

A1

VCLAB, National Tsing Hua University, Taiwan

display3
Display
  • Multiview display without glasses

C2

C2

C2

C1

Pixel 2

C2

C1

B2

C1

B2

B2

C1

B2

B1

A2

B1

A2

B1

A2

B1

Pixel 1

A2

A1

A1

A1

A1

VCLAB, National Tsing Hua University, Taiwan

conclusions
Conclusions
  • An overview of multiview video from capture to display.
  • Issues
    • Display
      • Reduction of brightness
      • Reduction of refresh rate
      • Reduction of temporal of spatial resolution
    • Coding efficiency
    • Decoding delay
    • 2D  3D
    • View synthesis

VCLAB, National Tsing Hua University, Taiwan

reference
Reference

A. Vetro, “Representation and Coding Formats for Stereo and Multiview Video,” Tech. Rep. TR2010-011, April, 2010.

S. Gaël, “Depth Map Estimation and Use for 3DTV,” Tech. Rep. n0379, February, 2010.

Y.-S. Ho and K.-J. Oh, “Overview of Multi-view Video Coding,” IWSSIP, 2007.

許精益 and 黃乙白 “3D立體顯示技術之發展與研究,” 光學工程第98期, 96年6月.

VCLAB, National Tsing Hua University, Taiwan