Understanding Digital Video: Visual Perception & Color Theory

1. CS 414 – Multimedia Systems DesignLecture 5 – Digital Video Representation Klara Nahrstedt Spring 2011 CS 414 - Spring 2011

2. Administrative • MP1 is out (January 28) • Deadline of MP1 is February 16 • Wednesday midnight – submit via compass • Help-session for MP1 – Monday, February 2, 7pm , Room: 1111 SC CS 414 - Spring 2011

3. Outline • Human Visual System • Visual Perception • Resolution • Brightness • Temporal Resolution • Television • 3D Video CS 414 - Spring 2011

4. Color and Visual System • Color refers to how we perceive a narrow band of electromagnetic energy • source, object, observer • Visual system transforms light energy into sensory experience of sight

5. Human Visual System • Eyes, optic nerve, parts of the brain • Transforms electromagnetic energy

6. Human Visual System • Image Formation • cornea, sclera, pupil,iris, lens, retina, fovea • Transduction • retina, rods, and cones • Processing • optic nerve, brain

7. Retina and Fovea • Retina has photosensitive receptors at back of eye • Fovea is small, dense region of receptors • only cones (no rods) • gives visual acuity • Outside fovea • fewer receptors overall • larger proportion of rods Retina Fovea

8. Transduction (Retina) • Transform light to neural impulses • Receptors signal bipolar cells • Bipolar cells signal ganglion cells • Axons in the ganglion cells form optic nerve Bipolar cells Rods Ganglion Cones Optic nerve

9. Contain photo-pigment Respond to low energy Enhance sensitivity Concentrated in retina, but outside of fovea One type, sensitive to grayscale changes Contain photo-pigment Respond to high energy Enhance perception Concentrated in fovea, exist sparsely in retina Three types, sensitive to different wavelengths Rods vs Cones Cones Rods CS 414 - Spring 2011

10. Tri-stimulus Theory • 3 types of cones (6 to 7 million of them) • Red = L cones, Green = M cones, Blue = S cones • Ratio differentiates for each person • E.g., Red (64%), Green (32%), rest S cones • E.g., L(75.8%), M(20%), rest S cones • E.g., L(50.6%), M(44.2%), rest S cones • Source of information: • See ‘cone cell’ in wikipedia • www.colorbasics.com/tristimulus/index.php • Each type most responsive to a narrow band • red and green absorb most energy, blue the least • Light stimulates each set of cones differently, and the ratios produce sensation of color

11. Color Perception (Color Theory) Hue Scale • Hue • distinguishes named colors, e.g., RGB • dominant wavelength of the light • Saturation • Perceived intensity of a specific color • how far color is from a gray of equal intensity • Brightness (lightness) • perceived intensity Original Saturation lightness CS 414 - Spring 2011 Source: Wikipedia

12. Visual Perception: Resolution and Brightness • Spatial Resolution (depends on: ) • Image size • Viewing distance • Brightness • Perception of brightness is higher than perception of color • Different perception of primary colors • Relative brightness: green:red:blue=59%:30%:11% • B/W vs. Color CS 414 - Spring 2011 Source: wikipedia

13. Visual Perception: Temporal Resolution CS 414 - Spring 2011 Effects caused by inertia of human eye Perception of 16 frames/second as continuous sequence Special Effect: Flicker

14. Temporal Resolution • Flicker • Perceived if frame rate or refresh rate of screen too low (<50Hz) • Especially in large bright areas • Higher refresh rate requires: • Higher scanning frequency • Higher bandwidth CS 414 - Spring 2011

15. Visual Perception Influence Viewing distance Display ratio (width/height – 4/3 for conventional TV) Number of details still visible Intensity (luminance) CS 414 - Spring 2011

16. Television History • 1927, Hoover made a speech in Washington while viewers in NY could see, hear him • AT&T Bell Labs had the first “television” • 18 fps, 2 x 3 inch screen, 2500 pixels

17. Television Concepts • Production (capture) • 2D array of light energy to electrical signals • signals must adhere to known, structured formats • Representation and Transmission • popular formats include NTSC, PAL, SECAM, HDTV • Re-construction • CRT technology and raster scanning • display issues (refresh rates, temporal resolution) • relies on principles of human visual system CS 414 - Spring 2011

18. Video Representations • Composite • NTSC - 6MHz (4.2MHz video), 29.97 fps • PAL - 6-8MHz (4.2-6MHz video), 25 fps • Component • Maintain separate signals for color • Color spaces • RGB, YUV, YCRCB, YIQ CS 414 - Spring 2011

19. PAL video standard Y is luminance UV are chrominance YUV from RGB Y = .299R + .587G + .114BU = 0.492 (B - Y)V = 0.877 (R - Y) Color Coding: YUV Y U-V plane at Y=0.5 U V CS 414 - Spring 2011 Source: wikipedia

20. YCrCb • Subset of YUV that scales and shifts the chrominance values into range 0..1 Y = 0.299R + 0.587G + .114BCr = ((B-Y)/2) + 0.5Cb = ((R-Y)/1.6) + 0.5 CS 414 - Spring 2011

21. YIQ • NTSC standard • YIQ from RGB Y = .299R + .587G + .114B I = .74 (R - Y) - .27 (B - Y)Q = 0.48 (R - Y) + 0.41 (B - Y) YIQ with Y=0.5 CS 414 - Spring 2011 Source: wikipedia

22. NTSC Video • 525 scan lines per frame; 29.97 fps • 33.37 msec/frame (1 second / 29.97 frames) • scan line lasts 63.6 usec (33.37 msec / 525) • aspect ratio of 4/3, gives 700 horizontal pixels • 20 lines reserved for control information at the beginning of each field • so only 485 lines of visible data CS 414 - Spring 2011

23. NTSC Video • Interlaced scan lines divide each frame into 2 fields, each of which is 262.5 lines • phosphors in early TVs did not maintain luminance long enough (caused flicker) • scanning also interlaced; can cause visual artifacts for high motion scenes CS 414 - Spring 2011

24. HDTV • Digital Television Broadcast (DTB) System • Twice as many horizontal and vertical columns and lines as traditional TV • Resolutions: • 1920x1080 (1080p) – Standard HDTV • Frame rate: options 50 or 60 frames per second CS 414 - Spring 2011

25. Pixel Aspect Ratio • Computer Graphics parameter • Mathematical ratio describing horizontal length of a pixel to its vertical height • Used mainly in digital video editing software to properly scale and render video • Into a new format CS 414 - Spring 2011 Source: wikipedia

26. CS 414 - Spring 2011

27. HDTV • Interlaced and/or progressive formats • Conventional TVs – use interlaced formats • Computer displays (LCDs) – use progressive scanning • MPEG-2 compressed streams • In Europe (Germany) – MPEG-4 compressed streams CS 414 - Spring 2011

28. Aspect Ratio and Refresh Rate • Aspect ratio • Conventional TV is 4:3 (1.33) • HDTV is 16:9 (2.11) • Cinema uses 1.85:1 or 2.35:1 • Frame Rate • NTSC is 60Hz interlaced (actually 59.94Hz) • PAL/SECAM is 50Hz interlaced • Cinema is 24Hz non-interlaced CS 414 - Spring 2011 Source: wikipedia

29. SMPTE Time Codes • Society of Motion Picture and Television Engineers defines time codes for video • HH:MM:SS:FF • 01:12:59:16 represents number of pictures corresponding to 1hour, 12 minutes, 59 seconds, 16 frames • If we consider 30 fps, then 59 seconds represent 59*30 frames, 12 minutes represent 12*60*30 frames and 1 hour represents 1*60*60*30 frames. • For NTSC, SMPTE uses a 30 drop frame code • increment as if using 30 fps, when really NTSC has only 29.97fps • defines rules to remove the difference error CS 414 - Spring 2011

30. Take Home Exercise • Given a SMPTE time stamp, convert it back to the original frame number • e.g., 00:01:00:10 CS 414 - Spring 2011

31. Summary • Digitization of Video Signals • Composite Coding • Component Coding • Digital Television (DTV) • DVB (Digital Video Broadcast) • Satellite connections, CATV networks – best suited for DTV • DVB-S – for satellites (also DVB-S2) • DVB-C – for CATV CS 414 - Spring 2011