Multimedia

1. Multimedia Chapter 29 Data communications and networking

2. What is multimedia? • Definition of multimedia • Hard to find a clear-cut definition • In general, multimedia is an integration of text, graphics, still and moving images, animation, sounds, and any other medium where every type of information can be represented, stored, transmitted and processed digitally • Network Multimedia applications: sending and receiving audio and video (“continuous media”) across networks

3. Streaming Streaming means a user can (listen /watch) the file after the downloading has started. We can divide audio and video services into three broad categories: Ex: stored Songs, books on tape, lectures, movies, TV shows, music video clips. Ex: Some radio stations & TV channels broadcast programs on the Internet. Applications: Internet telephony and Internet teleconferencing.

4. DIGITIZING AUDIO AND VIDEO Digitizing Audio An analog signal, such as audio, can be digitized to produce a digital signal. Nyquist theorem: Sampling with the rate of 2f sample per second Voice: 8000 samples /second (4KHz*2)*8 bits /sample  Digital signal of 64 kbps. Music: 44,100 samples / second *16 bits / sample.  Monaural 705.6 kbps Stereo: 1.411 Mbps

5. Digitizing Video Video consists of a sequence of frames. In North America 25 frames per second  may have flickering repaints each frame twice Frames are made of pixels Black and white TV 1 pixel = 8-bits  represents one of 256 different gray levels. Color TV 1 pixel = 24 bits (8 red, 8 green, and 8 blue). Ex: What is the data rate for colored video transmission in North America with the resolution of 1024 x 768 pixels? 2 x25 x I024x768x 24=944 Mbps We need compression to send video over the Internet.

6. Audio Compression Two categories of techniques are used for audio compression: 1. Predictive Encoding • Encodes the differences between the samples instead of all the samples. • Normally used for speech • Some standers: GSM (13 kbps), G.729 (8 kbps), G.723.3 (6.4 or 5.3 kbps). 2. Perceptual Encoding: • MP3≡ MP3 (MPEG audio layer 3), • used to create CD-quality audio • based on the perceptual of psychoacoustics & masking from the auditory system • MP3 data rates: 96 kbps, 128 kbps, and 160 kbps. • (1.411 Mbps mentioned without compression) • MP3 uses two phenomena, frequency and temporal masking, to compress audio signals

7. Video Compression • Image Compression: JPEG lossy compression • Divides the picture into blocks (8*8) • Discrete Cosine Transform (DCT) phase: changes the 64 values so that the relative relationships between pixels are kept but the redundancies are revealed. • Quantization phase: to reduce the number of bits needed for encoding. It is the only phase in the process that is not reversible • Compression: to remove the redundant 0s.

8. Multimedia Applications: STREAMING STORED AUDIONIDEO We have four different downloading approaches • First Approach: Using a Web Server • Simple and does not involve streaming. • Uses HTTP protocol and send a GET message to download the file from the server. • The client’s browser includes application (media player) to play the file. • Drawback: Even with compression the file is usually large and it needs downloading completely before it can be played.

9. First Approach

10. Second Approach: Using a Web Server with Metafile The media player is directly connected to the Web server The Web server stores two files: the actual media file and a metafile which is sent in the response. The client’s media player uses the URL in the metafile to access the audio/video file. Problem? It is using HTTP which is designed to run over TCP. This is appropriate for retrieving the metafile, but not for retrieving the audio/video file.

11. Second Approach: Using a Web Server with Metafile

12. Third Approach: Using a Media Server The media player and the media server use UDP in their communications

13. Fourth Approach: Using a Media Server and RTSP The Real-Time Streaming Protocol (RTSP) is a control protocol designed to add more functionalities to the streaming process. RTSP is an out-of-band control protocol that is similar to the second connection in FTP.

14. Fourth Approach: Using a Media Server and RTSP 1. The HTTP client accesses the Web server by using a GET message. 2. The information about the metafile comes in the response. 3. The metafile is passed to the media player. 4. The media player sends a SETUP message to create a connection with the media server. 5. The media server responds. 6. The media player sends a PLAY message to start playing (downloading). 7. The audio/video file is downloaded by using another protocol that runs over UDP. 8. The connection is broken by using the TEARDOWN message. 9. The media server responds.

15. STREAMING LIVE AUDIONIDEO • Similar to the broadcasting just using the Internet • Streaming stored audio/video & streaming live audio/video Similarities: They are both sensitive to delay; neither can accept retransmission. Difference: Streaming stored media communication is unicast and on-demand. Streaming live media communication is multicast and live. Presently, live streaming is still using TCP and multiple unicasting instead of multicasting. It uses protocols such as UDP and RTP.

16. REAL-TIME INTERACTIVE AUDIONIDEO • Example : Internet phone or voice over IP & Video conferencing • Characteristics There are many features that can be considered in the application layer: 1. Time Relationship& jitter 2. Timestamp 3. Playback Buffer 4. Ordering 5. Multicasting 6. Translation 7. Mixing

17. Time Relationship: Perfect scenario Only 1 sec time difference between client and server

18. Real Scenario Packets arrive with different delays The client starts playing and packets arrive later► Jitter. Jitter is introduced in real-time data by the delay between packets.

19. Solutions 1-Timestamp The Packets include filed that shows the time it was produced relative to the previous packet. The receiver calculates the time to starts the playback without having the gaps by adding the timestamp to the playback time.

20. Solutions Cont.  What happened if we playback too early? too late??

21. Solutions Cont. 2- Playback Buffer We need a buffer to store the data until they are played back. Threshold: The maximum bits that buffer can hold before they are process The receiver delays playing the data until a threshold is reached. The replay does not start until the time units of data are equal to the threshold value.

22. Solutions Cont. If the delay < the play back time for the threshold amount there is no jitter

23. Solutions Cont. • Ordering We have to use sequence number for each packet to inform the receiver of packets • Multicasting We need multicasting to distribute heavy traffic. • Translation - Translator is needed when we have receivers with different data rate. - Translation means changing the encoding of a payload to a lower quality to match the bandwidth of the receiving network. - Translator decodes the signal and encode the signal again at lower rate  less bandwidth.

24. Solutions Cont. • Mixing - Mixing means combining several streams of traffic into one stream. • Ex: multiparty video conferencing with more than one session or source

25. Support from Transport Layer Protocol • TCP or UDP? Even though TCP does provide ordering (sequence), it is not suitable for interactive traffic • It has retransmission feature and error control mechanism. • Against idea of time-stamping and playback • It does not provision time-stamping • It does not support multicasting. • UDP is more suitable for interactive multimedia traffic It has no retransmission strategy. • But UDP has no provision for time-stamping, sequencing, or mixing???  We need new transport protocol RTP

26. Real-time Transport Protocol RTP is a protocol designed to handle real-time traffic on the Internet. It is an end to end functions RTP is encapsulated in UDP by using a temporary even-numbered UDP port.

27. Real-time Transport Protocol

28. Real-time Transport Control Protocol RTCP RTCP is a protocol designed to provide flow control and quality RTCP has five types of messages: Include statistics for all RTP packets, absolute timestamp for synchronization For passive participants (Qos feedback msg: packet lost, interval jiter..) Periodic message (SDES’s information) Source sends it; it is useful to the mixer. If the participants want to use new applications

29. VOICE OVER IP Allows us to: use the Internet as a telephone network with some additional capabilities. Instead of communicating over a circuit-switched network we have packet-switched Internet. Famous protocols - SIP Session Initiation Protocol - H.323

30. Summary • Streaming audio/video, their application, characteristics and requirements • Audio & video digitization & compression (ex: JPEG, MPEG) • Differences in multimedia applications: using: o Web server o Web server with a metafile o Media server o Media server and RTSP • Delay Jitter, playback buffer • Timestamp & the relationship between packets of a session.