REQUIREMENTS FOR MULTIMEDIA PROCESSING

1. REQUIREMENTS FORMULTIMEDIA PROCESSING MULTIMEDIA SYSTEMS IREK DEFEE

2. WHAT ARE THE RQUIREMENTS FOR HANDLING MULTIMEDIA DATA ? 1. PERCEPTUAL REQUIREMENTS For the user we would like to provide maximal perceptual stimulation THIS REQUIRES: - OPTIMAL STIMULATION OF EACH SENSE - OPTIMAL COORDINATION OF ALL

3. SENSES - INVOLVEMENT OF POSSIBLE MANY SENSES (ACTIVE) THIS MEANS INVOLVEMENT OF: - ACOUSTICAL (HEARING,VOICE) - VISUAL - HAPTIC (TOUCH, MOTORIC, VESTIBULAR (orientation in space) ) ON FULL SCALE (e.g. SPATIAL PROPERTIES)

4. THE PROBLEM IS HOW THIS CAN BE DONE TECHNICALLY: - ACOUSTICAL STIMULATION THIS IS NOW RELATIVELY EASY (SPATIAL (MULTICHANNEL SOUND) - VISUAL STIMULATION SPATIAL (3-D) IMAGES ARE PROBLEM, HIGH RESOLUTION TOO

5. HAPTIC STIMULATION THERE ARE MANY INPUT DEVICES (JOYSTICKS), BUT FULL STIMULATION INCLUDING MOTOR SENSE, SKIN SENSORS AND VESTIBULAR SYSTEM IS DIFFICULT IN FACT WE CAN TALK ABOUT TWO TYPES OF SYSTEMS FOR STIMULATION:

6. - CLOSELY COUPLED TO SENSORY SYSTEM (BY GLOVES, UNIFORMS, EYE DISPLAYS, HEADPHONES, MOVING SIMULATORS) THEY ALLOW ”FULL IMMERSION”, OR CREATION OF VIRTUAL REALITY MIGHT BE TIRING TO USE - LOOSELY COUPLED TO SENSORY SYSTEM (NORMAL SCREENS, SPEAKERS, JOYSTICKS ETC.) MIGHT BE MORE COMFORTABLE

7. IN THIS COURSE WE ARE TALKING ABOUT ’LOOSELY’ COUPLED MULTIMEDIA SYSTEMS WITH STANDARD SENSORY INTERFACES VIRTUAL REALITY IS A TOPIC OF OTHER COURSES

8. WHAT ARE THE TECHNICAL REQUIREMENTS FOR MULTIMEDIA SYSTEMS ? • HANDLING OF LARGE DATA SETS E.G. AUDIO 5 CHANNELS X 16-BIT X 44 000 SAMPLES/SEC = 3 520 000 BITS/SEC ~450 KB/sec = 27 MB/min=1,6GB/h

9. SECOND EXAMPLE, VIDEO RESOLUTION OF STANDARD TV PICTURE IS 720X576 POINTS, EACH POINT IS 24-BIT (8-BIT R, G, B) THERE ARE 25 PICTURES PER SECOND SO THE DATA RATE IS 720X576X25X24~ 250 Mb/sec = 1.8GB/min = 108 GB/h FOR HIGHER RESOLUTIONS:

10. A FORMAT FOR HIGH-DEFINITION TV IS PROPOSED PICTURE SIZE IS 1920X1080 POINTS IT IS ABOUT 5X TIMES LARGER THAN STANDARD TV. DATA RATES ARE THEN, 1.3 Gb/sec, 9 GB/min, 540 GB/h DATA RATES FOR AUDIO AND VIDEO ARE NOT PRACTICAL, COMPRESSION IS REQUIRED

11. - COORDINATION AND SYNCHRONIZATION - HUMAN SENSORY SYSTEM IS VERY SENSITIVE TO COORDINATION OF EVENTS IN TIME AND SPACE MULTIMEDIA DATA, ESPECIALLYAUDIO AND VIDEO DEPEND CRITICALLY ON TIME AND SPACE

12. EXAMPLES: - LIP SYNCHRONIZATION IN TV AND CINEMA, VOICE AND LIP MOVEMENT MUST BE SYNCHRONIZED, OTHERWISE IT IS VERY ANNOYING SYNCHRONIZATION ERROR FOR BIG PICTURES MUST BE IN THE LIMIT OF +20 ; –40 MILISECONDS

13. - SPATIAL SOUND HEARING SYSTEM ESTABLISHES LOCATION OF SOUND SOURCES BASED ON DIFFERENCE OF TIME OF ARRIVAL AT EARS PRECISION REQUIRED IS ON THE ORDER OF SEVERAL 10’s OF MICROSECONDS.

14. THIRD REQUIREMENT: FAST RESPONSE WHENEVER WE RESPOND (E.G. BY CLICKING), WE EXPECT IMMEDIATE RESPONSE. IMMEDIATE MEANS ”NO NOTICEABLE DELAY” FOR PEOPLE. IT MEANS LESS THAN 300 ms=0.3 sec

15. THUS, MULTIMEDIA DATA REQUIRE HIGH-DATA RATE & TIME/SPACE CRITICAL PROCESSING WITH FAST RESPONSE TO EXTERNAL EVENTS THESE ARE ATTRIBUTES OF REAL-TIME PROCESSING

16. DEMANDS FOR COMPUTATIONAL POWER ARE HIGH • ONLY RECENTLY THIS BECAME POSSIBLE

17. Multimedia systems requirements Recall the multimedia system is for service offering: CONSUMER CONTENT DELIVERY DISTRIBUTION

18. What users expect from a service they use? • They expect satisfying Quality of Service – QoS QoS includes technical aspects which are most important for engineers (e.g. good communication link) QoS in broad sense includes all other aspects which make customer satisfied (e.g. support, price)

19. What is QoS ? • QoS is degree to which customer expectations are satisifed • QoS includes technical parameters for quality: e.g. reliability of communication • QoS may also include other parameters for satisfaction (attraction, support, price, etc.)

20. What service owners expect? They expect proper environment that they can run their service Especially that their buisness can run without problems Unfortunately for network buisnesses there are big problems with this What is absolutely necessary is SERVICE PROTECTION

21. Good buiness over the network needs as the most critical requirement STRONGSERVICE PROTECTION CONSUMER CONTENT DELIVERY DISTRIBUTION Every part of the system has to be strongly protected against those who want to destroy the service integrity

22. Overall architecture of multimedia system There are two ”layers”: Hardware layer: servers, networks and terminals Software layer: content, service, provisioning, networking, middleware, applications

23. Multimedia system • Composed of hardware/software components • Goal: Creation and delivery of media content from production to consumer at proper QoS in an economic way This means business is sustainable and consumers are satisfied

24. Examples of multimedia systems Content delivery system examples: • Television/(radio?) (old, basic system) • Web/Internet type • Mobile (radio networks) Content distribution type examples: • Music • Movies (audio/visual content • Games • Virtual worlds (Check Second life on the Web) • eBooks? eMagazines? eNewspapers?