1 / 72

MULTIMEDIA SIGNAL PROCESSING BASIC PROBLEMS IN PROCESSING MEDIA INFORMATION

MULTIMEDIA SIGNAL PROCESSING BASIC PROBLEMS IN PROCESSING MEDIA INFORMATION. Kinect – new media interface. Before we proceed we mention important development in the progress of media interfaces This is a device and system called Kinect made by Microsoft . Kinect is available as product

rhys
Download Presentation

MULTIMEDIA SIGNAL PROCESSING BASIC PROBLEMS IN PROCESSING MEDIA INFORMATION

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MULTIMEDIA SIGNAL PROCESSINGBASIC PROBLEMS IN PROCESSINGMEDIA INFORMATION MMSP Irek Defée

  2. Kinect – new media interface • Before we proceed we mention important development in the progress of media interfaces • This is a device and system called Kinect made by Microsoft. Kinect is available as product from the beginning of November 2010 Kinect is a part of Microsoft Xbox game platform but it can be bought separately! MMSP Irek Defée

  3. What is Kinect? • Kinect is a new type of hardware for interacting with people - with proper software support of course • Kinect looks like this MMSP Irek Defée

  4. What is inside Kinect? There is a hardware worth about 40 euro, working in the following schematics plus software which extracts signals and sends them for processing to Xbox. Processing takes about 5% of Xbox power (Xeon processor) MMSP Irek Defée

  5. How the Kinect works? Kinect has FOUR microphones to retrieve spatial sound and attenuate noise, interferences and compensate for room acoustics Kinect has small color camera with 640x480 resolution MMSP Irek Defée

  6. Most advanced aspect Kinect ”eyes” Eyes of Kinect are made by ab INFRARED MEASUREMENT SYSTEM- Laser beam is send from the objective and received by sensor as can be seen above. These sensors can move to adjust for the distance and height. This device produces MAP OF DEPTH to objects in a room. The device can thus ’see’ in bad light or in darkness. Before the use it is TRAINED with movements of persons in the room. You can see on the right that in infrared the beam makes lots of measurement dots MMSP Irek Defée

  7. What Kinect does? Kinect recognizes voice IN ROOMS and can be used for voice control of applications Kinect recognizes persons and body movements which is used in applications But before this Kinect is TRAINED interactively like shown in pictures After the training person and body movements will be recognized. More than one person can be identified in a scene MMSP Irek Defée

  8. Why Kinect is revolutionary? • It is the first practical natural interface for machines communicating with people • It works in normal rooms • It is combining acoustical and visual sense • It is recognizing full body movements, even complicated ones • It is recognizing persons • It works well, it is not perfect but one can predict there will be much more in the future MMSP Irek Defée

  9. Kinect applications Games and interactive playing (sports, dancing) More applications: exercising, rehbilitation, child development Control of devices by voice, gestures Automation, robotics More…. we do not know yet… but the public drivers are partially available MMSP Irek Defée

  10. Back to the lectures • We continue with the overview of the biological systems and priniciples of sensory information processing to finish it with some conclusions MMSP Irek Defée

  11. FROM PREVIOUS LECTURES WE KNOW THAT MULTIMEDIA INFORMATION PROCESSING IS EXCELLENTLY DONE BY THE HUMAN INFORMATION PROCESSING SYSTEM MMSP Irek Defée

  12. OUR PROBLEM IS: Biological systems perform processing of audiovisual information using special ”hardware” (which could be called ’wetware’) and ’software’ that is algorithms. The question is: Can we make processing of audiovisual information using different hardware and software? Maybe algorithms could be similar? MMSP Irek Defée

  13. Let us take visual processing as example IN HUMAN VISUAL SYSTEM PROCESSING STARTS IMMEDIATELY IN THE RETINA AND THERE ARE COLOR PROCESSING AND BLACK AND WHITE LIGHT ACQUISITION AND PROCESSING SYSTEMS MMSP Irek Defée

  14. FROM COLOR AND BLACK & WHITE RECEPTORS SIGNALS GO TO INITIAL PROCESSING ELEMENTS IT IS IMPORTANT TO NOTICE THAT THE NUMBER OF COLOR PROCESSING ELEMENTS IS MUCH LOWER THAN BLACK AND WHITE OUTPUT LINKS MMSP Irek Defée

  15. WHAT THESE PROCESSING ELEMENTS DO? I MOST RECENT MEASUREMENTS OF RETINAL NEURAL CELLS SHOW THAT THEIR RECEPTIVE FIELDS ARE QUITE IRREGULAR IN THE FOLLOWING PAGES SOME INFORMATION ABOUT WHAT THESE CELLS ARE DOING IS GIVEN MMSP Irek Defée

  16. BAR OF LIGHT IS MOVED OVER PHOTORCEPTORS IN DIFFERENT DIRECTIONS OUTPUT OF THE PHOTORECPTORS IS SUMMED WITH POSITIVE SIGN (EXICITATION) OR NEGATIVE SIGN (INHIBITION) MMSP Irek Defée

  17. DEPENDING ON THE DIRECTION OF MOTION SIGNALS SUM UP STRONGLY OR NOT MMSP Irek Defée

  18. HERE THE MEASURED SIGNALS ARE SHOWN FOR CELLS WHICH REACT STRONGLY TO WHITE BAR ON BLACK BACKGROUND AND OPPOSITE (off) MMSP Irek Defée

  19. HERE WE SEE THE RESPONSE MEASURED IN TIME MMSP Irek Defée

  20. WE CAN SEE THAT INITIAL PROCESSING IN THE EYE INCLUDES DETECTION OF DIRECTIONAL CHANGES IN LIGHT INTENSITY THIS MIGHT BE DONE FOR DIFFERENT COLORS TOO MMSP Irek Defée

  21. WE CAN NOW ASK FOLLOWING QUESTIONS: WHY THE PROCESSING IS ORGANISED IN THIS WAY? FOR THE ANSWER WE CAN THINK THAT THE PROCESSING IS OPTIMISED IN SOME WAY. WHAT MIGHT BE OPTIMISATION CRITERIA? WHAT ARE THE GENERAL PRINCIPLES OF HUMAN/BIOLOGICAL INFORMATION PROCESSING? MMSP Irek Defée

  22. OVERLAPPING SQUARES OR NOT??? MMSP Irek Defée

  23. THIS IS BECAUSE THE VISUAL SYSTEM PRODUCES INTERPRETATION WHICH IS MOST PLAUSIBLE (GENERIC) BUT IT MAY BE WRONG TOO, ALTHOUGH WE WOULD BE SURPRISED IT WOULD REALLY BE!!! • WHY WE SEE HERE THREE SQUARES AND NOT CUT OUT SQUARES? NOTE THAT ONLY ONE SQUARE IS FULLY VISIBLE, OTHERS ARE COVERED, IN FACT THEY MAY NOT BE SQUARES MMSP Irek Defée

  24. THE INTERPRETATION PRODUCED IS FOR DETECTING MOST PROBABLE OBJECTS THE UPPER FIGURE IS DETECTED AS ARCH OVERLAID ON THE SAWTOOH THIS IS THE MOST PROBABLE INTERPRETATION THE BOTTOM FIGURE INTERPRETATION IS SURPRISING, BUT IT COULD ALSO BE PRODUCED IF THERE WILL BE MORE EVIDENCE MMSP Irek Defée

  25. LIGHT DIRECTION • VISUAL SYSTEM ASSUMES THAT LIGHT IS COMING FROM TOP SAME PICTURE UPSIDE DOWN MMSP Irek Defée

  26. The statistics-based system works normally in almost perfect way. As we could see it fails sometimes when input signals are highly improbable and/or if most probable interpretation is not correct. This can be seen in visual illusions. We will look at them closer since recent statistical approach is explaining them. This provides for us a hint what kind of processing is done. MMSP Irek Defée

  27. WE CAN NOW ASK FOLLOWING QUESTIONS: WHY THE PROCESSING IS ORGANISED IN THIS WAY? FOR THE ANSWER WE CAN THINK THAT THE PROCESSING IS OPTIMISED IN SOME WAY. WHAT MIGHT BE OPTIMISATION CRITERIA? WHAT ARE THE GENERAL PRINCIPLES OF HUMAN/BIOLOGICAL INFORMATION PROCESSING? MMSP Irek Defée

  28. Principles we can identify now: • Statistical processing matched to the real world signal statistics – provides responses to most probable signals. This is very natural principle • Minimization of information processed, as much information as possible is eliminated, minimum information needed to provide response is used. This principle allows to minimize energy and processing effort. MMSP Irek Defée

  29. A book which appeared in 2005 based on earlier research: MMSP Irek Defée

  30. The authors are visual psychologists, they consider vision as a system interpreting world from images projected onto the eye: Light from external source bounces of objects and is projected. This projection is not unique (e.g. objects of different size will have the same projection depending on their distance MMSP Irek Defée

  31. In visual illusions projection gives rise to improper interpretation Stimuli changes, illusion persists, Natural scene, illusion persists MMSP Irek Defée

  32. This picture gives strong of depth • because of combination of many • mutually consistent cues: • perspective • texture gradient • Shading and shadow MMSP Irek Defée

  33. Geometry of natural scenes Geometrical illusions represent wrong interpretation od real world. To find out why researchers took pictures with depth map Real pictures with corresponding distances marked by colors Laser range scanner for Measuring distance MMSP Irek Defée

  34. If large number of such pictures is taken a database can be created in which real world objects are matched with distances and statistics is calculated. Example: subjective metrics Let’s think about lines of different lengths which are seen in real world. If all length would have the same probability there would be linear relation between the stimulation for every length. But if this is not the case, some length will be stimulated more often. This can lead to distortions in perception. MMSP Irek Defée

  35. Example: Line length illusion Variation of apparent length as function of orientation In experiments people report changing length depending on angle MMSP Irek Defée

  36. The points in the picture were compared with measured by laser range to see if they correspond to lines in real world. Total of 1.2x10^7 line segments were collected • Why it is so? Let’s sample lines in pictures from database Grid of templates to overlay on picture with straight lines White – accepted lines, Black – rejected lines Probability distribution of of lines vs. length for different orientations Cumulative distribution (lines shorter than x) This shows how many lines at certain orientation corresponded to real lines of length shorter or equal to x MMSP Irek Defée

  37. Prediction of apparent length based on probability Take e.g lines of length 7 at orientation 20 deg, their cumulative probability is 0.15 which means that 15% lines is shorter than 7 pixels and 85% is longer. For all orientations we get this plot This is very smilar to the one measured in experiments with people!!! MMSP Irek Defée

  38. Why such biases exist? In nature lines do not appear often, horizontal lines are typically generated from horizontal flat surfaces Vertical lines are limited by gravity and by this rare and lines at 20-30deg even more Rare, and they are mostly projected from perspective MMSP Irek Defée

  39. Visual illusions: Angles All angles in this picture have 90 deg but when they are projected on the eye, projections may differ up to 60 deg • Bias in angle estimation between two • lines • B,C,D) Angle illusions MMSP Irek Defée

  40. To explain this a database of angles is made, as before Probability distributions for different Types of angles (bottom line) in natural scenes and scenes with human created objects We can see bias: angles close to 90 deg are less likely to occur Extraction of angles MMSP Irek Defée

  41. Probability distribution of angles is not linear, cumulative probability is biased Thus predicted perceived angle is different from actual one, for 90 deg it is the same The magnitude of angle misperception (lines) vs. experimentally measured values • Bias and illusions Angles close to 90 deg are more likely To come from planar surface, which is typically larger than surface from lines interesecting at smaller angles. Thus 90 deg angles are less likely MMSP Irek Defée

  42. Explanation of angle illusions Why vertical line is tilted? We take reference line at 60 deg (black) and check probability of occurence of physical sources of a second line oriented at different angles. Since the angle between the lines is 30 deg we look at the probability for 30 deg and then into cumulative probability (previous page) which gives value 0.184 which multiplied by 180 gives angle 33,2 deg in agreement with measurements MMSP Irek Defée

  43. According to the previous explanations the reason for this illusion is: Probability distributions of the possible sources of the targets, given their different contexts, are different To check this hypothesis database was searched for circular objects and probabilities of the sources of targets in the context were calculated: • Size illusion Various size illusions of center and surrounding MMSP Irek Defée

  44. Experimental conditions a) The inner circle is surrounded by the 4 circles with changing diameters b) Probability of occurence of center circle with specific size for outer circles with different diameters. Dashed line shows probability for circle with 14 pixels diameter. (Bigger surrounding circles are much less likely to appear) c) Cumulative probability for 14 pixel circle d) Examples of scenes with large circles and small circles Why there are statistical differences? Circles originate from planar projections, larger circles are less likely. Why the presence of surrounding circles changes the occurence of target central circles differently? Larger circles arise from larger planes in the world, they are flat areas – then it is more probable that the central circle will be larger. In other words, the presence of larger surrounding circles increases the probability of of occurence of physical sources of larger central circles. In result probability Distribution of central circles is changing according to the size of surrounding circles. MMSP Irek Defée

  45. Changing the interval between center and surrounding circles Cumulative Probability for the 14 pixel circle Probabilities when the distance is changing Dashed line is for circle of size 14 MMSP Irek Defée

  46. Probability distribution of • singel circle vs. diameter • Probability for single circle • superimposed with probability • of central circle surrounded • by outer circle, dashed line • is for 24 pixel circle, probability • curve is for outer circle 32 pixel • diameter, cumulative probability • is much higher – there is bias • When the outer circle is much • bigger the cumulative • probability is smaller • Comparison of inner circle with single circle The changing cumulative probability ratios and dependence on the central and outer circle sizes is well seen – and illusion depends on these parameters in exactly the same way MMSP Irek Defée

  47. When objects are close perceived distance is overestimated to physical one • Objects which are close to each other are perceived as being at the same distance • The distance to close objects is overestimated, the distance to far objects is underestimated • Objects on the ground when they are about 7m distance appear closer and with increasing distance they appear more elevated • Distance illusions MMSP Irek Defée

  48. According to the methodology probability distribution of distances is measured but there are several variables here: Probability of all distances from scanner Probability of the differences in distances between objects for three different horizontal angles Probability of horizontal distances different heights with respect to eye level MMSP Irek Defée

  49. This curve for all distances has strong peak for distance • of 3m . This is in agreement with experiments in which • people seeing single objects hanging in completely dark • scene report them as being in the distance of 2-4 m • Interpretation of these probabilities • When the angular separation between the objects is small • they tend to be seen at equal distance but this tendency • decreases when the angle is increasing • The dependency of probability of distance vs. eye level • has peak at distance of 4 m. Thus for objects at distance • less than 4 m will be overestimated and those at • distance more than 4 m will be underestimated. • This agrees with experiments MMSP Irek Defée

  50. Why this happens? Again, for explanation database is searched for such patterns and probabilities are calculated. Here we consider case when both gigures ar inline, on the left/right The size illusion does not depend nn particular type of endings • The size illusion Templates used It can be induced even without line and even (but less strongly) with dots Templates overlaid on pictures MMSP Irek Defée

More Related