Advanced Multimedia

1. Advanced Multimedia Image Content Analysis Tamara Berg

2. Announcements • HW2 due March 13, 11:59pm • Questions? • HW3 online Thurs

3. How are images stored?

4. Images Images are sampled and quantized measurements of light hitting a sensor.

5. Images in the computer

6. Images in the computer

7. Color Images img = Ellsworth Kelly Red Blue Green, 1963

8. Color Images img = Ellsworth Kelly Red Blue Green, 1963 N

9. Color Images M img = Ellsworth Kelly Red Blue Green, 1963 N

10. Color Images M • Stored as 3d matrix of r, g, and b values at each pixel • Image matrix would be size NxMx3 • ? = img(:,:,1) ? = img(:,:,2) ? = img(:,:,3) img = Ellsworth Kelly Red Blue Green, 1963 N

11. Color Images M • Stored as 3d matrix of r, g, and b values at each pixel • Image matrix would be size NxMx3 • R = img(:,:,1) G = img(:,:,2) B = img(:,:,3) img = Ellsworth Kelly Red Blue Green, 1963 N

12. Red component img(:,:,1) =

13. Green component img(:,:,2) =

14. Blue component img(:,:,3) =

15. Color Images M • Stored as 3d matrix of r, g, and b values at each pixel • How could you get the r,g,b values for pixel (i,j)? img = Ellsworth Kelly Red Blue Green, 1963 N

16. Color Images M • Stored as 3d matrix of r, g, and b values at each pixel vals = img(i,j,:); Then r= vals(1); g = vals(2); b = vals(3); img = Ellsworth Kelly Red Blue Green, 1963 N

17. Color Images M j • Stored as 3d matrix of r, g, and b values at each pixel • So r,g,b values are stored in img(i,j,:). • In the case above these values might be [1 0 0]. i img = Ellsworth Kelly Red Blue Green, 1963 N

18. RGB color space • RGB cube • Easy for devices • But not perceptual • Where do the grays live? • Where is hue and saturation? Slide by Steve Seitz

19. HSV Cone • Hue, Saturation, Value (Intensity/Brightness) • Use rgb2hsv() and hsv2rgb() in Matlab

20. How should we represent images?

21. Motivation • Image retrieval • We have a database of images • We have a query image • We want to find those images in our database that are most similar to the query

22. Motivation • Image retrieval • We have a database of images • We have a query image • We want to find those images in our database that are most similar to the query • Similarly to text retrieval, & music retrieval we first need a representation for our data.

23. How should we represent an image?

24. First try Just represent the image by all its pixel values

25. First try Img1 = Say we measure similarity as: sim = sum(abs(img1 – img2)) Img2 =

26. First try How similar are these two images? Is this bad? Img1 = Say we measure similarity as: sim = average diff between values in img1 and img2 Img2 =

27. What do we want? • Features should be robust to small changes in the image such as: • Translation • Rotation • Illumination changes

28. Second Try Photo by: marielito Represent the image as its average pixel color

29. Second Try Photo by: marielito Represent the image as its average pixel color Pros? Cons?

30. Third Try Photo by: marielito Represent the image as a spatial grid of average pixel colors Pros? Cons?

31. QBIC system • First content based image retrieval system • Query by image content (QBIC) • IBM 1995 • QBIC interprets the virtual canvas as a grid of coloured areas, then matches this grid to other images stored in the database. QBIC link

32. Color is not always enough! The representation of these two umbrella’s should be similar…. Under a color based representation they look completely different!

33. What next? Edges! But what are they & how do we find them?

34. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo) signal f = filter g =

35. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo) * * * =

36. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo) * * * =

37. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo) * * * =

38. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo) * * * =

39. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo) * * * =

40. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo) * * * Convolution computes a weighted average. =

41. Images -> 2d filtering

42. Moving average • Let’s replace each pixel with a weighted average of its neighborhood • The weights are called the filter kernel • What are the weights for a 3x3 unweighted moving average? Source: D. Lowe

43. 1 1 1 1 1 1 1 1 1 “box filter” Moving average • Let’s replace each pixel with a weighted average of its neighborhood • The weights are called the filter kernel • What are the weights for a 3x3 unweighted moving average? Source: D. Lowe

44. 1 1 1 1 1 1 1 1 1 “box filter” g(x,y)

45. Moving Average In 2D Source: S. Seitz

46. Moving Average In 2D Source: S. Seitz

47. Moving Average In 2D Source: S. Seitz

48. Moving Average In 2D Source: S. Seitz

49. Moving Average In 2D Source: S. Seitz

50. Moving Average In 2D What is this filter doing? Source: S. Seitz