Advanced Multimedia - PowerPoint PPT Presentation

advanced multimedia n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Advanced Multimedia PowerPoint Presentation
Download Presentation
Advanced Multimedia

play fullscreen
1 / 76
Advanced Multimedia
132 Views
Download Presentation
trory
Download Presentation

Advanced Multimedia

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Advanced Multimedia Image Content Analysis Tamara Berg

  2. Announcements • HW2 due March 13, 11:59pm • Questions? • HW3 online Thurs

  3. How are images stored?

  4. Images Images are sampled and quantized measurements of light hitting a sensor.

  5. Images in the computer

  6. Images in the computer

  7. Color Images img = Ellsworth Kelly Red Blue Green, 1963

  8. Color Images img = Ellsworth Kelly Red Blue Green, 1963 N

  9. Color Images M img = Ellsworth Kelly Red Blue Green, 1963 N

  10. Color Images M • Stored as 3d matrix of r, g, and b values at each pixel • Image matrix would be size NxMx3 • ? = img(:,:,1) ? = img(:,:,2) ? = img(:,:,3) img = Ellsworth Kelly Red Blue Green, 1963 N

  11. Color Images M • Stored as 3d matrix of r, g, and b values at each pixel • Image matrix would be size NxMx3 • R = img(:,:,1) G = img(:,:,2) B = img(:,:,3) img = Ellsworth Kelly Red Blue Green, 1963 N

  12. Red component img(:,:,1) =

  13. Green component img(:,:,2) =

  14. Blue component img(:,:,3) =

  15. Color Images M • Stored as 3d matrix of r, g, and b values at each pixel • How could you get the r,g,b values for pixel (i,j)? img = Ellsworth Kelly Red Blue Green, 1963 N

  16. Color Images M • Stored as 3d matrix of r, g, and b values at each pixel vals = img(i,j,:); Then r= vals(1); g = vals(2); b = vals(3); img = Ellsworth Kelly Red Blue Green, 1963 N

  17. Color Images M j • Stored as 3d matrix of r, g, and b values at each pixel • So r,g,b values are stored in img(i,j,:). • In the case above these values might be [1 0 0]. i img = Ellsworth Kelly Red Blue Green, 1963 N

  18. RGB color space • RGB cube • Easy for devices • But not perceptual • Where do the grays live? • Where is hue and saturation? Slide by Steve Seitz

  19. HSV Cone • Hue, Saturation, Value (Intensity/Brightness) • Use rgb2hsv() and hsv2rgb() in Matlab

  20. How should we represent images?

  21. Motivation • Image retrieval • We have a database of images • We have a query image • We want to find those images in our database that are most similar to the query

  22. Motivation • Image retrieval • We have a database of images • We have a query image • We want to find those images in our database that are most similar to the query • Similarly to text retrieval, & music retrieval we first need a representation for our data.

  23. How should we represent an image?

  24. First try Just represent the image by all its pixel values

  25. First try Img1 = Say we measure similarity as: sim = sum(abs(img1 – img2)) Img2 =

  26. First try How similar are these two images? Is this bad? Img1 = Say we measure similarity as: sim = average diff between values in img1 and img2 Img2 =

  27. What do we want? • Features should be robust to small changes in the image such as: • Translation • Rotation • Illumination changes

  28. Second Try Photo by: marielito Represent the image as its average pixel color

  29. Second Try Photo by: marielito Represent the image as its average pixel color Pros? Cons?

  30. Third Try Photo by: marielito Represent the image as a spatial grid of average pixel colors Pros? Cons?

  31. QBIC system • First content based image retrieval system • Query by image content (QBIC) • IBM 1995 • QBIC interprets the virtual canvas as a grid of coloured areas, then matches this grid to other images stored in the database. QBIC link

  32. Color is not always enough! The representation of these two umbrella’s should be similar…. Under a color based representation they look completely different!

  33. What next? Edges! But what are they & how do we find them?

  34. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo) signal f = filter g =

  35. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo) * * * =

  36. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo) * * * =

  37. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo) * * * =

  38. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo) * * * =

  39. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo) * * * =

  40. Filtering Alternatively you can convolve the input signal with a filter to get frequency limited output signal. Convolution: (convolution demo) * * * Convolution computes a weighted average. =

  41. Images -> 2d filtering

  42. Moving average • Let’s replace each pixel with a weighted average of its neighborhood • The weights are called the filter kernel • What are the weights for a 3x3 unweighted moving average? Source: D. Lowe

  43. 1 1 1 1 1 1 1 1 1 “box filter” Moving average • Let’s replace each pixel with a weighted average of its neighborhood • The weights are called the filter kernel • What are the weights for a 3x3 unweighted moving average? Source: D. Lowe

  44. 1 1 1 1 1 1 1 1 1 “box filter” g(x,y)

  45. Moving Average In 2D Source: S. Seitz

  46. Moving Average In 2D Source: S. Seitz

  47. Moving Average In 2D Source: S. Seitz

  48. Moving Average In 2D Source: S. Seitz

  49. Moving Average In 2D Source: S. Seitz

  50. Moving Average In 2D What is this filter doing? Source: S. Seitz