Advanced Techniques in Image Database Retrieval

1. Lecture 25 Searching Image Databases CSE 6367 – Computer Vision Spring 2010 Vassilis Athitsos University of Texas at Arlington

2. Image Databases • Photographs.

3. Image Databases • Art clips.

4. Video Databases • Movies.

5. Video Databases • Movies, TV footage, YouTube, …

6. Retrieval and Indexing • Retrieval: identifying content of interest. • Indexing: preprocessing the data, so as to allow fast retrieval.

7. Holy Grail: Query by Content • Find pictures of a rocky coast.

8. Holy Grail: Query by Content • Find pictures of John with trees and snow in the background..

9. Problem: Knowing the Content • How can computers recognize rocky coasts?

10. How Does Google Image Work?

11. How Does Flickr Work?

12. Comparing Sources of Information • Learning to recognize classes of images: • Plus: once they are built, they run by themselves. • Minus: very costly to build. • They require tons of training data, labeled by humans. • Minus: they tend to be very inaccurate in practice. • Accuracy improves year by year… • Minus: they can be very slow.

13. Comparing Sources of Information • Surrounding text: • Plus: in some cases, readily available. • E.g.: web pages. • Plus: when available, it can be used for free. • Plus: oftentimes pretty accurate. • Minus: in some cases, not available. • E.g.: many photo albums. • Minus: oftentimes pretty inaccurate. • You will never find a picture of Michael Jordan unless the words “Michael Jordan” are nearby.

14. Comparing Sources of Information • Manual labeling: • Plus: labels anticipate what users would look for. • Improves accuracy. • Minus: in some cases, not available. • E.g.: many photo albums, web pages. • Minus: oftentimes pretty inaccurate. • It is hard to anticipate what keywords a user will employ: • “bike ride in the sunset” vs. “sports activity at the end of the day.”

15. Similarity-Based Retrieval • Query: an example of what we are looking for. query results

16. Describing Color • How do we describe the color content of this image? query

17. Describing Color • How do we describe the color content of this image? • Green and blue. query

18. Describing Color • How do we describe the color content of this image? • Green and blue. • Green at the bottom, blue at the top. query

19. Describing Color • How do we describe the color content of this image? • A command the computer can understand: • Find images that are green and blue like this. query

20. A Color Histogram Example • Choose 100 representative colors (or any other number). • In conventional image formats, 16 million colors. • Create an array (color histogram) of length 100. • For each color: • Count how many pixels in the image are closer to that color than to any other of the 100 colors. • Store result in the array. representative colors query

21. Comparing Histograms • (x1, x2, …, x100) | x1 - y1| + … + | x100 – y100| • (y1, y2, …, y100) | x1 - z1| + … + | z100 – z100| • (z1, z2, …, z100) query a database image another database image

22. Expected Results similar images query results

23. Expected Results similar images query results

24. Expected Results query Colors do not match well enough

25. Beyond Color Histograms • Some times, shape information is important.

26. Shape-Based Similarity Measures • Chamfer Distance. • Shape Context.

27. Shape Context • Choose r1, r2, …, rk • Choose s = number of sectors. • Create a template consisting of rings and sectors, as shown in the image. • Give a number to each sector of each ring. • For each edge pixel: • Center the template on the pixel. • For each sector of each ring, count the number of edge pixels in that sector. • Result: each point ismapped to ? numbers. source: Wikipedia

28. Shape Context • Choose r1, r2, …, rb • Choose s = number of sectors. • Create a template consisting of rings and sectors, as shown in the image. • Give a number to each sector of each ring. • For each edge pixel: • Center the template on the pixel. • For each sector of each ring, count the number of edge pixels in that sector. • Result: each point ismapped to sb numbers. source: Wikipedia

29. Shape Representation • Pick T points from each shape, uniformly sampled. • Extract, for each point, the shape context vector. • Then, each shape is represented as a matrix of size ? source: Wikipedia

30. Shape Representation • Pick T points from each shape, uniformly sampled. • Extract, for each point, the shape context vector. • Then, each shape is represented as a matrix of size T * k. • T: number of points we pick from each shape. • k = s * b. • s: number of sectors in each ring. • b: number of rings. source: Wikipedia

31. Shape Matching • Each shape is mapped to a matrix of size T*k. • T: number of points we pick from each shape. • k = s * b. • s: number of sectors in each ring. • b: number of rings. • What is the cost of matching two shapes?

32. Shape Matching • Each shape is mapped to a matrix of size T*k. • T: number of points we pick from each shape. • k = s * b. • s: number of sectors in each ring. • b: number of rings. • What is the cost of matching two shapes? • Simpler question: what is the cost of matching two shape contexts?

33. Shape Matching • Each shape is mapped to a matrix of size T*k. • T: number of points we pick from each shape. • k = s * b. • s: number of sectors in each ring. b: number of rings. • What is the cost of matching two shapes? • Simpler question: what is the cost of matching two shape contexts? • One answer: Euclidean or Manhattan distance. • Better answer: chi-square distance. • g(k) and h(k): k-th valuesof the two shape contexts.

34. Shape Matching • Each shape is mapped to a matrix of size T*k. • T: number of points we pick from each shape. • k = s * b. • s: number of sectors in each ring. b: number of rings. • What is the cost of matching two shapes? • Key problem: we do not know what point in one image corresponds to what point in the other image. • Solution: find optimal 1-1 correspondences. • The cost of each correspondence is the matching cost of the shape contexts of the two corresponding points. • What algorithm can be used here?

35. Shape Matching • Each shape is mapped to a matrix of size T*k. • T: number of points we pick from each shape. • k = s * b. • s: number of sectors in each ring. b: number of rings. • What is the cost of matching two shapes? • Key problem: we do not know what point in one image corresponds to what point in the other image. • Solution: find optimal 1-1 correspondences. • The cost of each correspondence is the matching cost of the shape contexts of the two corresponding points. • This is a bipartite matching problem. • Solution: Hungarian Algorithm. • Complexity:

36. Shape Matching • Each shape is mapped to a matrix of size T*k. • T: number of points we pick from each shape. • k = s * b. • s: number of sectors in each ring. b: number of rings. • What is the cost of matching two shapes? • Key problem: we do not know what point in one image corresponds to what point in the other image. • Solution: find optimal 1-1 correspondences. • The cost of each correspondence is the matching cost of the shape contexts of the two corresponding points. • This is a bipartite matching problem. • Solution: Hungarian Algorithm. • Complexity: cubic to the number of points.

37. Shape Context Distance • Proposed by Belongie et al. (2001). • Error rate: 0.63%, with database of 20,000 images. • Uses bipartite matching (cubic complexity!). • 22 minutes/object, heavily optimized.