Efficient Image Retrieval Approaches for Different Similarity Requirements

1. Efficient Image Retrieval Approachesfor Different Similarity Requirements C.Y. Tsai, A.L.P. Chen and K. Essig Proc. SPIE Conference on Storage and Retrieval for Image and Video Databases, 2000

2. Outline • Introduction • Feature Extraction • Query Processing • Similarity Measures • Experiments • Conclusion

3. Introduction • The issues of image retrieval • effectiveness（or accuracy） • efficiency • similarity requirements of images

4. Image-A Image-B Image-C Introduction (cont.) • Similarity requirements of images • color information • spatial information

5. Introduction (cont.) • The partition-based approach • m×n equal-size sub-images (or blocks) • one color is extracted from a block • Similarity requirements of images • similar color composition in the same area • The region-based approach • region extraction：block-level process • Similarity requirements of images • similar color composition • spatial information is unimportant

6. Introduction (cont.) • Definition • neighbor colors of a color • the colors adjacent to it in the color space • dominant color of a block • maximum number of pixels • representative color of a block • a dominant color • has a large enough number of pixels

7. Feature Extraction • Color model • RGB model vs. HSV model

8. Feature Extraction (cont.) • Representative color of a block • a dominant color • two-stage examination • threshold：30% • the 1st stage：only the dominant color • the 2nd stage：consider its neighbor colors • fail the examination • no representative color in this block

9. Feature Extraction (cont.) • The features for the partition-based approach • the representative colors of the blocks

10. Feature Extraction (cont.) • The region-based approach • region extraction • an example • three properties of a region • shape of region • MBR (minimum bounding rectangle) • the ratio of the shorter edge to the longer edge of the MBR • size of region • the number of blocks contained in the region • representative color of region

12. C 10 Filter by the size of region (10%)  40 * 10% = 4 A 5 B 13 • The properties of region C • ratio of region = 3/4 = 0.75 • size of region：10blocks • representative color of region：brown

13. Query Processing • The partition-based approach • image similarity：similar colors in the corresponding blocks • similarity between blocks • corresponding blocks • both have a representative color • the distance of two colors in the color space • similarity between images • the average similarity degree of the blocks

14. Query Processing (cont.) • The region-based approach • image similarity：similarity of regions of the query image and database image • each region in the query image has a similarity degree • a weighted summationbased on the size of regions

15. Query Processing (cont.) • similarity between regions • based on the three properties • the regions in the query image match with the regions in the database image • one-to-onematch • matching of regions is determined by maximizing the summation of the region similarity degrees • unmatched：similarity degree is zero

16. C 5 B 10 G 5 F 9 A 13 D 11 E 8 Query image Database image A matchs with D B matchs with F C matchs with E

17. Similarity Measures • The partition-based approach k is the # of blocks that both have a representative color

18. 1 (5,2,2) 2 (9,1,2) 1 (4,3,1) 2 (8,1,1) 3 4 (2,2,2) 3 (3,3,1) 4 (12,1,1) Similarity Measures (cont.) Quantization Scheme：15*3*3 d = 5 11：  1.0 - (1.73/5) = 0.65 22：  1.0 - (1.41/5) = 0.72 33：  44：  0.0

19. Similarity Measures (cont.) • The region-based approach • the similarity function for regions

20. Similarity Measures (cont.) • the similarity function for images n, m are the numbers of regions in the query image and the database image, respectively  denotes〝Un-matched〞

21. Similarity Measures (cont.) SIZE = 13 + 10 + 5 = 28

22. Experiments • The evaluation policy • Precision and Recall; Precision vs. Recall relevant set retrieved set A B

23. Experiments (cont.) • Two data sets • 150 images each with 192*128 pixels • data set A • images with similar color composition and distribution • data set B • some images with similar color compositionbut different distributions

24. The number of major colors (2117 images)

25. Experiments (cont.) • Deciding representative colors • major colors: colors that have enough number of pixels • 30% was chosen as the threshold • if the quantization is too fine, it is difficult to find a representative color; if it is too rough, it will affect the accuracy of the queries • smaller block sizes are more suitable for the region-based approach

26. Experiments (cont.) • The choice of color model • three approaches • the histogram-based approach • the partition-based approach • the region-based approach • the value of precisionand recall • based on the first 15 retrieved images

27. The histogram-based approach • image similarity depends on the difference ofthe numbers of pixelsin the corresponding color bins RGB Model HSV Model

28. The partition-based approach • image similarity depends on the difference of the colors RGB Model HSV Model

29. The region-based approach • image similarity depends on the difference of the colors RGB Model HSV Model

30. Experiments (cont.) • Summary • HSV color model is better than RGB color model in our approaches • parameters • the size of block：8 * 8 • the quantization scheme：15 * 3 * 3

31. Experiments (cont.) • Effectiveness • two data sets for different similarity requirements of images • three approaches • the relationship between precision and recall • Efficiency • the average execution time for a single query

32. Dataset-A for three approaches

33. Dataset-B for three approaches

34. the execution time for a single query

35. Experiments (cont.) • The effectiveness of the partition-based and the histogram-based approaches is similar • less information does not decrease the accuracy (in the first experiment) • more information can cause less accuracy (in the second experiment)

36. Conclusion • We propose two efficient and effective image retrieval approaches • block-based information • representative colors of the blocks • region extraction：block-level process • similarity requirements of images • Future Work • representation of regions • spatial relationship between regions • index structure