Content Based Image Retrieval Using MPEG-7 Dominant Color Descriptor

1. Content Based Image Retrieval Using MPEG-7 Dominant Color Descriptor Student : Mr. Ka-Man Wong Supervisor : Dr. Lai-Man Po MPhil Examination Department of Electronic Engineering City University of Hong Kong August 2004

2. Outlines of this presentation • Objectives • MPEG-7 visual descriptors • A new similarity measure for dominant color descriptor • Merged Palette Histogram Similarity Measure • A new relevance feedback for dominant color descriptor • Merged Palette Histogram Relevance Feedback • MIRROR – A CBIR system using MPEG-7 visual descriptors • Conclusions

3. Objective of this research study • To investigate Content Based Image Retrieval (CBIR) based on color features • To develop efficient techniques for MPEG-7 Dominant Color Descriptor (DCD) • Merged Palette Histogram Similarity Measure • Merged Palette Histogram Relevance Feedback • Apply proposed methods into a real system

4. MPEG-7 visual descriptors • Color • Color structure, scalable color, dominant color, color layout • Texture • Homogeneous texture, edge histogram, texture browsing • Shape • Contour shape, region shape, 3D shape • Motion (for video contents) • Motion activity, camera motion, motion trajectory, parametric motion • They describe image/video contents in different aspects

5. MPEG-7 color descriptors • Dominant color descriptor (DCD) • A compact color descriptor generated by color quantization • Color structure descriptor (CSD) • Color histogram generated by structure block scanning approach • Scalable color descriptor (SCD) • Color histogram in a quantized HSV space with Haar transform. • Color layout descriptor (CLD) • A compact color-spatial descriptor generated by dividing the image by a 8x8 gird with DCT transform.

6. Relevance feedback • Color might perform well, but it might not match user’s expectation • Effectiveness could be further improved by involving users in the searching

7. MPEG-7 color descriptors • Two major problems are found in DCD make it unable to perform well • Problems of its original similarity measure method • Cannot use relevance feedback easily • We will focus on DCD in this research study • New methods are developed to utilize DCD • Similarity measure • Relevance feedback

8. Merged palette histogram similarity measure for dominant color descriptor Dominant Color Descriptor Shortcomings of the existing similarity function Proposed Merged Palette Histogram Similarity Measure

9. percentage color Dominant Color Descriptor Dominant Color Descriptor • Feature representation • The dominant colors • Percentage of area of the dominant color • Maximum of 8 colors

10. Dominant Color Descriptor • Feature extraction • GLA color quantization • Each color have at least Td distance away in a perceptually uniform CIELuv percentage CQ Original Image Color Quantized Image color Dominant Color Descriptor

11. Percentage p color Percentage q color Dominant Color Descriptor • Similarity measure • A modified Quadratic Histogram Distance Measure (QHDM)

12. Percentage p color Percentage q color Dominant Color Descriptor • Since each DCD may have different set of colors, QHDM is used to account for identical colors and similar colors.

13. Shortcomings of the QHDM similarity function • Limitations of QHDM • Distance upper bound is not fixed • Completely different image cannot be identified by its upper bound • The similarity coefficient does not well model color similarity • It does not balance between color distance and area of matching • The new Merged Palette Histogram Similarity Measure method • Can compare identical colors as well as similar colors • Use area of matching for similarity measure

14. Proposed Merged Palette Histogram Similarity Measure • MPHSM Process - 1 • Find the closest pair of colors using Euclidian distance in CIELuv color space

15. Proposed Merged Palette Histogram Similarity Measure • MPHSM process - 2 • If the distance smaller than a threshold Td, merge them to form a new common palette color • Step 1 – 2 iterates until the minimum distance larger than Td

16. Common Palette Merged Palette Histogram Dominant Color Descriptor Proposed Merged Palette Histogram Similarity Measure • MPHSM process - 3 • A new common palette is then generated • Form new descriptors based on the common palette

17. Proposed Merged Palette Histogram Similarity Measure • MPHSM process - 4 • Histogram intersection is used to measure the similarity • Count the non-overlapping area as the distance

18. Experimental results • MPHSM improves DCD for both datasets • While using Corel_1k dataset MPHSM outperforms QHDM significantly *ANMRR (smaller means better)

19. Experimental results • Visual results - Query #32 from MPEG-7 CCD Query image QHDM results, ANMRR=0.4 MPHSM result, ANMRR=0.0111

20. Experimental results • Visual results – Query #15 from Corel_1k Query image QHDM result, ANMRR=0.6464 MPHSM result, ANMRR=0.4819

21. Conclusions on Merged Palette Histogram Similarity Measure • MPHSM generates a common palette • Can match similar colors • Uses area of matching as the similarity • Boosts DCD in terms of ANMRR • Gives better visual results

22. Merged palette histogram for dominant color descriptor relevance feedback Feature weighting relevance feedbacktechnique and its limitations Proposed Merged Palette Histogram Relevance Feedback Experimental results

23. Feature weighting relevance feedbacktechnique and its limitations • Feature weighting relevance feedback technique • Assumes a fixed feature space (histograms) • Taking liner combinations on matching histogram bins. • Simple approach: Histogram averaging ( + ) / 2 =

24. H1 H’ H2 Feature weighting relevance feedbacktechnique and its limitations • But DCDs of images might have different set of colors, similar images might not have any exactly matched colors. • Two problems

25. H1 H’ H2 Limitation of feature weighting relevance feedbacktechnique • Problems • The number of colors in updated query may greatly exceed the limit of the number of colors defined by MPEG-7 as the number of selected images increase. • Similar colors are separated. By definition of DCD, similar colors should be grouped together.

26. Limitation of feature weighting relevance feedbacktechnique • The Merged Palette Histogram Relevance Feedback • The updated query contains common colors among selected images • Represent the selected images efficiently

27. Proposed Merged Palette Histogram for Relevance Feedback • Merged Palette Histogram Relevance Feedback (MPH-RF) process - initialize • Obtain all DCD from selected images

28. + + = Proposed Merged Palette Histogram for Relevance Feedback • Merged Palette Histogram Relevance Feedback (MPH-RF) process - 1 • Link all DCD together 8 colors 6 colors 20 colors 6 colors

29. Proposed Merged Palette Histogram for Relevance Feedback • Merged Palette Histogram Relevance Feedback (MPH-RF) process - 2 • Palette Merging • Find the closest pair of colorsbased on Euclidian distance in CIELuv • If minimum distance smaller than Td merge the color pair and sum up the percentages of merged colors • Iterate until minimum distance > Td 9 colors 20 colors

30. Proposed Merged Palette Histogram for Relevance Feedback • Merged Palette Histogram Relevance Feedback (MPH-RF) process - 3 • Approximation • Cut the least significant colors if number of colors >8 9 colors 8 colors

31. Proposed Merged Palette Histogram for Relevance Feedback • Merged Palette Histogram Relevance Feedback (MPH-RF) process - 4 • Re-normalization • Adjust the histogram sum into 1 • An updated query is generated Approximated MPH Updated QueryHistogram Sum =1

32. Experimental results • MPH-RF gives improvement on all combinations of similarity measures and datasets. • Combination of MPHSM and MPH-RF gives a significant improvement • Three iterations of relevance feedback give a significant result *ANMRR – smaller means better

33. Experimental results • Visual results – Query #50 from MPEG-7 CCD, MPHSM Query image First RF retrieval, 6 of 8 ground truths hit, NMRR=0.2782 Ground truth images Initial retrieval, 4 of 8 ground truths hit, NMRR=0.5 Second RF retrieval, 7 of 8 ground truths hit, NMRR=0.1541

34. Experimental results • Visual results – Query #13 from Corel_1k, MPHSM Query image First RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688 Ground truth images Initial retrieval, 7 of 11 ground truths hit, NMRR=0.3043 Second RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688

35. Conclusions on Merged Palette Histogram Relevance Feedback • MPH-RF generates a new DCD query using palette merging technique • Represents the selected relevant images naturally and effectively • MPH-RF boosts all situations of DCD searching

36. MIRROR – A CBIR system using MPEG-7 visual descriptors MPEG-7 Image Retrieval Refinement based On Relevance feedback Systems structure Demo

37. MIRROR – A CBIR system using MPEG-7 visual descriptors • System structure user initial input user feedback user feedback reference image relevant image(s ) ) Feature Extraction Similarity Relevance MPEG - - 7 7 Measure Feedback data Similarity Sorting Image Output Images DB

38. MIRROR – A CBIR system using MPEG-7 visual descriptors • Demo • Demo 1: Similarity Measure • Demo 2: Relevance Feedback • http://www.ee.cityu.edu.hk/~mirror/

39. Conclusions of this research work • By utilizing MPHSM and MPH-RF DCD, DCD becomes compact as well as accurate • Similarity measure • Merged Palette Histogram Similarity Measure • Relevance Feedback • Merged Palette Histogram Relevance Feedback • Proposed methods are implemented into a real system. • CBIR functions • Evaluation tools

40. Q & A