1 / 45

Supplementary Slides

Supplementary Slides. More experiment al results. MPHSM already push out many irrelevant images. Query image. QHDM result, 4 of 36 ground truth found ANMRR= 0.6464. MPHSM result, 9 of 36 ground truth found ANMRR= 0.4819. More about experimental results.

beau-moore
Download Presentation

Supplementary Slides

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Supplementary Slides

  2. More experimental results • MPHSM already push out many irrelevant images Query image QHDM result, 4 of 36 ground truth found ANMRR=0.6464 MPHSM result, 9 of 36 ground truth found ANMRR=0.4819

  3. More about experimental results • Still some irrelevant image found • No spatial information • Cannot identify background colors • Does not account for unmatched colors • Initial query might not be accurate Black Background Green Background

  4. More about experimental results • Can be improved by • Relevance Feedback • Makes relevant images to have higher ranks • Irrelevant normally can’t have higher similarity by RF • But relevant images does • Give more information about the interested objects • Inconsistent backgrounds can be averaged out

  5. More on experimental results • Irrelevant images got lower rank / out of top 20 after RF A E Query image B D First RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688 Ground truth images A B C D A G E F E B D Initial retrieval, 7 of 11 ground truths hit, NMRR=0.3043 Second RF retrieval, 9 of 11 ground truths hit, NMRR=0.1688

  6. More about experimental results • Still some irrelevant images found • Some colors are very common (Blue sky, black night, green grass, etc.) • Different semantics might have similar color distribution • No single feature can do perfect retrieval • Can be improved by several approaches • Choose suitable features • Combining features

  7. Suggestions on further developments • For DCD • Use unmatched colors • Challenge 1: Did the unmatched colors representing object of interest? Or just a obstacle? • Challenge 2: How to define the similarity function? • Separate foreground/background • Challenge 1: Can we identify it by only using DCD? Or in RF? • Challenge 2: Or we need to combine other shape/texture descriptors? • The DCD generation is not very accurate • GLA generates an optimal for quantizing the image, it might not be accurate dominant colors. • Can quantize up to 16 or more colors, and then approximate the least significant colors to obtain an 8 color DCD

  8. Suggestions on further developments • For general CBIR • No single descriptor gives perfect retrieval • Choosing suitable features • Combining features (color+shape, color+texture, etc.) • Automatically? Manually? How to set weights?

  9. Online Process User initial input Feature extraction Similarity measure Results output Similarity= 100% = 50% = 30% ... Similarity= 50% … Feature extraction Offline Process … Image DB Stored Features … … Visual description about a CBIR • System flow of a CBIR system

  10. Color based CBIR approaches • Three major approach of CBIR based on colors • Area of matching – Count the area with matched colors (CSD, SCD, DCD) • Color distance – Use color distance to adjust the similarity (DCD-QHDM) • Spatial distribution – Matches colors having similar layout (CLD)

  11. Optional parameters • Spatial coherence • obtained by a simple connected component analysis. A smooth surface gives a higher spatial coherence value. • Color variances • computed as variances of the pixel values within each cluster. But this parameter is for a dedicated similarity measure algorithm. So it is not commonly used.

  12. Spatial coherency adjustment • Similarity measure • MPEG-7 suggests to use a modified Quadratic Histogram Distance Measure (QHDM) to measure the dissimilarity between descriptors • Spatial coherency adjustment

  13. Results with Spatial Coherence • MPHSM improves DCD for both datasets to be more close to other non-compact descriptors • While using Corel_1k dataset MPHSM outperforms CLD slightly • MPHSM benefits from spatial coherency adjustment as well as QHDM

  14. DCD-QHDM upper bound problem • Analysis of problem 1 • The upper-bound of the distance measured varies by number of color in the descriptor • Maximum of positive part is not a constant • Maximum of negative part is zero • So, the maximum of QHDM is not fixed • This property makes DCD unable to identify completely different images by the values measured Positive part Negative part

  15. Upper bound problem - example • Problem 1 – The upper bound problem • Consider the following images with their DCD • I1, I2 are visually more similar than I1, I3 • For a similarity measure that matches human perception, we can expect the distance between F1, F2 should be smaller than that of between F1, F3 I I I 2 3 1 1/2 1/2 F F F 1/3 2 1 3

  16. Upper bound problem - example • But distance between F1,F3 is smaller while measuring their distance using QHDM • The extra blue color pull down the distance • D2(F1,F2)>D2(F1,F3) implies that I1 is more similar to I3 than I2 • This shows that QHDM does not meet human perception

  17. T d d a = 1.2 44% similar 16.67% similar 0% similar DCD-QHDM Similarity coefficient problem • The similarity coefficient use the color distance to fine tune the similarity • Difficult to define a quantitative similarity between colors, since the sensitivity of human eye depends on many conditions (e.g. light source of the room, spatial layout of the image, etc.)

  18. Similarity coefficient problem • It is easy to count 50% of area is similar.But it is difficult to count the colors are 50% similar. • This method is unable to consider the area of matching and the color distance together.

  19. I I I 2 1 4 1 1/2 1/2 F F F 2 1 4 Similarity coefficient problem - example • Problem 2 – The similarity coefficient a1i,2j problem • Consider the following images • I1, I2 are visually more similar than I1, I4 • For a similarity measure that matches human perception, we can expect the distance between F1, F2 should be smaller than that of between F1, F4

  20. Similarity coefficient problem - example • But distance between F1,F4 is smaller while measuring their distance using QHDM • One exactly matched color considered more important than a whole area of similar color • D2(F1,F2)>D2(F1,F4) implies that I1 is more similar to I4 than I2 • But in natural perception, images having similar color distribution is more likely to have similar semantics • This shows that QHDM does not meet human perception again

  21. Flow of MPHSM Initial DCDs Find a pair of colors with minimum distance d d<Td ? Merge colors having minimum distance Y N Common Palette Update each DCD based on the common palette Histogram Intersection

  22. Merge colors Merge colors Merge colors Merge colors Palette Merging process, visually • Example • Two images with DCD, palette merging stage Find the closest pair Common Palette Remaining colors If a remaining color is similar to any colors in the common palette. It will not included in common palette Dominant Color Descriptor

  23. About slide 23 • Relationship between CBIR and Relevance Feedback (RF) • The key component is query update

  24. MPH-RF flow Load add DCDs A Append all DCD Cut least significant colors Find closes pair of colors Merge colors and percentages Adjust histogram sum into 1 Minimum distance < Td ? Y Updated query N A

  25. RF of other MPEG-7 visual descriptors • Relevance feedback for MPEG-7 descriptors • Apart from the MPH-RF for DCD, we directly apply feature weighting technique on several MPEG-7 visual descriptors • RF on CLD: • RF on CSD:

  26. RF of other MPEG-7 visual descriptors • RF on SCD:

  27. MIRROR – A CBIR system using MPEG-7 visual descriptors • A set of visual descriptors • Relevance feedback functions is added • Evaluation tools • MIRROR is also a development platform of MPEG-7 visual descriptors

  28. Performance of color descriptors • Performance of color descriptors • Evaluation tools • Unmodified MPEG-7 reference software XM • MPEG-7 Common Color Dataset (MPEG-7 CCD) with 5466 images and 50 sample queries • Corel 1000 images dataset with 20 sample queries • ANMRR performance metric (smaller means better)

  29. Performance of color descriptors • Investigation of performances • Color structure descriptor performs best among color descriptors due to its large descriptor size • Dominant color descriptor performs worst, even worse than a more compact color layout descriptor • “Area of matching” is still the most efficient approach for color based CBIR • New methods will be proposed in this research to boost DCD

  30. Complete results • MPHSM improves DCD for both datasets to be more close to other non-compact descriptors • While using Corel_1k dataset MPHSM outperforms CLD slightly • MPHSM benefits from spatial coherency adjustment as well as QHDM

  31. Complete results • MPR-RF gives significant improvement on all combinations of similarity measures and datasets. • By using MPH-RF DCD can perform as good as another compact descriptor CLD, and very close to a lesser compact descriptor SCD. • Three iterations of relevance feedback give a significant result

  32. Complete results • The MPH-RF improvement on DCD is more significant than feature weighting for other color descriptors • Color structure descriptor gives impressive results among all color descriptor, and its only drawback is the descriptor size is too large.

More Related