Inaoe at imageclef2007 towards annotation based image retrieval
Download
1 / 17

INAOE at ImageCLEF2007 - PowerPoint PPT Presentation


  • 951 Views
  • Updated On :

INAOE at ImageCLEF2007 Towards Annotation based Image Retrieval. H. Jair Escalante, Carlos Hernández, Aurelio López, Heidi Marín, Manuel Montes , Eduardo Morales, Enrique Sucar, Luis Villaseñor Language Technologies Laboratory National Institute of Astrophysics, Optics and Electronics

Related searches for INAOE at ImageCLEF2007

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'INAOE at ImageCLEF2007' - LeeJohn


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Inaoe at imageclef2007 towards annotation based image retrieval l.jpg

INAOE at ImageCLEF2007Towards Annotation based Image Retrieval

H. Jair Escalante, Carlos Hernández, Aurelio López, Heidi Marín,Manuel Montes, Eduardo Morales, Enrique Sucar, Luis Villaseñor

Language Technologies Laboratory

National Institute of Astrophysics, Optics and Electronics

Tonantzintla, Mexico

[email protected]

http://ccc.inaoep.mx/~mmontesg


Overview of the talk l.jpg
Overview of the talk

  • Our first participation at ImageCLEF; the goal was to build the basic infrastructure

    • Some textual and mixed strategies for image retrieval

  • However we could do something more…

    • A Web based query expansion method, and

    • An annotation based image retrieval approach


Textual and mixed strategies l.jpg

Terms for annotations

Example images

CBIR

Query

QueryExpansion

TBIR

Topic statement

RelevantImages

Textual and mixed strategies

  • VSM IR System for textual retrieval (baseline)

  • Late fusion of independent retrievers (LF)

  • Intermedia feedback (IMFB)

Topic statement

TBIR

Fusion

Query

RelevantImages

CBIR

Example images


Some new things l.jpg
Some new things…

  • Web-based query expansion:

    • Original statement + top-k snippets (NQE)

    • Original statement + top-l more repeated words from the top-k snippets (WQE)

  • Annotation based expansion (ABE)

    • Use automatic image annotation methods for obtaining text from images, then…

    • Expand documents and/or queries with automatic annotations, finally…

    • Apply some strategy for textual image retrieval


Basis of our idea l.jpg
Basis of our idea

  • Region-level annotations are generally complementary to manual (image-level) annotations

sky

palm

palm, sky, sand,

grass, sea, clouds

clouds

Flamingo Beach Original name in Portuguese: “Praia do Flamengo”; Flamingo Beach is considered as one of the most beautiful beaches of Brazil;

sea

sand

sand

grass

Flamingo Beach Original name in Portuguese: “Praia do Flamengo”; Flamingo Beach is considered as one of the most beautiful beaches of Brazil;

Flamingo Beach Original name in Portuguese: “Praia do Flamengo”; Flamingo Beach is considered as one of the most beautiful beaches of Brazil;


Automatic image annotation l.jpg

x1

x1

x1

x2

x2

x2

……..

……..

……..

xn

xn

xn

Automatic image annotation

  • Assign labels (words) to regions within segmented images

Automatic image

Annotation

method

. . .

Sky

Elephant

Grass 0.6

Sky 0.2

Tree 0.1

Ground 0.1

Annotation

improvement

Rock 0.5

Church 0.2

Elephant 0.2

Entrance 0.1

Grass 0.5

Tree 0.3

Ground 0.1

Jet 0.1

Grass


Improving the automatic annotation l.jpg

R1R2R3R4

c1 grasspeopletreechurch

c2 grass tree treechurch

c.. ….….….……

ci rockpeopletree church

c.. ….….….……

cj treepeopletree church

c256 buildingjetjet elephant

Idea: select the best label’s configuration, taking into account:

1. The prior probabilities of each label, and

2. The semantic cohesion of the entire configuration

Improving the automatic annotation

Grass 0.6

Tree 0.2

rock 0.1

building 0.1

People 0.4

Tree 0.3

Mountain 0.2

Jet 0.1

Tree 0.5

Grass 0.3

Sky 0.1

Jet 0.1

Grass, Tree, Rock, Building, People, Mountain, Jet, Sky, Church, Elephant

Church 0.3

Grass 0.3

Sky 0.2

Elephant 0.2



Some problems with the labels l.jpg
Some problems with the labels

  • 2000 training annotated-regions (2%)

  • 98000 regions to annotate (98%)

  • Imbalanced training set

  • Limited vocabulary


Annotation based query expansion l.jpg

accommodation with swimming pool

sky water tree

sand boats sky tree people

water sand sky tree buildings

accommodation with swimming pool +

sand boats sky tree people water buildings +

three given images

Annotation based query expansion


Annotation based document expansion l.jpg
Annotation based document expansion

The surroundings of the

Valle Francés Torres del Paine National Park, Chile March 2002

furniture grasspeople clouds

The volcano Tungurahua

Baños, EcuadorMarch 2002

sand clouds sky mountain


Experimental results l.jpg
Experimental results

Top ranked runs for each configuration considered.


Visual english run l.jpg

Automatic

annotations

AutomaticAnnotation

Exampleimages

TBIR

RelevantImages

CBIR

Terms for

manual annotations

Visual-English run

  • No textual query was used, but at the end the recovery was done based on textual data.

  • It combines intermedia feedback and our annotation based expansion technique.



Initial conclusions l.jpg
Initial conclusions

  • Intermedia feedback is an effective way for mixing visual and textual information

  • Methods based on web-query expansion showed better performance

  • Anotation based expansion is a promising way for expanding text using image’s visual content

  • Annotations can be useful for image retrieval, though several issues should be addressed


Our current work l.jpg
Our current work

  • Work on the improvement of automatic image annotation methods

  • Investigate different (better) ways for measuring the semantic cohesion between labels and manual annotations

  • Use such semantic cohesion estimates for improving image retrieval from annotated collections


Thanks for your attention l.jpg
Thanks for your attention

Language Technologies Laboratory

National Institute of Astrophysics, Optics and Electronics

Tonantzintla, México

Manuel Montes y Gómez

[email protected]

http://ccc.inaoep.mx/~mmontesg