inaoe at imageclef2007 towards annotation based image retrieval
Download
Skip this Video
Download Presentation
INAOE at ImageCLEF2007 Towards Annotation based Image Retrieval

Loading in 2 Seconds...

play fullscreen
1 / 17

INAOE at ImageCLEF2007 - PowerPoint PPT Presentation


  • 951 Views
  • Uploaded on

INAOE at ImageCLEF2007 Towards Annotation based Image Retrieval. H. Jair Escalante, Carlos Hernández, Aurelio López, Heidi Marín, Manuel Montes , Eduardo Morales, Enrique Sucar, Luis Villaseñor Language Technologies Laboratory National Institute of Astrophysics, Optics and Electronics

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'INAOE at ImageCLEF2007' - LeeJohn


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
inaoe at imageclef2007 towards annotation based image retrieval

INAOE at ImageCLEF2007Towards Annotation based Image Retrieval

H. Jair Escalante, Carlos Hernández, Aurelio López, Heidi Marín,Manuel Montes, Eduardo Morales, Enrique Sucar, Luis Villaseñor

Language Technologies Laboratory

National Institute of Astrophysics, Optics and Electronics

Tonantzintla, Mexico

[email protected]

http://ccc.inaoep.mx/~mmontesg

overview of the talk
Overview of the talk
  • Our first participation at ImageCLEF; the goal was to build the basic infrastructure
    • Some textual and mixed strategies for image retrieval
  • However we could do something more…
    • A Web based query expansion method, and
    • An annotation based image retrieval approach
textual and mixed strategies

Terms for annotations

Example images

CBIR

Query

QueryExpansion

TBIR

Topic statement

RelevantImages

Textual and mixed strategies
  • VSM IR System for textual retrieval (baseline)
  • Late fusion of independent retrievers (LF)
  • Intermedia feedback (IMFB)

Topic statement

TBIR

Fusion

Query

RelevantImages

CBIR

Example images

some new things
Some new things…
  • Web-based query expansion:
    • Original statement + top-k snippets (NQE)
    • Original statement + top-l more repeated words from the top-k snippets (WQE)
  • Annotation based expansion (ABE)
    • Use automatic image annotation methods for obtaining text from images, then…
    • Expand documents and/or queries with automatic annotations, finally…
    • Apply some strategy for textual image retrieval
basis of our idea
Basis of our idea
  • Region-level annotations are generally complementary to manual (image-level) annotations

sky

palm

palm, sky, sand,

grass, sea, clouds

clouds

Flamingo Beach Original name in Portuguese: “Praia do Flamengo”; Flamingo Beach is considered as one of the most beautiful beaches of Brazil;

sea

sand

sand

grass

Flamingo Beach Original name in Portuguese: “Praia do Flamengo”; Flamingo Beach is considered as one of the most beautiful beaches of Brazil;

Flamingo Beach Original name in Portuguese: “Praia do Flamengo”; Flamingo Beach is considered as one of the most beautiful beaches of Brazil;

automatic image annotation

x1

x1

x1

x2

x2

x2

……..

……..

……..

xn

xn

xn

Automatic image annotation
  • Assign labels (words) to regions within segmented images

Automatic image

Annotation

method

. . .

Sky

Elephant

Grass 0.6

Sky 0.2

Tree 0.1

Ground 0.1

Annotation

improvement

Rock 0.5

Church 0.2

Elephant 0.2

Entrance 0.1

Grass 0.5

Tree 0.3

Ground 0.1

Jet 0.1

Grass

improving the automatic annotation

R1R2R3R4

c1 grasspeopletreechurch

c2 grass tree treechurch

c.. ….….….……

ci rockpeopletree church

c.. ….….….……

cj treepeopletree church

c256 buildingjetjet elephant

Idea: select the best label’s configuration, taking into account:

1. The prior probabilities of each label, and

2. The semantic cohesion of the entire configuration

Improving the automatic annotation

Grass 0.6

Tree 0.2

rock 0.1

building 0.1

People 0.4

Tree 0.3

Mountain 0.2

Jet 0.1

Tree 0.5

Grass 0.3

Sky 0.1

Jet 0.1

Grass, Tree, Rock, Building, People, Mountain, Jet, Sky, Church, Elephant

Church 0.3

Grass 0.3

Sky 0.2

Elephant 0.2

some problems with the labels
Some problems with the labels
  • 2000 training annotated-regions (2%)
  • 98000 regions to annotate (98%)
  • Imbalanced training set
  • Limited vocabulary
annotation based query expansion

accommodation with swimming pool

sky water tree

sand boats sky tree people

water sand sky tree buildings

accommodation with swimming pool +

sand boats sky tree people water buildings +

three given images

Annotation based query expansion
annotation based document expansion
Annotation based document expansion

The surroundings of the

Valle Francés Torres del Paine National Park, Chile March 2002

furniture grasspeople clouds

The volcano Tungurahua

Baños, EcuadorMarch 2002

sand clouds sky mountain

experimental results
Experimental results

Top ranked runs for each configuration considered.

visual english run

Automatic

annotations

AutomaticAnnotation

Exampleimages

TBIR

RelevantImages

CBIR

Terms for

manual annotations

Visual-English run
  • No textual query was used, but at the end the recovery was done based on textual data.
  • It combines intermedia feedback and our annotation based expansion technique.
initial conclusions
Initial conclusions
  • Intermedia feedback is an effective way for mixing visual and textual information
  • Methods based on web-query expansion showed better performance
  • Anotation based expansion is a promising way for expanding text using image’s visual content
  • Annotations can be useful for image retrieval, though several issues should be addressed
our current work
Our current work
  • Work on the improvement of automatic image annotation methods
  • Investigate different (better) ways for measuring the semantic cohesion between labels and manual annotations
  • Use such semantic cohesion estimates for improving image retrieval from annotated collections
thanks for your attention
Thanks for your attention

Language Technologies Laboratory

National Institute of Astrophysics, Optics and Electronics

Tonantzintla, México

Manuel Montes y Gómez

[email protected]

http://ccc.inaoep.mx/~mmontesg