What s happening in iclef the iclef flickr challenge
This presentation is the property of its rightful owner.
Sponsored Links
1 / 28

What’s happening in iCLEF? (the iCLEF Flickr Challenge) PowerPoint PPT Presentation


  • 50 Views
  • Uploaded on
  • Presentation posted in: General

What’s happening in iCLEF? (the iCLEF Flickr Challenge). Julio Gonzalo (UNED), Paul Clough (U. Sheffield), Jussi Karlgren (SICS), Javier Artiles (UNED), Víctor Peinado (UNED). A piece of good news for CLEF!. People have started to hyperlink MT versions of web pages!. The original URL.

Download Presentation

What’s happening in iCLEF? (the iCLEF Flickr Challenge)

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


What s happening in iclef the iclef flickr challenge

What’s happening in iCLEF?(the iCLEF Flickr Challenge)

Julio Gonzalo (UNED),

Paul Clough (U. Sheffield),

Jussi Karlgren (SICS),

Javier Artiles (UNED),

Víctor Peinado (UNED)


A piece of good news for clef

A piece of good news for CLEF!

People have started to hyperlink MT versions of web pages!


What s happening in iclef the iclef flickr challenge

The original URL


The world is in danger

The world is in danger


What s happening in iclef the iclef flickr challenge

The world needs Multilingual Information Access research/solutions

…(that do care about users)


Retrospective goals for iclef 2006

Retrospective: goals for iCLEF 2006

  • Find realistic scenarios where multilingual search arises naturally

     multilingual web communities growing around

    (mostly) non-textual items (ebay, Flickr)

    Explore Web 2.0 “new text” challenges: folksonomies, blogs, wikis, etc.

  • Reduce the cost of entry for new participants.

     Provide a default multilingual search interface to Flickr.

  • Improve the cost-effectiveness of the traditional experiment design.

     Let participants work on their own experimental designs.


Flickr

Flickr


Flickr advantages

Flickr advantages

  • Naturally multilingual

  • Web 2.0 challenges (folksonomies)

  • Replicability (unlike general web image search facilities)


Iclef 2006

iCLEF 2006

Methodology

  • Participants must propose their own evaluation measures and experiment design.

Topics

  • Ad hoc: find as many photographs of (different) european parliaments as possible.

  • Creative: find five illustrations for this article about saffron in Italy.

  • Visual: What is the name of the beach where this crab is lying on?


Experiments

Experiments

  • UNED focused on user’s behaviour

    • How users deal with native/passive/unknown languages?

    • Do they actually use CLIR facilities when available?

  • SICS focused on user’s perceptions

    • Satisfaction (all tasks)

    • Completeness (creative,ad-hoc)

    • Quality (creative)

  • Sheffield also measured search effectiveness

    • How many facets were retrieved (creative, ad-hoc)

    • Was the image found? (visual)


Example result choice of target languages along time

Example result: choice of target languages along time.


Achievements

Achievements

  • Found a natural CLIR task

  • Flickr works for CLIR testing

  • New perspectives for experiment design.

  • Moved from bilingual to multilingual search problems.

 Stick to Flickr in 2007


Why participation was low

Why participation was low?

  • Kept on recruiting users and forcing them to search whatever we asked for.

    • Experiment design still too costly.

    • User populations still too small.

  • Cost of entry still too high

    • Building a Flickr interface / adapting the provided interface was not trivial.

    • Experiment design too open for newcomers.

  • Schedule collapses with standard tracks.

    • A problem for CLEF groups potentially interested.


How to improve

How to improve?

  • Kept on recruiting users and forcing them to search whatever we asked for.

    • Experiment design still too costly.

    • User populations still too small.

  • Cost of entry still too high

    • Building a Flickr interface / adapting the provided interface was not trivial.

    • Experiment design too open for newcomers.

  • Schedule collapses with standard tracks.

    • A problem for CLEF groups potentially interested.

  • Move to Flickr/Web users

  • Make search task a game

  • Provide experiment design

  • Participants ≈ Searchers

 Adjust iCLEF calendar


Iclef 2007 i target search task

iCLEF 2007 (i)Target search task

海 (sea, wave)

Find this image

  • Clear goal for the user

  • Clearly defined measures of success (appropriate for a “Hall of Fame” experience)

  • Invokes different searching strategies


Iclef 2007 ii evaluation aspects

iCLEF 2007 (ii) Evaluation aspects

  • User’s behaviour (Observational/Log Analysis)

    • Search strategies with native/passive/unknown languages

  • User’s perceptions

    • Usefulness

    • Difficulty

    • Clarity, etc.

  • Search effectiveness

    • Success rate and its correlation with user profiles, search strategies, etc.


Iclef 2007 iii cl search interface

iCLEF 2007 (iii) CL search interface

Single web interface for all experiments. Available for:

  • Groups willing to recruit users and conduct their own experiments.

  • Any web user wishing to enter the iCLEF Hall of Fame (by finding more images than anyone else!).

  • Groups willing to experience CL searching from the user’s perspective: who is the best CLEF team searching cross-language?

Full experiments

Log analysis


Iclef 2007 2008 schedule

iCLEF 2007/2008 schedule

  • End of October: Feedback from you! on

    • the multilingual search interface

    • experiment design

    • evaluation measures

  • Mid-December: Guidelines + interface goes public

    • Groups start their own experiments.

    • People can start competing in the search challenge.

    • Organization starts collecting logs from Flickr users

  • Mid-April:

    • Flickr/Web user Logs released to interested groups

    • Thematic session in Chorus Conference on Multimedia Search and Access (hosted by Yahoo!).

  • Paper submission: aligned with CLEF 2008


Start testing the interface now

Start testing the interface now!

http://bender.lsi.uned.es:8080/iCLEF2007

Feedback to [email protected]


  • Login