What s happening in iclef the iclef flickr challenge
Sponsored Links
This presentation is the property of its rightful owner.
1 / 28

What’s happening in iCLEF? (the iCLEF Flickr Challenge) PowerPoint PPT Presentation


  • 55 Views
  • Uploaded on
  • Presentation posted in: General

What’s happening in iCLEF? (the iCLEF Flickr Challenge). Julio Gonzalo (UNED), Paul Clough (U. Sheffield), Jussi Karlgren (SICS), Javier Artiles (UNED), Víctor Peinado (UNED). A piece of good news for CLEF!. People have started to hyperlink MT versions of web pages!. The original URL.

Download Presentation

What’s happening in iCLEF? (the iCLEF Flickr Challenge)

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


What’s happening in iCLEF?(the iCLEF Flickr Challenge)

Julio Gonzalo (UNED),

Paul Clough (U. Sheffield),

Jussi Karlgren (SICS),

Javier Artiles (UNED),

Víctor Peinado (UNED)


A piece of good news for CLEF!

People have started to hyperlink MT versions of web pages!


The original URL


The world is in danger


The world needs Multilingual Information Access research/solutions

…(that do care about users)


Retrospective: goals for iCLEF 2006

  • Find realistic scenarios where multilingual search arises naturally

     multilingual web communities growing around

    (mostly) non-textual items (ebay, Flickr)

    Explore Web 2.0 “new text” challenges: folksonomies, blogs, wikis, etc.

  • Reduce the cost of entry for new participants.

     Provide a default multilingual search interface to Flickr.

  • Improve the cost-effectiveness of the traditional experiment design.

     Let participants work on their own experimental designs.


Flickr


Flickr advantages

  • Naturally multilingual

  • Web 2.0 challenges (folksonomies)

  • Replicability (unlike general web image search facilities)


iCLEF 2006

Methodology

  • Participants must propose their own evaluation measures and experiment design.

Topics

  • Ad hoc: find as many photographs of (different) european parliaments as possible.

  • Creative: find five illustrations for this article about saffron in Italy.

  • Visual: What is the name of the beach where this crab is lying on?


Experiments

  • UNED focused on user’s behaviour

    • How users deal with native/passive/unknown languages?

    • Do they actually use CLIR facilities when available?

  • SICS focused on user’s perceptions

    • Satisfaction (all tasks)

    • Completeness (creative,ad-hoc)

    • Quality (creative)

  • Sheffield also measured search effectiveness

    • How many facets were retrieved (creative, ad-hoc)

    • Was the image found? (visual)


Example result: choice of target languages along time.


Achievements

  • Found a natural CLIR task

  • Flickr works for CLIR testing

  • New perspectives for experiment design.

  • Moved from bilingual to multilingual search problems.

 Stick to Flickr in 2007


Why participation was low?

  • Kept on recruiting users and forcing them to search whatever we asked for.

    • Experiment design still too costly.

    • User populations still too small.

  • Cost of entry still too high

    • Building a Flickr interface / adapting the provided interface was not trivial.

    • Experiment design too open for newcomers.

  • Schedule collapses with standard tracks.

    • A problem for CLEF groups potentially interested.


How to improve?

  • Kept on recruiting users and forcing them to search whatever we asked for.

    • Experiment design still too costly.

    • User populations still too small.

  • Cost of entry still too high

    • Building a Flickr interface / adapting the provided interface was not trivial.

    • Experiment design too open for newcomers.

  • Schedule collapses with standard tracks.

    • A problem for CLEF groups potentially interested.

  • Move to Flickr/Web users

  • Make search task a game

  • Provide experiment design

  • Participants ≈ Searchers

 Adjust iCLEF calendar


iCLEF 2007 (i)Target search task

海 (sea, wave)

Find this image

  • Clear goal for the user

  • Clearly defined measures of success (appropriate for a “Hall of Fame” experience)

  • Invokes different searching strategies


iCLEF 2007 (ii) Evaluation aspects

  • User’s behaviour (Observational/Log Analysis)

    • Search strategies with native/passive/unknown languages

  • User’s perceptions

    • Usefulness

    • Difficulty

    • Clarity, etc.

  • Search effectiveness

    • Success rate and its correlation with user profiles, search strategies, etc.


iCLEF 2007 (iii) CL search interface

Single web interface for all experiments. Available for:

  • Groups willing to recruit users and conduct their own experiments.

  • Any web user wishing to enter the iCLEF Hall of Fame (by finding more images than anyone else!).

  • Groups willing to experience CL searching from the user’s perspective: who is the best CLEF team searching cross-language?

Full experiments

Log analysis


iCLEF 2007/2008 schedule

  • End of October: Feedback from you! on

    • the multilingual search interface

    • experiment design

    • evaluation measures

  • Mid-December: Guidelines + interface goes public

    • Groups start their own experiments.

    • People can start competing in the search challenge.

    • Organization starts collecting logs from Flickr users

  • Mid-April:

    • Flickr/Web user Logs released to interested groups

    • Thematic session in Chorus Conference on Multimedia Search and Access (hosted by Yahoo!).

  • Paper submission: aligned with CLEF 2008


Start testing the interface now!

http://bender.lsi.uned.es:8080/iCLEF2007

Feedback to julio@lsi.uned.es


  • Login