What s happening in iclef the iclef flickr challenge
Download
1 / 28

What’s happening in iCLEF? (the iCLEF Flickr Challenge) - PowerPoint PPT Presentation


  • 78 Views
  • Uploaded on

What’s happening in iCLEF? (the iCLEF Flickr Challenge). Julio Gonzalo (UNED), Paul Clough (U. Sheffield), Jussi Karlgren (SICS), Javier Artiles (UNED), Víctor Peinado (UNED). A piece of good news for CLEF!. People have started to hyperlink MT versions of web pages!. The original URL.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' What’s happening in iCLEF? (the iCLEF Flickr Challenge)' - kadeem-emerson


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
What s happening in iclef the iclef flickr challenge

What’s happening in iCLEF?(the iCLEF Flickr Challenge)

Julio Gonzalo (UNED),

Paul Clough (U. Sheffield),

Jussi Karlgren (SICS),

Javier Artiles (UNED),

Víctor Peinado (UNED)


A piece of good news for clef
A piece of good news for CLEF!

People have started to hyperlink MT versions of web pages!




The world needs Multilingual Information Access research/solutions

…(that do care about users)


Retrospective goals for iclef 2006
Retrospective: goals for iCLEF 2006 research/solutions

  • Find realistic scenarios where multilingual search arises naturally

     multilingual web communities growing around

    (mostly) non-textual items (ebay, Flickr)

    Explore Web 2.0 “new text” challenges: folksonomies, blogs, wikis, etc.

  • Reduce the cost of entry for new participants.

     Provide a default multilingual search interface to Flickr.

  • Improve the cost-effectiveness of the traditional experiment design.

     Let participants work on their own experimental designs.


Flickr
Flickr research/solutions


Flickr advantages
Flickr advantages research/solutions

  • Naturally multilingual

  • Web 2.0 challenges (folksonomies)

  • Replicability (unlike general web image search facilities)


Iclef 2006
iCLEF 2006 research/solutions

Methodology

  • Participants must propose their own evaluation measures and experiment design.

Topics

  • Ad hoc: find as many photographs of (different) european parliaments as possible.

  • Creative: find five illustrations for this article about saffron in Italy.

  • Visual: What is the name of the beach where this crab is lying on?


Experiments
Experiments research/solutions

  • UNED focused on user’s behaviour

    • How users deal with native/passive/unknown languages?

    • Do they actually use CLIR facilities when available?

  • SICS focused on user’s perceptions

    • Satisfaction (all tasks)

    • Completeness (creative,ad-hoc)

    • Quality (creative)

  • Sheffield also measured search effectiveness

    • How many facets were retrieved (creative, ad-hoc)

    • Was the image found? (visual)


Example result choice of target languages along time
Example result: research/solutionschoice of target languages along time.


Achievements
Achievements research/solutions

  • Found a natural CLIR task

  • Flickr works for CLIR testing

  • New perspectives for experiment design.

  • Moved from bilingual to multilingual search problems.

 Stick to Flickr in 2007


Why participation was low
Why participation was low? research/solutions

  • Kept on recruiting users and forcing them to search whatever we asked for.

    • Experiment design still too costly.

    • User populations still too small.

  • Cost of entry still too high

    • Building a Flickr interface / adapting the provided interface was not trivial.

    • Experiment design too open for newcomers.

  • Schedule collapses with standard tracks.

    • A problem for CLEF groups potentially interested.


How to improve
How to improve? research/solutions

  • Kept on recruiting users and forcing them to search whatever we asked for.

    • Experiment design still too costly.

    • User populations still too small.

  • Cost of entry still too high

    • Building a Flickr interface / adapting the provided interface was not trivial.

    • Experiment design too open for newcomers.

  • Schedule collapses with standard tracks.

    • A problem for CLEF groups potentially interested.

  • Move to Flickr/Web users

  • Make search task a game

  • Provide experiment design

  • Participants ≈ Searchers

 Adjust iCLEF calendar


Iclef 2007 i target search task
iCLEF 2007 (i) Target search task research/solutions

海 (sea, wave)

Find this image

  • Clear goal for the user

  • Clearly defined measures of success (appropriate for a “Hall of Fame” experience)

  • Invokes different searching strategies


Iclef 2007 ii evaluation aspects
iCLEF 2007 (ii) Evaluation aspects research/solutions

  • User’s behaviour (Observational/Log Analysis)

    • Search strategies with native/passive/unknown languages

  • User’s perceptions

    • Usefulness

    • Difficulty

    • Clarity, etc.

  • Search effectiveness

    • Success rate and its correlation with user profiles, search strategies, etc.


Iclef 2007 iii cl search interface
iCLEF 2007 (iii) CL search interface research/solutions

Single web interface for all experiments. Available for:

  • Groups willing to recruit users and conduct their own experiments.

  • Any web user wishing to enter the iCLEF Hall of Fame (by finding more images than anyone else!).

  • Groups willing to experience CL searching from the user’s perspective: who is the best CLEF team searching cross-language?

Full experiments

Log analysis


Iclef 2007 2008 schedule
iCLEF 2007/2008 schedule research/solutions

  • End of October: Feedback from you! on

    • the multilingual search interface

    • experiment design

    • evaluation measures

  • Mid-December: Guidelines + interface goes public

    • Groups start their own experiments.

    • People can start competing in the search challenge.

    • Organization starts collecting logs from Flickr users

  • Mid-April:

    • Flickr/Web user Logs released to interested groups

    • Thematic session in Chorus Conference on Multimedia Search and Access (hosted by Yahoo!).

  • Paper submission: aligned with CLEF 2008


Start testing the interface now
Start testing the interface now! research/solutions

http://bender.lsi.uned.es:8080/iCLEF2007

Feedback to [email protected]


ad