what s happening in iclef the iclef flickr challenge
Download
Skip this Video
Download Presentation
What’s happening in iCLEF? (the iCLEF Flickr Challenge)

Loading in 2 Seconds...

play fullscreen
1 / 28

What’s happening in iCLEF? (the iCLEF Flickr Challenge) - PowerPoint PPT Presentation


  • 79 Views
  • Uploaded on

What’s happening in iCLEF? (the iCLEF Flickr Challenge). Julio Gonzalo (UNED), Paul Clough (U. Sheffield), Jussi Karlgren (SICS), Javier Artiles (UNED), Víctor Peinado (UNED). A piece of good news for CLEF!. People have started to hyperlink MT versions of web pages!. The original URL.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' What’s happening in iCLEF? (the iCLEF Flickr Challenge)' - kadeem-emerson


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
what s happening in iclef the iclef flickr challenge

What’s happening in iCLEF?(the iCLEF Flickr Challenge)

Julio Gonzalo (UNED),

Paul Clough (U. Sheffield),

Jussi Karlgren (SICS),

Javier Artiles (UNED),

Víctor Peinado (UNED)

a piece of good news for clef
A piece of good news for CLEF!

People have started to hyperlink MT versions of web pages!

slide5
The world needs Multilingual Information Access research/solutions

…(that do care about users)

retrospective goals for iclef 2006
Retrospective: goals for iCLEF 2006
  • Find realistic scenarios where multilingual search arises naturally

 multilingual web communities growing around

(mostly) non-textual items (ebay, Flickr)

Explore Web 2.0 “new text” challenges: folksonomies, blogs, wikis, etc.

  • Reduce the cost of entry for new participants.

 Provide a default multilingual search interface to Flickr.

  • Improve the cost-effectiveness of the traditional experiment design.

 Let participants work on their own experimental designs.

flickr advantages
Flickr advantages
  • Naturally multilingual
  • Web 2.0 challenges (folksonomies)
  • Replicability (unlike general web image search facilities)
iclef 2006
iCLEF 2006

Methodology

  • Participants must propose their own evaluation measures and experiment design.

Topics

  • Ad hoc: find as many photographs of (different) european parliaments as possible.
  • Creative: find five illustrations for this article about saffron in Italy.
  • Visual: What is the name of the beach where this crab is lying on?
experiments
Experiments
  • UNED focused on user’s behaviour
    • How users deal with native/passive/unknown languages?
    • Do they actually use CLIR facilities when available?
  • SICS focused on user’s perceptions
    • Satisfaction (all tasks)
    • Completeness (creative,ad-hoc)
    • Quality (creative)
  • Sheffield also measured search effectiveness
    • How many facets were retrieved (creative, ad-hoc)
    • Was the image found? (visual)
achievements
Achievements
  • Found a natural CLIR task
  • Flickr works for CLIR testing
  • New perspectives for experiment design.
  • Moved from bilingual to multilingual search problems.

 Stick to Flickr in 2007

why participation was low
Why participation was low?
  • Kept on recruiting users and forcing them to search whatever we asked for.
    • Experiment design still too costly.
    • User populations still too small.
  • Cost of entry still too high
    • Building a Flickr interface / adapting the provided interface was not trivial.
    • Experiment design too open for newcomers.
  • Schedule collapses with standard tracks.
    • A problem for CLEF groups potentially interested.
how to improve
How to improve?
  • Kept on recruiting users and forcing them to search whatever we asked for.
    • Experiment design still too costly.
    • User populations still too small.
  • Cost of entry still too high
    • Building a Flickr interface / adapting the provided interface was not trivial.
    • Experiment design too open for newcomers.
  • Schedule collapses with standard tracks.
    • A problem for CLEF groups potentially interested.
  • Move to Flickr/Web users
  • Make search task a game
  • Provide experiment design
  • Participants ≈ Searchers

 Adjust iCLEF calendar

iclef 2007 i target search task
iCLEF 2007 (i) Target search task

海 (sea, wave)

Find this image

  • Clear goal for the user
  • Clearly defined measures of success (appropriate for a “Hall of Fame” experience)
  • Invokes different searching strategies
iclef 2007 ii evaluation aspects
iCLEF 2007 (ii) Evaluation aspects
  • User’s behaviour (Observational/Log Analysis)
    • Search strategies with native/passive/unknown languages
  • User’s perceptions
    • Usefulness
    • Difficulty
    • Clarity, etc.
  • Search effectiveness
    • Success rate and its correlation with user profiles, search strategies, etc.
iclef 2007 iii cl search interface
iCLEF 2007 (iii) CL search interface

Single web interface for all experiments. Available for:

  • Groups willing to recruit users and conduct their own experiments.
  • Any web user wishing to enter the iCLEF Hall of Fame (by finding more images than anyone else!).
  • Groups willing to experience CL searching from the user’s perspective: who is the best CLEF team searching cross-language?

Full experiments

Log analysis

iclef 2007 2008 schedule
iCLEF 2007/2008 schedule
  • End of October: Feedback from you! on
    • the multilingual search interface
    • experiment design
    • evaluation measures
  • Mid-December: Guidelines + interface goes public
    • Groups start their own experiments.
    • People can start competing in the search challenge.
    • Organization starts collecting logs from Flickr users
  • Mid-April:
    • Flickr/Web user Logs released to interested groups
    • Thematic session in Chorus Conference on Multimedia Search and Access (hosted by Yahoo!).
  • Paper submission: aligned with CLEF 2008
start testing the interface now
Start testing the interface now!

http://bender.lsi.uned.es:8080/iCLEF2007

Feedback to [email protected]

ad