russian information retrieval evaluation seminar romip http romip ru en n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Russian Information Retrieval Evaluation Seminar (ROMIP) http://romip.ru/en/ PowerPoint Presentation
Download Presentation
Russian Information Retrieval Evaluation Seminar (ROMIP) http://romip.ru/en/

Loading in 2 Seconds...

play fullscreen
1 / 12

Russian Information Retrieval Evaluation Seminar (ROMIP) http://romip.ru/en/ - PowerPoint PPT Presentation


  • 58 Views
  • Uploaded on

Russian Information Retrieval Evaluation Seminar (ROMIP) http://romip.ru/en/. Igor Nekrestyanov , Pavel Braslavski CLEF 2010. ROMIP at a glance. TREC-like Russian initiative Started 2002 Several text and image collections 10-15 participants per year (total 50 +)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Russian Information Retrieval Evaluation Seminar (ROMIP) http://romip.ru/en/' - coby


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
russian information retrieval evaluation seminar romip http romip ru en

Russian Information Retrieval Evaluation Seminar (ROMIP)http://romip.ru/en/

Igor Nekrestyanov, PavelBraslavski

CLEF 2010

romip at a glance
ROMIP at a glance
  • TREC-like Russian initiative
  • Started 2002
  • Several text and image collections
  • 10-15 participants per year (total 50+)
    • Academia and industry, students support
  • ~3 000 man-hours of evaluation (2009)
  • Remote participation + live meeting
  • Collections are freely available
  • Popular testbed for IR research in Russia
  • Related activities: summer school in IR

ROMIP

slide3
Why?
  • Russia specifics
    • Strong IR industry
    • Limited research in academia
    • Participation in global events considered complicated for Russian groups (language barrier, costs, etc.)
    • Russian language was not covered in international campaigns
  • Objectives
    • Consolidate IR community
    • Stimulate research in the area
    • Independent evaluation

ROMIP

evaluation methodology
Evaluation methodology
  • Similar to TREC approaches
  • What’s special?
    • Russian language collections
    • Some tasks are unique
      • E.g. news clustering, snippet generation, etc.
    • Mix of widely used and custom metrics
      • E.g. snippet informativeness/readability
    • Typically 2+ assessors (agreement 80-85%)
    • Domain experts for legal-related tracks
    • Rules and methodology are adjusted yearly

ROMIP

text documents tracks
Text documents tracks
  • Classic tracks run for years
    • Ad-hoc text retrieval
    • Text categorization (Web pages & sites, legal)
  • Experimental tracks every year
    • Snippet generation
    • QA and fact extraction
    • News clustering
    • Search by sample document

ROMIP

image collections
Image collections
  • Photo collection: 20 000 images from Flickr
  • Dups collection: 15 hrs video  37 800 frames

ROMIP

9

image tracks
Image tracks
  • Content based image retrieval (started 2008)
    • 750 tasks labeled
  • Near-duplicate detection (started 2008)
    • ~1500 clusters
  • Image annotation (started 2010)
    • ~ 1000 labeled images

ROMIP

10

romip timeline
ROMIP timeline

3000 man-hours eval.

news

ROMIP

image tracks

QAimage tagging

news

snippets

legal 2007 BY.Web KM.RU

legal

search classification

ROMIP

slide11

Thank you! Questions?

PavelBraslavski

pb@yandex-team.ru

Igor Nekrestyanov

romip@romip.ru

ROMIP

russir
RuSSIR
  • Put RuSSIRpic here
  • Annual event
  • 100+ participants
  • 4thRuSSIR: Voronezh 13-18 September
  • http://romip.ru/russir2010/

ROMIP