the kdd 2008 review process research track bing liu sunita sarawagi n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
The KDD 2008 review process (Research track) Bing Liu & Sunita Sarawagi PowerPoint Presentation
Download Presentation
The KDD 2008 review process (Research track) Bing Liu & Sunita Sarawagi

Loading in 2 Seconds...

play fullscreen
1 / 9

The KDD 2008 review process (Research track) Bing Liu & Sunita Sarawagi - PowerPoint PPT Presentation


  • 78 Views
  • Uploaded on

The KDD 2008 review process (Research track) Bing Liu & Sunita Sarawagi. Bid-based paper assignment. Reviewers bid on papers Scale between 3=Eager and 0=not-willing Initial Assignment Globally maximize total bids subject to load, count constraints Easily solved using any LP-package

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'The KDD 2008 review process (Research track) Bing Liu & Sunita Sarawagi' - candy


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
bid based paper assignment
Bid-based paper assignment
  • Reviewers bid on papers
    • Scale between 3=Eager and 0=not-willing
  • Initial Assignment
    • Globally maximize total bids subject to load, count constraints
    • Easily solved using any LP-package
  • Manual inspection and readjustments
    • Effort varies from chair to chair

KDD-08 Opening August 24, 2008 Bing Liu & Sunita Sarawagi

problems of bid based assignment
Problems of bid-based assignment
  • Two surprising dynamics
    • Unfair on papers on hot topics
      • Top fewpapers had bids from 25%of the PC.
      •  Random PC member reads it.
    • Unfair on reviewers who bid low
      • Old cynics (no eager bids) versus young interested (80 eager bids)
      • Random paper goes to low bidders

KDD-08 Opening August 24, 2008 Bing Liu & Sunita Sarawagi

manual readjustments not easy
Manual readjustments not easy
  • Scale: 500 papers, 190 reviewers,
    • Difficult for chairs to be familiar with the expertise of each reviewer
  • Tightly constrained system: any change spirals off a cascade of other changes.

KDD-08 Opening August 24, 2008 Bing Liu & Sunita Sarawagi

modeling reviewer to paper affinity
Modeling reviewer to paper affinity
  • Reviewer profile: abstracts of past publications
    • Challenge: crawling for abstracts
    • DBLP with pointers to electronic edition + some manual gathering/cleaning
      • (Thanks to IITB undergrads: Ankit Gupta, AnkurGoel)
  • Paper-reviewer affinity
    • TF-IDF similarity between paper abstract and reviewer profile
    • Okapi, BM25 etc tuned for short queries and long documents

KDD-08 Opening August 24, 2008 Bing Liu & Sunita Sarawagi

the assignment
The assignment

Maximize weighted sum of bid and affinity subject to load,countconstraints

KDD-08 Opening August 24, 2008 Bing Liu & Sunita Sarawagi

manual readjustments still needed
Manual readjustments still needed
  • Chairs go over assignments and give input as
    • Short list of reviewers for a paper
  • Re-invoke LP with additional constraints
    • Chairs spared of handling cascaded changes
    • But, need a stable LP solver to minimize changes
    • Current algorithm (LpSolve) seems stable
  • We did three rounds, working10 days non-stop!
  • Coding easy: One week with LpSolve+Lucene

KDD-08 Opening August 24, 2008 Bing Liu & Sunita Sarawagi

improvements
Improvements
  • Better modeling of reviewer expertise
    • Time decaying topic models?
  • Better affinity match
    • Citation distance?
  • Human intervention is unavoidable.
    • Good interactive UI tools for paper assignment

KDD-08 Opening August 24, 2008 Bing Liu & Sunita Sarawagi

other issues
Other issues
  • Author feedback
  • Conditional accept
  • Early notification of sure rejects
  • Vice chairs select PC and assign papers
    • KDD is homogeneous
    • Topics keep shifting
    • Load balancing across tracks difficult

KDD-08 Opening August 24, 2008 Bing Liu & Sunita Sarawagi