Learning conference reviewer assignments
Download
1 / 25

Learning Conference Reviewer Assignments - PowerPoint PPT Presentation


  • 57 Views
  • Uploaded on

Learning Conference Reviewer Assignments. Adith Swaminathan Guide : Prof. Soumen Chakrabarti. Department of Computer Science and Engineering, Indian Institute of Technology, Bombay. Future Work (from BTP1). Given WWW2010’s assignments, learn Affinity_Param, Topic_Param and Irritation

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Learning Conference Reviewer Assignments' - evan


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Learning conference reviewer assignments

Learning Conference Reviewer Assignments

Adith Swaminathan

Guide :

Prof. Soumen Chakrabarti

Department of Computer Science and

Engineering,

Indian Institute of Technology, Bombay


Future work from btp1
Future Work (from BTP1)

  • Given WWW2010’s assignments, learn Affinity_Param, Topic_Param and Irritation

  • Citations as edge features

  • Load-Constrained Partial Assignments

  • Better estimation of Assignment Quality


Background
Background

  • Conference Reviewer-Paper Assignment as a Many-Many-matching [1]

  • Minimum Cost Network Flow (MCF)


Conference reviewer assignment
Conference Reviewer Assignment

  • Set of Reviewers, R, max #papers = L_i

  • Set of Papers, P, min #reviews = K

  • Assumption : Only require #reviews, not quality

  • Suppose we have cost function A_ij(y) for <R_i, P_j>


Ilp assumption mcf
ILP-> Assumption -> MCF


Two problems
Two problems

  • Integer Linear Programs are NP-Hard!

    • Relax?

    • More assumptions?

  • How to determine A_ij?

    • M * N ~ 10000

    • Multimodal clues


Ilp assumption mcf1
ILP ->Assumption-> MCF

  • Enforce structure on A_ij

    • Better model multimodality

    • Fewer parameters to fix

  • “Learn” A_ij using Structured Learning Techniques

  • A_ij = wTΦ(R_i, P_j, y_ij)


Ramifications of structured costs
Ramifications of Structured Costs

  • Costs decompose over <R_i, P_j> pairs

    • Decomposable Preference Auction

    • Polynomial Algorithms for DPAs [2]

  • Restricted notion of optimality

    • Per-reviewer/Per-paper constraint could be combinatorial

    • Stability?



Minimum cost network flow
Minimum Cost Network Flow

  • Directed graph G=(V,E), capacities u(E)>= 0, costs c(E)

  • Nodes have numbers b(V) : Sum(b(V)) = 0

  • Task : Find a function f: E->R+ which satisfies the b-flow at minimum cost

  • Successive Shortest Path Algorithm


Node features and edge features
Node features and Edge features

Affinity

Cites

Bid

Topic Overlap


The loss function
The Loss Function

  • L_ij = w_1 * exp(-Affinity_ij) + w_2 * [[1 – Topic_Overlap_ij]] + w_3 * Bid_Cost

  • Bid_Cost = Potential(R_i, P_j, y_ij)

  • Irritation (I) and Disappointment (D) needs to be set


Assignment quality measures
Assignment Quality Measures

  • Number of Bids Violated?

    • Not a reliable measure.

  • +ve Bids Violated

  • –ve Bids Violated

  • Assignments satisfying Topic Match

  • Confidence?


Confidence quality
Confidence == Quality?

  • Very sparse

    • Fewer than 5% observed

    • Extrapolated Confidence?

  • Reliable

    • Bids as a precursor of Confidence [3]

    • Confidence-Augmented Loss?


Learning w s
Learning w’s

  • Transductive Ordinal Regression

    • Assume : Assignments are independent (Naïve)

    • Heuristic : Augment observed dataset

    • Extrapolate observed Confidence [4]

    • Learn w over extrapolated dataset

  • Support Vector Machine for Structured Outputs

    • Cast as soft-margin SVM formulation [5]

    • Upper-bound objective with a convex fn (Optimality?)

    • Minimize, using Cutting Plane (Approximate)



Svm struct 7
SVM Struct. [7]

Loss Augmented Inference ~ Most Violated Constraint

Loss is decomposable -> Modified MCF




Bimodal behaviour
Bimodal Behaviour

  • Reviewer either gets few or L_i papers

  • Load Penalties [8]

    • Introduce more parameters

    • Infer using modified MCF

    • Learning parameters?

  • Load Rebalancing

    • Tradeoff between MCF optimum and old assignment




Avenues for future work
Avenues for Future Work

  • Document Modelling for Affinity Scores

  • Objective Assignment Evaluation

  • Transitive Citation Scores

  • Load Penalty Parameter Estimation


References
References

  • The Conference Paper Assignment Problem, J. Goldsmith, R.H. Sloan, 2007

  • MultiAgent Systems: Algorithmic, Game-Theoretic, and Logical Foundations, Y. Shoham, K. Leyton-Brown, 2009

  • Automating the Assignment of Submitted Manuscripts to Reviewers, S.T. Dumais, J. Nielson, 1992

  • Semisupervised Regression with cotraining algorithms, Z. Zhou, M. Li, 2007


References contd
References – contd.

5. Learning structured prediction models : A Large Margin Approach, B. Taskar, et al, 2005

6. Ologit : Ordinal Logistic Regression for Zelig, G. King, et al, 2007

7. SVM Learning for Interdependant and Structured Output Spaces, I. Tsochantaridis, et al, 2004

8. Word Alignment via Quadratic Assignment, S. Lacoste-Julien, et al, 2006


ad