1 / 16

Paper Rating vs. Paper Ranking

what PCs. do now. Paper Rating vs. Paper Ranking. what I am. advocating. John R. Douceur. Program Committee Selects Papers. Step 1: Reviewers Rate Papers. 2.0. Step 2: PC Meeting. 2.0. 0.9. 3.3. 3.2. 1.6. 2.3. 1.9. 2.8. 1.5. 1.3. 1.4. 1.0. 0.6. 1.9. 3.9. 3.6. 1.0. 3.0.

georgebsmith
Download Presentation

Paper Rating vs. Paper Ranking

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. what PCs do now Paper Ratingvs.Paper Ranking what I am advocating John R. Douceur

  2. Program Committee Selects Papers

  3. Step 1: Reviewers Rate Papers 2.0

  4. Step 2: PC Meeting 2.0 0.9 3.3 3.2 1.6 2.3 1.9 2.8 1.5 1.3 1.4 1.0 0.6 1.9 3.9 3.6 1.0 3.0 3.1 2.5 2.8 2.2

  5. Submission Quality Varies per Year papers papers papers 2006 Quality Quality Quality good good good bad bad bad 2007 2008

  6. Conflate Assessment with Standards

  7. Problem #1 with Ratings The reviewing process conflatesreviewers’ standards of stringencyand leniency with the reviewers’assessments of the merits andweaknesses of each paper.

  8. Insufficient Comparisons

  9. Brehm’s Phenomenon =

  10. Problem #2 with Ratings In the PC meeting, early acceptancedecisions are based upon insufficientinformation; these decisions becomeentrenched for psychological reasons.

  11. What’s the Alternative? I’m glad you asked...

  12. Reviewer Assessment

  13. PC Meeting

  14. PC Meeting – Phase I I thought was this paper much weaker than This paper has that one. a great evaluation. How well evaluated are higher ones?

  15. PC Meeting – Phase II min max

  16. Summary Rating what PCs do now Conflates reviewers’ assessments with standards. Entrenches decisions based on insufficient data. Ranking what I am advocating Compares each paper to its competition. Defers decisions until relative quality is known.

More Related