Slide1 l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 11

Best Practices for the Care and Feeding of a Program Committee PowerPoint PPT Presentation


Review management software. Best practices. Ideas for new ways to do things. How well do ... reviews. Some are too argumentative, dominate the rest of the PC, or ...

Download Presentation

Best Practices for the Care and Feeding of a Program Committ...

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Slide1 l.jpg

Best Practices for the Care and Feeding of a Program Committee, and Other Thoughts on Conference Organization

Fred Douglis

IBM T.J. Watson Research Center

Best Practices


Wow it s wowcs l.jpg

WOW … It’s WOWCS

  • First reaction: what a great idea!

  • Second reaction: will anyone actually submit?

    • Immediate fear: spend time writing something up, only to find out it’s been canceled

  • Final reaction: duty

    • I’ve chaired USENIX’98, USITS’99, and a few other conferences

    • Most went well but there were some issues

  • End goal: guidebook (wiki) for program/general chairs

    • https://wiki.usenix.org/bin/view/Main/Conference/CollectedWisdom

Best Practices


Slide3 l.jpg

Wiki

  • Goals

    • Living document

    • Allow discussion

  • Two types of info eventually

    • Guide for new program chairs or conference organizers

      • Review management software

      • Best practices

    • Ideas for new ways to do things

      • How well do rebuttals work? Ratings? Bidding?

Best Practices


Problem 1 bad reviewers l.jpg

Problem 1: Bad Reviewers

  • Not everyone is a good PC member

    • Some never do anything

    • Some do really crummy reviews

    • Some are too argumentative, dominate the rest of the PC, or have other personality issues

  • How to find who is a known bad reviewer?

    • Past experience

    • Word of mouth… but people tend not to know to ask

  • How to expand word of mouth?

    • Reviewer database?

Best Practices


Problem 2 inexperienced reviewers l.jpg

Problem 2: Inexperienced Reviewers

  • Always want to bring some “new blood” into a PC

    • Search for repeat authors who haven’t served (need a script to scour Google Scholar for this)

    • Caveat: not everyone knows what to expect and may not be cut out for it. Don’t take too many people you don’t know if you can help it

  • Tips for ensuring the best PC

    • Set expectations early (# reviews, timing, … no surprises!)

    • Have multiple deadlines: force people to miss early rather than all at once just before the decisions

    • Help with calibration (average scores of reviewers compared to the scores of their peers on same papers)

Best Practices


Problem 3 pc composition l.jpg

Problem 3: PC Composition

  • Avoid inbreeding, or the appearance of it

    • Overlap from year to year

    • Institutional overlap

  • There are enough people who will have published at an established conference that a chair should draw on them rather than outside the conference community

    • Best fit

    • Reward participation

    • Avoid problems with calibration

Best Practices


Pc bells and whistles l.jpg

PC Bells and Whistles

  • Options for running a PC

    • Rebuttals (Seem to be a nice idea, but add lead time)

    • Reviewer ratings (hard to calibrate; don’t want to insult)

Best Practices


Running a conference l.jpg

Running a Conference

  • Sponsorship

    • Some institutions make it a pleasure to run a conference, and some make it much harder

    • Doing it alone has risks (e.g., liability): who “owns” the conference risks?

  • Setting expectations

    • It’s hard to know just how many submissions a new conference will get: better to be swamped (and add PC members) than miss all your targets

    • Publicity is amazingly critical (who knew?)

  • Scheduling

    • Watch for conflicts and hope the others watch too

    • Consider the impact of rejected papers from one conference being routed to another. (Feed reviews?)

Best Practices


Backup l.jpg

Backup

Best Practices


Problem 4 self plagiarism l.jpg

Problem 4: (Self)plagiarism

  • Simultaneous submission seems to be a big problem

    • Least publishable units

    • True plagiarism

  • We need a better mechanism for detecting

    • Serendipity doesn’t cut it

Best Practices


Concrete proposals challenges l.jpg

Concrete Proposals/Challenges

  • Plagiarism

    • Neutral agent to collect papers and analyze for overlap

    • Get multiple organizations to buy in

  • Reviewer database

    • Neutral agent to collect feedback, and if someone gets multiple negative reports, somehow blacklist

  • See Internet Computing columns Sept-Oct/Nov-Dec 2007

Best Practices


  • Login