Best Practices for the Care and Feeding of a Program Committee - PowerPoint PPT Presentation

Slide1 l.jpg
Download
1 / 11

Review management software. Best practices. Ideas for new ways to do things. How well do ... reviews. Some are too argumentative, dominate the rest of the PC, or ...

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Best Practices for the Care and Feeding of a Program Committ...

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Slide1 l.jpg

Best Practices for the Care and Feeding of a Program Committee, and Other Thoughts on Conference Organization

Fred Douglis

IBM T.J. Watson Research Center

Best Practices


Wow it s wowcs l.jpg

WOW … It’s WOWCS

  • First reaction: what a great idea!

  • Second reaction: will anyone actually submit?

    • Immediate fear: spend time writing something up, only to find out it’s been canceled

  • Final reaction: duty

    • I’ve chaired USENIX’98, USITS’99, and a few other conferences

    • Most went well but there were some issues

  • End goal: guidebook (wiki) for program/general chairs

    • https://wiki.usenix.org/bin/view/Main/Conference/CollectedWisdom

Best Practices


Slide3 l.jpg

Wiki

  • Goals

    • Living document

    • Allow discussion

  • Two types of info eventually

    • Guide for new program chairs or conference organizers

      • Review management software

      • Best practices

    • Ideas for new ways to do things

      • How well do rebuttals work? Ratings? Bidding?

Best Practices


Problem 1 bad reviewers l.jpg

Problem 1: Bad Reviewers

  • Not everyone is a good PC member

    • Some never do anything

    • Some do really crummy reviews

    • Some are too argumentative, dominate the rest of the PC, or have other personality issues

  • How to find who is a known bad reviewer?

    • Past experience

    • Word of mouth… but people tend not to know to ask

  • How to expand word of mouth?

    • Reviewer database?

Best Practices


Problem 2 inexperienced reviewers l.jpg

Problem 2: Inexperienced Reviewers

  • Always want to bring some “new blood” into a PC

    • Search for repeat authors who haven’t served (need a script to scour Google Scholar for this)

    • Caveat: not everyone knows what to expect and may not be cut out for it. Don’t take too many people you don’t know if you can help it

  • Tips for ensuring the best PC

    • Set expectations early (# reviews, timing, … no surprises!)

    • Have multiple deadlines: force people to miss early rather than all at once just before the decisions

    • Help with calibration (average scores of reviewers compared to the scores of their peers on same papers)

Best Practices


Problem 3 pc composition l.jpg

Problem 3: PC Composition

  • Avoid inbreeding, or the appearance of it

    • Overlap from year to year

    • Institutional overlap

  • There are enough people who will have published at an established conference that a chair should draw on them rather than outside the conference community

    • Best fit

    • Reward participation

    • Avoid problems with calibration

Best Practices


Pc bells and whistles l.jpg

PC Bells and Whistles

  • Options for running a PC

    • Rebuttals (Seem to be a nice idea, but add lead time)

    • Reviewer ratings (hard to calibrate; don’t want to insult)

Best Practices


Running a conference l.jpg

Running a Conference

  • Sponsorship

    • Some institutions make it a pleasure to run a conference, and some make it much harder

    • Doing it alone has risks (e.g., liability): who “owns” the conference risks?

  • Setting expectations

    • It’s hard to know just how many submissions a new conference will get: better to be swamped (and add PC members) than miss all your targets

    • Publicity is amazingly critical (who knew?)

  • Scheduling

    • Watch for conflicts and hope the others watch too

    • Consider the impact of rejected papers from one conference being routed to another. (Feed reviews?)

Best Practices


Backup l.jpg

Backup

Best Practices


Problem 4 self plagiarism l.jpg

Problem 4: (Self)plagiarism

  • Simultaneous submission seems to be a big problem

    • Least publishable units

    • True plagiarism

  • We need a better mechanism for detecting

    • Serendipity doesn’t cut it

Best Practices


Concrete proposals challenges l.jpg

Concrete Proposals/Challenges

  • Plagiarism

    • Neutral agent to collect papers and analyze for overlap

    • Get multiple organizations to buy in

  • Reviewer database

    • Neutral agent to collect feedback, and if someone gets multiple negative reports, somehow blacklist

  • See Internet Computing columns Sept-Oct/Nov-Dec 2007

Best Practices


  • Login