slide1 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Best Practices for the Care and Feeding of a Program Committee PowerPoint Presentation
Download Presentation
Best Practices for the Care and Feeding of a Program Committee

Loading in 2 Seconds...

play fullscreen
1 / 11

Best Practices for the Care and Feeding of a Program Committee - PowerPoint PPT Presentation


  • 197 Views
  • Uploaded on

Review management software. Best practices. Ideas for new ways to do things. How well do ... reviews. Some are too argumentative, dominate the rest of the PC, or ...

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Best Practices for the Care and Feeding of a Program Committee' - Kelvin_Ajay


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Best Practices for the Care and Feeding of a Program Committee, and Other Thoughts on Conference Organization

Fred Douglis

IBM T.J. Watson Research Center

Best Practices

wow it s wowcs
WOW … It’s WOWCS
  • First reaction: what a great idea!
  • Second reaction: will anyone actually submit?
    • Immediate fear: spend time writing something up, only to find out it’s been canceled
  • Final reaction: duty
    • I’ve chaired USENIX’98, USITS’99, and a few other conferences
    • Most went well but there were some issues
  • End goal: guidebook (wiki) for program/general chairs
    • https://wiki.usenix.org/bin/view/Main/Conference/CollectedWisdom

Best Practices

slide3
Wiki
  • Goals
    • Living document
    • Allow discussion
  • Two types of info eventually
    • Guide for new program chairs or conference organizers
      • Review management software
      • Best practices
    • Ideas for new ways to do things
      • How well do rebuttals work? Ratings? Bidding?

Best Practices

problem 1 bad reviewers
Problem 1: Bad Reviewers
  • Not everyone is a good PC member
    • Some never do anything
    • Some do really crummy reviews
    • Some are too argumentative, dominate the rest of the PC, or have other personality issues
  • How to find who is a known bad reviewer?
    • Past experience
    • Word of mouth… but people tend not to know to ask
  • How to expand word of mouth?
    • Reviewer database?

Best Practices

problem 2 inexperienced reviewers
Problem 2: Inexperienced Reviewers
  • Always want to bring some “new blood” into a PC
    • Search for repeat authors who haven’t served (need a script to scour Google Scholar for this)
    • Caveat: not everyone knows what to expect and may not be cut out for it. Don’t take too many people you don’t know if you can help it
  • Tips for ensuring the best PC
    • Set expectations early (# reviews, timing, … no surprises!)
    • Have multiple deadlines: force people to miss early rather than all at once just before the decisions
    • Help with calibration (average scores of reviewers compared to the scores of their peers on same papers)

Best Practices

problem 3 pc composition
Problem 3: PC Composition
  • Avoid inbreeding, or the appearance of it
    • Overlap from year to year
    • Institutional overlap
  • There are enough people who will have published at an established conference that a chair should draw on them rather than outside the conference community
    • Best fit
    • Reward participation
    • Avoid problems with calibration

Best Practices

pc bells and whistles
PC Bells and Whistles
  • Options for running a PC
    • Rebuttals (Seem to be a nice idea, but add lead time)
    • Reviewer ratings (hard to calibrate; don’t want to insult)

Best Practices

running a conference
Running a Conference
  • Sponsorship
    • Some institutions make it a pleasure to run a conference, and some make it much harder
    • Doing it alone has risks (e.g., liability): who “owns” the conference risks?
  • Setting expectations
    • It’s hard to know just how many submissions a new conference will get: better to be swamped (and add PC members) than miss all your targets
    • Publicity is amazingly critical (who knew?)
  • Scheduling
    • Watch for conflicts and hope the others watch too
    • Consider the impact of rejected papers from one conference being routed to another. (Feed reviews?)

Best Practices

backup
Backup

Best Practices

problem 4 self plagiarism
Problem 4: (Self)plagiarism
  • Simultaneous submission seems to be a big problem
    • Least publishable units
    • True plagiarism
  • We need a better mechanism for detecting
    • Serendipity doesn’t cut it

Best Practices

concrete proposals challenges
Concrete Proposals/Challenges
  • Plagiarism
    • Neutral agent to collect papers and analyze for overlap
    • Get multiple organizations to buy in
  • Reviewer database
    • Neutral agent to collect feedback, and if someone gets multiple negative reports, somehow blacklist
  • See Internet Computing columns Sept-Oct/Nov-Dec 2007

Best Practices