Health of Conferences Committee Update

1. Health of ConferencesCommittee Update February 10, 2006 @ SIG Governing Board Meeting

2. Executive Summary • Working for six weeks to find some best practices • Five questions yielded responses from 30 SIGS • Reviewer load • Non-incremental • Program committees • Workshops, etc. • Catch all • Have some first-impression results • Want feedback • markhill@cs.wisc.edu or baglio@hq.acm.org

3. Talk Outline • Set Up • Charge • Process • Response Sources so Far • Caveats • Questions & Example Responses • Appendix: Full text of questions

4. Charge from ACM President David Patterson • The idea is to collect the best practicesonto a web page so that conference organizers can see innovative waysto cope with the demands of paper submissions, refereeing, and presentations as the number of papers increase.The hope is that organizers will either try good new ideas or at least avoid the mistakes of others.

5. Process • Dec. 2005 committee formed; Baglio & Hill start up • Jan. 5 call to divide responsibly & set questions • Marks – ACM SIGs with BIG conferences (> 1K attendees) • Hall – ACM SIGs with MEDIUM • Baglio – ACM SIG with SMALL (< 100) • Gaudiot/Prinetto – IEEE in manner to be done • Iterated on questions; sent out; got many responses • Feb 6 call to check status & Feb 8/9 info to Hill • Feb 10 status to SGB (this talk) • Feb 27 call to review data on Wiki (we hope)

6. Response Sources so Far • ACM Big: • DAC, ICSE, OOPSLA, SIGCSE, SIGGRAPH • ACM Medium: • SIGART, SIGARCH, SIGCHI , SIGIR , SIGACT,SIGPLAN , SIGMOD, SIGCOMM , SIGKDD • ACM Small: • SIGAda, SIGAPP, SIGBED, SIGACCESS, SIGCSE, SIGDA, SIGDOC, SIGecom, SIGITE, SIGMETRICS, SIGMICRO, SIGMIS, SIGSAC, SIGSAM, SIGSIM, SIGUCCS, SIGWEB • IEEE: To be done ,

7. Regarding First-Impression Results • Goal • Unearth actionable ideas • Reveal failed ideas • Non-Goal • Determine summary statistics, becauseaudience interested in SIGs like theirs • Caveat • Data arrived to me yesterday & day before

8. Q1: Reviewer Load • Tracking reviews? Increasing PC size? Fee? Others? • Big (SIGCSE): Maintain reviewer database • Medium (SIGART): Senior PC supervises whole PC • Small (SIGUCCS): tracking reviews among conferences (like journals)

9. Q2: Non-Incremental • Big ideas sessions? More papers? Shorter papers? Deemphasizing detailed evaluation? Others? • Big (ICSE): fun flames over beer & snacks • Medium (SIGKDD): KDD Cup competition to evaluate alternative approaches to the same problem • Small (SIGDOC): Earlier international acceptances to ease visa problems

10. Q3: Program Committees • Double blind submissions? Program committee submission restrictions? Rebuttals? Large PCs? Program subcommittees? Others? • Big (SIGCSE): Encourage large PCs • Medium (SIGCHI): Rebuttals avoid compounding misunderstandings, make authors feel better, & help get reviews on time • Small (SIGITE): Single-person PC

11. Q4: Workshops, etc. • Workshop co-located at conferences? Stand-alone workshops? Panels? Crazy idea sessions? • Big (OOPSLA): 5-minute lightning talks • Medium (SIGARCH): Some workshops get special issues in its newsletter • Small (SIGAda): Extended abstracts & experience reports

12. Q5: Catch All • Are there other approaches your community has tried or abandoned that the rest of us can learn from? • Big (SIGGRAPH): Don’t be afraid to try; don’t be afraid to stop • Medium (SIGPLAN): Professionally supported PC software is big win • Small (SIGMICRO): Need better ways to handle conflicts of interest

13. Executive Summary • Working for six weeks to find some best practices • Five questions yielded responses from 30 SIGS • Reviewer load • Non-incremental • Program committees • Workshops, etc. • Catch all • Have some first-impression results • Want feedback • markhill@cs.wisc.edu or baglio@hq.acm.org

14. Q1: Reviewer Load • Has your community recently adopted new practices to deal with growing reviewer load, such as: • tracking reviews of rejected papers from conference to conference as is done in journal reviewing • increasing program committee size • charging a review fee • others? • For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

15. Q2: Non-Incremental • Has your community recently adopted new practices to promote non-incremental new ideas? • big ideas sessions • more papers • shorter papers • deemphasizing detailed evaluation • others? • For each practice you are using, what is your view of how well it is working within your community? Please comment on the merit of the other strategies as applies to your community.

16. Q3: Program Committees • Does your community practice: • double blind submissions • program committee submission restrictions • rebuttals (author responses) • large program committees • program subcommittees • others? • Do these practices seem to help or hurt promoting your field?

17. Q4: Workshops, etc. • Does your community provide venue for work not mature enough for your major conferences, such as: • workshop co-located at conferences • stand-alone workshops • Panels • crazy idea sessions • On balance, are these other venues effect for advancing your field? What mechanisms, if any, do you use allow good papers from these venues to later achieve wider dissemination?

18. Q5: Catch All • Are there other approaches your community has tried or abandoned that the rest of us can learn from?