EasyChair Reviews, comments and meta-reviews

1. EasyChairReviews, comments and meta-reviews Interim Presentation Team: Super-Chairs Aaron Donk, Vatsal Shah,  Ammar Taki El-Din, Gilberto Bardales

2. Activities so far..... • 1 prototype with minimal change and most adaptable to the existing website • 1 prototype with substantial changes to the web layout but still in consistency with original website • Pilot Studies • 2 total (1 cancelled, planned for 3) • Subjects were asked to navigate through 5 tasks on the original website as well as the 2 prototypes. • Followed by interview and critiquing session • this was complimented by showing the subject the webpage in question in order to recollect their memories of the experience.

3. Plan of Action • User reactions to prototypes. • what is better? • what is worse? • is the interface usable? • where did the users have problems? • Make necessary changes in the prototypes' designs. • make a layout of everything that users liked from both prototypes. • add what users wanted. • follow relevant HCI guidelines • Implement new design.

4. First Impressions Users thought... • the website was too crowded • the website contained too many links • the website contained irrelevant and multiple links with different names • the website was inconsistent as new menu items were added in between a list of existing items -- this confused the user and forced them to scan all the links again • the website was too wordy and that it was unnecessary to read all the text

5. Pilot Study #1 • The site is generally navigable • Subjects did not read all text • Pre-filled text boxes were misunderstood • Did not mind ratings including both numerical and word representation • The "show reviews" portion of the site took the longest to understand

6. Pilot Study #2 Original Website • Too cluttered • Too wordy • Default selections led the user to not make additional changes • Pre-filled templates led the user to believe that nothing needed to be changed • Notation next to the 'First name' was never understood and was confusing throughout the website; user did not read the instructions for it • '*' notation was commonly interpreted as a required field • User was confused between 'Review Form' and 'Add Review' • All notes next to text fields were usually ignored as the their attention was focused on the text box itself • User was bothered by inconsistency  in the numbering system as some measures included negative numbers • Confusion between two separate 'Comment' boxes and their usage • Didn't know what the comment was for (meta-review?)

7. Pilot Study #2 cont. Prototype Websites • Simpler main page (less crowded and less words) • More visibility of options without much need of scrolling • Confused between papers already reviewed and those that weren't • Worried about disclosing author information (same with original website) • Inconsistency between numbered grading was replaced in 1 prototype and the user liked that feature reasonably well • More relevant options were displayed towards the top in order to reduce user effort and users really liked that • Consistent menu items and increased afford-ability with use of buttons instead of text links • Overall issues with technicalities for rating papers 

8. Design and Prototype EasyChair Prototypes

9. Updated Timeline

10. Overall Problems • Misunderstandings with website layout • Issues with testing software • Scheduling conflicts • Unreliable subjects • Difficulty in finding subjects • Lengthy study times • Scope for improvement is large but time constraint is small • Too many webpages to study and analyze

11. Upcoming Decisions and Changes • 1-2 more prototypes based on results from up-coming subjects • User studies (7 subjects) • Recruitment and selection has already been done • 3-4 extra/back-up subject selected as well • Improvised script for user study in order to shorten time period of the study • Fewer interview questions • Further implementation of final product • Based on prototype that was most appreciated and liked by the users • Documentation • Final Report

12. Risk Analysis • Rescheduling of subjects • Time conflicts between subjects and teammates due to class-time restrictions • Subjects leaving in between due to other commitments, boredom, etc. • Subjects refusing to do a post-study interview • User needs are higher than technological capabilities allowed for the modification of the website • Conflicting user opinions lead to stagnant progress as prototype cannot accommodate conflicting changes • End result may end up being similar to original website if users find no problems with it

13. Thank You! Question?