Presentation Objectives

1. Developing Web Design and Usability Guidelines: An Evidence-Based ApproachNovember 4, 2002Craig LafondMary Frances TheofanosCommunication Technologies BranchOffice of Communications • National Cancer Institute

2. Discuss why evidence-based web design is important Describe the process used to create the guidelines Highlight the expert review process used to create stronger guidelines Presentation Objectives

3. We should base website design decisions on facts Attempt to: Move away from ‘country doctor’ model Move toward University-based Physician or Teaching Hospital Model Evidence-Based Solutions

4. Make the best web design decisions by putting the latest research (evidence) into practice early This leads to ensuring that sites always start at the highest level This prevents Exposure of users to unusable sites Wasting of resources Application of ineffective design ideas or design processes Evidence-Based Web Design

5. Identify and translate research into guidelines Create ‘Relative Importance’ ratings Generate ‘Strength of Evidence’ ratings Steps in the Process

6. Identify existing guidelines Yale Web Style Manual Ameritech Web Page User Interface Standards and Design Guidelines Sun Microsystems: Guide to Web Style IBM’s Web Design Guidelines The Library of Congress World Wide Web Style Guide Problems with style guides Contained conflicting guidance Yale: “Use longer pages to ease page maintenance” Sun: “Use shorter pages to make site more maintainable” Guidance too general No references Step 1: Identify and Translate Research into Guidelines

7. May not be specific Audiences Contexts/Situations May not be explicit May not consider usability/design experience Evidence from other disciplines may not be reflected Guideline Limitations

8. Purpose Determine which guidelines practitioners think are most important, Create ‘Relative Importance’ Ratings Reduce and strengthen the list 16 Reviewers 8 web designers 8 usability specialists December 11-31, 2001 Practitioner Review

10. No reliable difference between usability specialists and web designers in how they rated the guidelines ‘Relative Importance’ and ‘Strength of Research’ ratings are correlated Step 2 - Phase 1 Results

11. Allowed reviewers to confirm or change their ratings with knowledge of Their previous ratings The group mean for ‘Relative Importance’ Attempted to arrive at consensus Step 2 - Phase 2

12. 287 guidelines Same 16 reviewers Allowed reviewers to confirm or change their ratings with knowledge of Their previous ratings The group mean for ‘Relative Importance’ Attempted to arrive at consensus Step 2 - Phase 2

13. Reviewers ratings from Phase I New for Phase 2

14. Again no reliable difference between ratings of web designers and usability specialists No reliable difference between ‘Relative Importance’ and ‘Strength of Research’ ratings from Phase 1 to Phase 2 ‘Relative Importance’ and ‘Strength of Research’ are still correlated in Phase 2 (r=.81 in both phases) Step 2 - Phase 2 Results

15. Exclusively on experience Mostly on experience, though somewhat familiar with the research Mostly on experience, though very familiar with the research Half on experience, half on knowledge of the research Mostly on knowledge of the research, with strong confirmation from experience Mostly on knowledge of the research, with weak confirmation from experience Exclusively on knowledge of the research Basis for ‘Strength of Research’ Rating

16. Purposes: Learn whether researchers will rate the guidelines differently than practitioners Assess the level of agreement on an evidence-based rating Collect additional references Identify where there is no evidence to support common web practices Establish an ongoing process for future reviews Researcher Review

17. Nine usability experts All with Strong educational background Expert understanding of the usability literature Good understanding of experimental design issues Step 3 Participants

18. Step 1 Each reviewer put each of the 225 guidelines into one of 3 categories strong research support weak research support no research support If any one of the reviewers indicated that the guideline had any research support, it was kept in the process Reviewers also added new references Researcher Review

19. Step 2 There was little agreement on guidelines with ‘no research support’ Researchers met on August 8 to establish a common framework for rating the guidelines Decided to classify references into 9 types (hypothesis-based, expert opinion, survey data, literature review, etc.) Researcher Review

20. Step 3 Researchers rated each of the guidelines on a common scale: Strong Research Support Cumulative and compelling, supporting research-based evidence At least one formal, rigorous study with contextual validity No known conflicting research-based findings Expert opinion agrees with the research Moderate Research Support Cumulative research-based evidence There may or may not be conflicting research-based findings Expert opinion Tends to agree with the research, and Consensus seems to be building Researcher Review

21. Weak Research Support Limited research-based evidence Conflicting research-based findings may exist and/or There is mixed agreement of expert opinions Strong Expert Opinion Support No research-based evidence experts tend to agree, although there may not be a consensus Multiple supporting expert opinions (presented in textbooks, style guides, newsletters, etc.) Generally accepted as a 'best practice' (or reflects current 'state of practice') Weak Expert Opinion Support No research-based evidence Limited or conflicting expert opinion Researcher Review

22. References by type Observational study – 17 Experiment – 190 Model-based – 15 Expert opinion – 57 Literature review – 78 Survey – 7 Textbook – 27 Usability test – 24 Exploratory study - 20 Results to Date Results to Date

23. Research helps us better understand what works We need More research Better research Greater use of research findings Final goal: To help practitioners make design decisions that achieve higher-quality, professional user interfaces! Conclusions