automating assessment of web site usability l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Automating Assessment of Web Site Usability PowerPoint Presentation
Download Presentation
Automating Assessment of Web Site Usability

Loading in 2 Seconds...

play fullscreen
1 / 34

Automating Assessment of Web Site Usability - PowerPoint PPT Presentation


  • 252 Views
  • Uploaded on

Automating Assessment of Web Site Usability Marti Hearst Melody Ivory Rashmi Sinha University of California, Berkeley 196M new Web sites in the next 5 years [Nielsen99] ~20,000 user interface professionals [Nielson99] The Usability Gap

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Automating Assessment of Web Site Usability' - issac


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
automating assessment of web site usability
Automating Assessment of Web Site Usability

Marti Hearst

Melody Ivory

Rashmi Sinha

University of California, Berkeley

the usability gap

196M new Web sites in the next 5 years [Nielsen99]

~20,000 user interface professionals [Nielson99]

The Usability Gap

ASIS IA Summit, Feb 2001

the usability gap3

A shortage of user interface professionals [Nielson99]

The Usability Gap

196M new Web sites in the next 5 years [Nielsen99]

Most sites have inadequate usability [Forrester, Spool, Hurst]

(users can’t find what they want 39-66% of the time)

ASIS IA Summit, Feb 2001

the problem
The Problem
  • NON-professionals need to create websites
  • Guidelines are helpful, but
    • Sometimes imprecise
    • Sometimes conflict
    • Usually not empirically founded

ASIS IA Summit, Feb 2001

ultimate goal tools to help non professional designers
Ultimate Goal: Tools to Help Non-Professional Designers
  • Examples:
    • A “grammar checker” to assess guideline conformance
      • Imperfect
      • Only suggestions – not dogma
    • Automatic comparison to highly usable pages/sites
    • Automatic template suggestions

ASIS IA Summit, Feb 2001

a view of web site structure newman et al 00
A View of Web Site Structure (Newman et al. 00)
  • Information design
    • structure, categories of information
  • Navigation design
    • interaction with information structure
  • Graphic design
    • visual presentation of information and navigation (color, typography, etc.)

ASIS IA Summit, Feb 2001

Courtesy of Mark Newman

a view of web site design newman et al 00
A View of Web Site Design(Newman et al. 00)
  • Information Architecture
    • includes management and more responsibility for content
  • User Interface Design
    • includes testing and evaluation

Courtesy of Mark Newman

ASIS IA Summit, Feb 2001

the goal
The Goal
  • Eventually want to assess navigation structure and graphic design at the page and site level.
  • Farther down the line: information design and scent
  • Note: we are NOT suggesting we can characterize:
    • Aesthetics
    • Subjective preferences

ASIS IA Summit, Feb 2001

the investigation
The Investigation
  • Can we place web design guidelines onto an empirical foundation?
  • Can we build models of good design by looking at existing designs?

ASIS IA Summit, Feb 2001

example empirical investigation

Example Empirical Investigation

Is it all about the content?

ASIS IA Summit, Feb 2001

webby awards 2000
Webby Awards 2000
  • 6 criteria
  • 27 categories
    • We used finance, education, community, living, health, services
  • 100 judges
    • International Academy of Digital Arts & Sciences
    • 3 rounds of judging
  • 2000 sites initially

ASIS IA Summit, Feb 2001

webby awards 200012
Webby Awards 2000
  • 6 criteria
    • Content
    • Structure & navigation
    • Visual design
    • Functionality
    • Interactivity
    • Overall experience
  • Scale: 1-10 (highest)
  • Nearly normally distributed across judged sites
  • What are Webby judgements about?

ASIS IA Summit, Feb 2001

webby awards 200013
Webby Awards 2000
  • The best predictor of the overall score is the score for content
  • The worst predictor is visual design

ASIS IA Summit, Feb 2001

so webbys focus on content
So … Webbys focus on content!

ASIS IA Summit, Feb 2001

slide15

Comparing Two Categories

news

arts

ASIS IA Summit, Feb 2001

guidelines
Guidelines
  • There are MANY usability guidelines
  • A survey of 21 sets of web guidelines found little overlap (Ratner et al. 96)
  • Why?
    • Our hypothesis: not empirically validated
    • So … let’s figure out what works!

ASIS IA Summit, Feb 2001

web page metrics
Web Page Metrics
  • Web metric analysis tools report on what is easy to measure
    • Predicted download time
    • Depth/breadth of site
  • We want to worry about
    • Content
    • User goals/tasks
  • We also want to compare alternative designs.

ASIS IA Summit, Feb 2001

another empirical study
Another Empirical Study:

Which features distinguish

well-designed web pages?

ASIS IA Summit, Feb 2001

quantitative metrics
Quantitative Metrics
  • Identified 42 attributes from the literature
  • Roughly characterized:
    • Page Composition (e.g., words, links, images)
    • Page Formatting (e.g., fonts, lists, colors)
    • Overall Page Characteristics

(e.g., information & layout quality, download speed)

ASIS IA Summit, Feb 2001

metrics used in study
Word Count

Body Text Percentage

Emphasized Body Text Percentage

Text Positioning Count

Text Cluster Count

Link Count

Page Size

Graphic Percentage

Graphics Count

Color Count

Font Count

Metrics Used in Study

ASIS IA Summit, Feb 2001

data collection
Data Collection
  • Collected data for 1898 pages from 163 sites
    • Attempted to collect from 3 levels within each site
  • Six Webby categories
    • Health, Living, Community, Education, Finance, Services
  • Data constraints
    • At least 30 words
    • No pages with forms
    • Exhibit high self-containment (i.e., no style sheets, scripts, applets, etc.)

ASIS IA Summit, Feb 2001

method
Method
  • Collect metrics
    • from sites evaluated for Webby Awards 2000
  • Two comparisons
    • Top 33% of sites vs. the rest (using the overall Webby score)
    • Top 33% of sites vs. bottom 33% (using the Webby factor)
  • Goal: see if we can use the metrics to predict membership in top vs. other groups.

ASIS IA Summit, Feb 2001

questions
Questions:
  • Can we use the metrics to predict membership in top vs. other groups?
  • Do we see a difference in how the metrics behave in different content categories?

ASIS IA Summit, Feb 2001

findings
Findings
  • We can accurately classify web pages
    • Linear discriminant analysis
    • For top vs. rest
      • 67% correct for overall
      • 73% correct when taking categories into account
    • For top vs. bottom
      • 65% correct for overall
      • 80% correct using categories

ASIS IA Summit, Feb 2001

why does this work
Why does this work?
  • Content is most important predictor of overall score
  • BUT there is some predictive power in the visual design / navigation criteria
  • Also, it may just be that good design is good design all over
    • Film making analogy
    • This happens in other domains – automatic essay grading for one

ASIS IA Summit, Feb 2001

deeper analysis
Deeper Analysis
  • Which metrics matter?
    • All played a role
  • To get more insight:
    • We noticed that small, medium, and large pages behave differently
    • We subdivided pages according to size and category to find out which metrics matter and if they should have high or low values

ASIS IA Summit, Feb 2001

small pages 66 words on average
Small pages (66 words on average)
  • Good pages have slightly more content, smaller page sizes, less graphics and employ more font variations
  • The smaller page sizes and graphics count suggests faster download times for these pages (corroborated by a download time metric, not discussed in detail here).
  • Correlations between font count and body text suggest that good pages vary fonts used between header and body text.

ASIS IA Summit, Feb 2001

medium pages 230 words on average
Medium pages (230 words on average)
  • Good pages emphasize less of the body text
  • Text positioning and text cluster count indicate medium-sized good pages appear to organize text into clusters (e.g., lists and shaded table areas).
  • Negative correlations between body text and color count suggests that good medium-sized pages use colors to distinguish headers.

ASIS IA Summit, Feb 2001

large pages 827 words on average
Large pages (827 words on average)
  • Good pages have less body text and more colors (suggesting pages have more headers and text links)
  • Good pages are larger but have fewer graphics

ASIS IA Summit, Feb 2001

future work
Future work
  • Distinguish according to page role
    • Home page vs. content vs. index …
  • Better metrics
    • Separate info design, nav design, graphic design
  • Site level as well as page level
  • Compare against results of live user studies

ASIS IA Summit, Feb 2001

future work31
Future work
  • Category-based profiles
    • Can use clustering to create profiles of good and poor sites for each category
    • These can be used to suggest alternative designs
  • More information: CHI 2001 paper

ASIS IA Summit, Feb 2001

ramifications
Ramifications
  • It is remarkable that such simple metrics predict so well
    • Perhaps good design is good overall
    • There may be other factors
  • A foundation for a new methodology
    • Empirical, bottom up
  • But, there is no one path to good design!

ASIS IA Summit, Feb 2001

in summary
In Summary
  • Automated Usability Assessment should help close the Web Usability Gap
  • We can empirically distinguish between highly rated web pages and other pages
    • Empirical validation of design guidelines
    • Can build profiles of good vs. poor sites
    • Are validating expert judgements with usability assessments via a user study
  • Eventually want to build tools to help end-users assess their designs

ASIS IA Summit, Feb 2001

slide34
More information:
  • http://webtango.berkeley.edu
  • http://www.sims.berkeley.edu/~hearst

ASIS IA Summit, Feb 2001