1 / 15

Chapter 25

Chapter 25. Analysis and interpretation of user observation evaluation data. Analysis and interpretation. Analysis: “just the facts, ma’am” usually just the statistical number crunching (if you have quantifiable data, that is)

uttara
Download Presentation

Chapter 25

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Chapter 25 Analysis and interpretation of user observation evaluation data

  2. Analysis and interpretation • Analysis: “just the facts, ma’am” • usually just the statistical number crunching • (if you have quantifiable data, that is) • “observation” data: can be “quasi-scientific”, meaning a different set of analysis tools than “traditional” techniques like ANOVA • Interpretation: opinions • opportunity to opine about why you think data turned out the way it did

  3. Collating the data • Collating the data: • a rather weak section of the book--all they’re saying is: • bring pen & paper • don’t forget to number your notes with subject id’s • maybe type them up…gosh! • there can be a whole lot more to this…

  4. Collating the data • Collating the data (other considerations): • re-tool your app to record quantitative data, e.g., time to completion (computers make good egg timers) • automatically store the data in (gasp!) database records---can use .csv instead, then read into SQL (really helps in teasing out data later) • see if you can use a stats package to help with the analysis (e.g., the R language -- pretty powerful; will plot stuff too, fairly steep learning curve though) • maybe combine everything into a Makefile script (soooo convenient, especially for your advisor :)

  5. Summarizing the data • “Time spent to recover from an error” = end time - start time • Uh, that’s it? Wait, there’s more… • “…we have not included details or methods for the statistical analysis of evaluation data.” • Are they kidding? That’s where all the powerful insights are! It’s next to impossible to publish papers without that crucial “p < 0.01”

  6. Finding usability defects • Usability defect = usability problem in the UI (I think we get that by now) • Review the data to find defects • Make a table • So far pretty lame chapter…

  7. Quantitative Data • “Quantitative data are more eye catching for the readers…and are often regarded as more objective than qualitative data.” • That’s just an awful thing to say (They’re confusing two issues and mixing in marketing notions such as “pseudoscientific jargon”--4 out of 5 doctors recommends…) • Quantitative data allows numerical / statistical analysis with which you can make convincing arguments • Quantitative data can still be subjective, e.g., after quantifying a questionnaire

  8. Stats • “Be particularly careful to avoid statistical language such as `significant result’…” • …if you don’t know what that means :) • Good call here --- never B.S. your way out of a poorly designed study…if the data doesn’t support your hypothesis, tough • Summarizing quantitative data: • graphs, charts, yadda yadda---crunch the numbers first, then worry about the graphics • descriptive stats: mean, median, etc. • inferential stats: statistical significance---that’s where it’s at---your claim to a scientific study • you still need to know what that means though

  9. Descriptive Stats • “…many readers of usability reports either do not have statistical training or re too busy to challenge your figures.” • Ok, some readers are ignorant • (5 + 6 + 55) / 3 = 22 • 2 experts, 1 novice -- not enough data to say anything substantial but the point is made; can’t just blindly report stats, have to reason about the results

  10. ANOVA • A very popular approach based on assumption of normal distribution • Performs mean (and variance) comparisons • Consider (5 + 6 + 55) / 3 = 22 • Suppose we had a bunch of experts and a bunch of novices (say 10 each) • Do ANOVA to see if the groups differ

  11. Simple ANOVA example (using Excel’s data analysis) F(1,18) = 2452.74, p < 0.01 “Experts performed task significantly faster than novices (F(1,18) = 2452.74, p < 0.01).” “Experts are probably faster than novices due to their considerable experience.”

  12. Qualitative Data • Often adds insights into quantitative results • Look for trends in user responses, particularly in their statements, e.g., “this is difficult”, things like that • You may have to pay attention to body language, facial expressions, etc.

  13. Coding scheme? • How to group comments? • by chronological order? • by severity of defect? • by UI component? • your certainty of the cause of the comment? • difficulty of fixing the defect? • Perhaps a coding scheme is needed? • any method of assigning a group, number, or label to an item of data • Say…I can use this: “A usability coding scheme for eye movement metrics”, see CHI 2006 workshop

  14. Data (Results) Interpretation • Summarize data, assign severity rankings • Recommend changes (similar table)

  15. Write it up • Academic paper: • abstract • introduction • background • methodology • results • discussion • conclusions & future work

More Related