Focus Groups, Interviews and Surveys MISY 431/432
Focus Groups • Use to better understand product/issue/system/opinions/usage • Select participants • teachers • students • administrators • parents • Pre-determined questions to cover all important items (open-ended/probe)
Interviews • “Similar” to focus groups • Time-consuming but richer than surveys • Fixed questions or ability to probe • Harder to code and develop statistics • Helpful in developing survey items • What features are important… • Very good when you have small numbers • Can help build relationships
Surveys • Harder to develop (more precision) • Language very important • Understandable? Are you asking what you think you are asking? • Navigate can have many meanings – finding info on a screen or finding info across many screens? • Good for large numbers of respondents • Statistical analysis easier • Pre-test is CRITICAL • How to administer and return rates (surveymonkey)
Other Ways to Collect Data • Research – other people have done the work for you • Observation • Do the task or use the system yourself • Automatically collected data (hit counters)
Common Mistakes • Avoid compound questions • I find the site easy and fun to use… • Avoid vague questions • The reports are useful… (in this case there are many reports with different purposes) • The XYZ report maybe useful (what does maybe mean?) • Avoid technical terms unless audience understands the terms • URL, telephony, etc. • Avoid Sensitive Information If Possible • SSN, Exact Salary, Slovenly/Personal Habits, Passwords • Ask people who can answer the question(s)
Scales • Yes/No versus Likert scales: 1 to 5 or 1 to 7 • When to use each? • Yes/No (you can’t be a little bit dead) • Value of Likert scale (how much and strength of relationships can be determined) • Why 1 to 5 versus 1 to 7 (amount of variance you want) • Check Lists versus Likert scale on each list item • Frequency Scales • Never to Always, Daily to Monthly, etc. • Why use…frequent users may have different concerns from infrequent users
Validity and Reliability • Validity – Are you measuring what you want to measure? • Reliability – If the same person answers your survey twice would you get the same result both times?
Ease of Use and Usefulness • Ease of Use: strongly agree to strongly disagree (7 point scale) 1) Learning to use SIS is easy for me 2) I find it easy to get SIS to do what I want it to do 3) I find using SIS clear and understandable
Ease of Use and Usefulness • Usefulness • 7 point scale: strong agree to strongly disagree • Using SIS enables me to accomplish tasks quickly • Using SIS increases my productivity • Using SIS enhances my effectiveness on the job • Using SIS increases my performance
Example • Perceived Usefulness • Using the Books-a-million.com Web site would improve my performance in shopping or information seeking (e.g., save time or money). • Using the Books-a-million.com Web site would enhance my effectiveness in shopping or information seeking (e.g., get the best deal or find the most information about a product). • Using the Books-a-million.com Web site would make it easier for me to shop or find information. • Perceived Ease of Use • My interaction with the Books-a-million.com Web site is clear and understandable. • It is easy for me to become skillful at using the Books-a-million.com Web site. • I find the Books-a-million.com Web site easy to use.
Usage • The greater the perceived ease of use and usefulness the more likely XXX will be used for its intended purpose • Scales already demonstrated to be valid and reliable • Many other standard measures exist
Student Interview Questions – What do you think? • What is XXXXXX? Can you describe it for us/me? • How do you get access to XXXX? • What do you use the XXXXX website for? • Was the site easy to navigate? Why or why not? • What features did you find useful? Why? • What features did you find not useful (if any)? Why not? • Are there any features you would like to see added to this site? Please describe. • How much time do you normally spend on the site when you access it? • How frequently do you visit the site? • Does the website meet your needs? Why or why not?
2006 UD Advisors Project • Problem: Students not finding or using website resulting in unnecessary calls and visits by students • Analysis • Team used system (data on website not useful) • Web statistics – Individuals got stuck on a page prior to the information • Observed students using website • Focus group about information needed • Survey of users (n=150)
2006 UD Advisors Project • Analysis Continued • Analyzed results • Developed Mock up of new Website for presentation/discussion with Client • Modified proposed website based on Client feedback • Developed and installed new Website • Tested student use of new Website for Ease of Use and Usefulness • Everybody was happy!