Usability Study of the Apply Texas Application including Interviews with High School Counselors

1. Usability Study of theApply TexasApplication including Interviews with High School Counselors Sandra Martinez, Ph.D. Apply Texas Team

2. Usability Study • Performed Spring 2010 by Garrett Stettler as his Master’s project for UT School of Information • Garrett observed, interviewed, and videotaped 15 participants • He interviewed 5 high school counselors

3. Usability Study – the 3rd party • Getting a third party to perform a usability study is more effective than using developers because: • They have less personal involvement in the website, which produces less emotional response. • They are not biased by over-familiarity with the website • They have different expectations than developers

4. How are usability studies helpful? • They evaluate a product by testing it on actual users. • They discover errors and areas for improvement by observing people using the product. • They measure efficiency, accuracy, recall, and emotional responses of test subjects.

5. How the study will be used • The Apply Texas Committee, in authorizing the study, charged the Apply Texas team with four goals: • Fix simple errors • Fix big, urgent problems • Create list of usability issues for the Apply Texas committee to prioritize • Create list of suggested changes for the Apply Texas committee

6. Participants • 8 postsecondary-bound Texas high school students • 2 postsecondary-bound high school students from other states • 2 current college students (transfer) • 1 prospective graduate student • 1 “non-traditional” adult returning to school • 1 pilot participant

7. Areas of research for Apply Texas usability study • 1) Navigation • 2) Copy process • 3) Help • 4) Confusion and frustration • 5) Accuracy and completion

8. Tasks by Group • Group A (“new” users) 1. Create account, input profile information 2. Start and complete a new application 3. Enter essay information 4. Enter scholarship information 5. Logout • Group B (“returning” users) 1. Retrieve forgotten password 2. View/edit a saved profile 3. View/edit a saved application 4. View/edit saved essays 5. Copy an existing application

9. Levels of Concern • Critical concerns • Major concerns • Moderate concerns • Minor concerns

10. Critical concerns (None found) Description of critical concerns: • Critical data may be lost, or • The user may not be able to complete the task, or • The user may not want to continue using the application

11. Major concerns (2 found) Description of major concerns: • Users can accomplish the task but only with considerable frustration and/or unnecessary steps • Non-critical data may be lost • The user will have great difficulty in circumventing the problem • Users can overcome the problem only after they have been shown how to perform the task

12. Major concerns (1 of 2) • Error handling • Error messages are grouped at the top of the page. • Errors are not highlighted within the page itself • Participants tended to skim over error messages and used a “trial-and-error” approach to correcting the error

13. Major concerns (1 of 2)

14. Major concerns (1 of 2) • Error handling recommendations • Errors should be highlighted at the top of the page and at the location in the page where the error has occurred. • If possible, identify the error as the user is typing

15. Major concerns (2 of 2) • Home / My Account / Logout • Navigating to the main account page was confusing • Attempting to log out could be confusing

16. Major concerns (2 of 2) Current view of top toolbar Currently links to logout page Currently links to help page Currently links to logout page Currently links to application listing page

17. Major concerns (2 of 2) Changes for 2011-2012 cycle For new cycle, links to application listing page Links to application listing page (no change) For new cycle, links to logout page For new cycle, name changed to Logoff from Home/Logoff

18. Moderate concerns (5 found) Description of moderate concerns: • The user will be able to complete the task in most cases but will undertake some moderate effort in getting around the problem • The user may need to investigate several links or pathways through the system to determine which option will allow them to accomplish the intended task • Users will most likely remember how to perform the task on subsequent encounters with the system

19. Moderate concerns (1 of 5) • Helper text and other indicators • Every participant failed to notice help text at some point during the test. Helper Text

20. Moderate concerns (1 of 5) Color of helper text changed from gray to blue

21. Moderate concerns (2 of 5) • Buttons or important information appear ‘below the fold’ • Some participants had difficulty completing a task if the continue button required scrolling

22. Moderate concerns (2 of 5) Continue button off the screen

23. Moderate concerns (2 of 5) For new cycle, continue button moved to top of page

24. Moderate concerns (3 of 5) • Confusing terminology on essay page • Participants were unsure about clicking ‘Save’ if they had not entered an essay. • Some participants were unsure about the difference between save and submit.

25. Moderate concerns (3 of 5) Use of the term “save” before anything is entered

26. Moderate concerns (3 of 5) Changed to Enter Topic A

27. Moderate concerns (3 of 5) Changed to Edit after first save

28. Moderate concerns (4 of 5) • Difficulty using Perform High School/College Search function • Over one-third of participants clicked the “perform search” button before entering any text • Two participants failed to select the radio button • Several participants did not initially use the search feature

29. Moderate concerns (4 of 5)

30. Moderate concerns (4 or 5) • The Apply Texas team recommends reworking this page as follows: • Make the “perform high school name search” button more prominent • Ask applicants to enter the name of the school in a new window when the button is clicked • When the applicant chooses a radio button, automatically close the window and fill in information on the application

31. Moderate concerns (5 of 5) • Top Menu and Quick Links • Some participants were confused about the Menu bar and the Quick Links menu, and how they differed • Participants generally did not use the navigation bar at the top for moving between Admissions/Scholarships/Essays/Submit

32. Moderate concerns (5 of 5)

33. Moderate concerns (5 of 5) • The Apply Texas team recommends reworking the navigation and links: • Make the navigation links as a side navbar • Add quick links to the top menu bar with a “mouseover” dropdown.

34. Moderate concerns (5 of 5) Example of a page with a side navigation bar

35. Minor concerns ( 4 found) Description of minor concerns: • The concern is an irritant for the user • There is a cosmetic problem • There is a typographical error

36. Minor concerns (1 of 4) • Confusing/inconsistent terminology • 1) Applicants were unsure what was being asked for “Mini or short” semester

37. Minor concerns (1 of 4) • Confusing/inconsistent terminology • 2) Participants were unsure about the difference between ‘community and volunteer service’ and ‘extracurricular activities’ • 3) Under extracurricular activities, participants were unclear what ‘activity level’ meant • 4) One participant was unclear whether ‘household size’ included only children, or all persons in household.

38. Minor concerns (1 of 4) For mini/short semester, the following paragraph was added for the new cycle as clarification:

39. Minor concerns (2 of 4) • Constraining employment form • Two participants felt the employment form would be difficult to use, especially for applicants who have recurring summer jobs.

40. Minor concerns (3 of 4) • Custom Questions • Many participants found the custom question page to be irritating, particularly for institutions with many custom questions. • The page did not cause confusion, just user irritation as it requires scrolling down the page for each question in order to click the ‘save, acknowledge question button.’

41. Minor concerns (3 of 4)

42. Minor concerns – 3 of 4 • Suggestion: Schools with many custom questions may want to consider moving some of their questions to other modules instead of grouping them all on the custom question page.

43. Minor concerns – 4 of 4 • Yes/No/Reset Answer • Several participants were confused by the Yes/No/Reset Answer response. One suggestion would be to change the button labels from Reset Answer – Not Applicable to No Answer – Not Applicable

44. Usability Study • Overall, Apply Texas is a fairly usable product, with no critical concerns • Many improvements to reduce frustration could be done fairly easily • Some improvements will require reworking certain modules or processes • And now, let’s hear what the high school counselors had to say --

45. High School Counselor interviews • Suggestions for improving Apply Texas: • Give applicants a warning that if they select to send a paper resume, schools may not process their application until the paper resume is received. • Allow more space in online resumes, or provide an upload option into Apply Texas • Provide another mechanism for retrieval of lost usernames, passwords, and application id numbers. • Provide an online application deadline table equivalent to the one available in the paper copy of the Apply Texas application.

46. High School Counselor interviews Application deadline table

47. High School Counselor interviews • Suggestions for improving the High School Counselor System • Different navigation options • Different documentation on what data is available • The ability to separate four year and two year information

48. Usability Study • The Apply Texas team would like to thank Garrett Stettler for his time, effort, and excellent results. • We would also like to thank the Apply Texas committee for authorizing this study. • We are confident that implementing his results will make the application easier and less frustrating for our applicants.

49. Usability Study Questions?