DEVELOPING MOBILE APPS FOR THE VISUALLY IMPAIRED

1. DEVELOPING MOBILE APPS FOR THE VISUALLY IMPAIRED • HEARTLAND COMMUNITY COLLEGE • Nicolaas tenBroekTodd Simeone

2. WHO ARE YOU? • And why are you here?

3. HCC’S MOBILE CURRICULUM Computer Science I Android iOS Windows Phone Games Comm Apps for BLV Apps

4. HOW WE BECAME INVOLVED IN DEVELOPING APPS FOR THE BLIND • Independence Science • Desire for classroom apps on iOS • Chemistry • Basic Mathematics • Graphing • Advanced Mathematics • Other Educational Apps

5. ISSUES CONVERTING VISUAL INTERFACES TO SUPPORT VOICEOVER • The amount of information conveyed by a visual interface is tremendous. Graphics and the relationship between controls instantly convey functional information that would take paragraphs of text to explain. • Animation and graphical gloss are often used to explain what is happening within an app. • Many common idioms simply do not transfer. For instance: • What does a “Back” button mean to someone who did not see the screen move “forward”? • “Pull down to refresh” is only meaningful because you saw the hidden setting when the table bounced after scrolling to the top. • Items partially obscured by a translucent control instantly tell a sighted person that more things are available, but overlapping controls will confuse a screen reader

6. ISSUES CONVERTING VISUAL INTERFACES TO SUPPORT VOICEOVER • Most apps are designed to be used quickly and in non-ideal circumstances, but screen readers take a long time to read, and can interfere with a blind user’s interaction with the app, increasing frustration and confusion. • Gestures that make sense when tied to the visual display (i.e. those that mimic real world interactions) have to change when the screen reader is active, and often lose all real world meaning. For instance, if you used a swipe gesture to push a ball around a screen, that would mimic a real-world interaction. With the screen reader turned on, that same gesture will cycle between elements on the screen (from top to bottom and left to right).

7. HOW PEOPLE WITH BLV INTERACT WITH MOBILE DEVICES • Often listen to screen reader • Explore app by moving finger around • Use “List Mode” to cycle between items on screen • Tap off the selected control to use it

8. COMMON DESIGN APPROACHES AND THE PROBLEMS WE FOUND • Just add screen reader support to existing design • Can make it very difficult to reliably hit controls • Lack of spacing can cause incorrect controls to be selected • List Mode often rearranges the controls which can quickly remove all meaning as positioning of labels and images are often used to inform the user of the meaning of other controls • Constant screen reading slows down the user’s interaction with the app, making it even more difficult to use

9. COMMON DESIGN APPROACHES AND THE PROBLEMS WE FOUND • Add menus • Complicates the interaction of the app • Tends to lead to desktop style designs which do not work in the mobile world • Nested menus can cause users to become lost in the structure, and makes finding features even more difficult • Contextual menus require users to see when and where they appear

10. COMMON DESIGN APPROACHES AND THE PROBLEMS WE FOUND • Add vibration • Used too often it will lose its meaning • Causes user’s fingers to go numb • Speak commands • Not an ideal computer interface in a classroom, nor frankly, any public location • Computer comprehension of speech is not good enough to work on its own. When it fails we fall back to on-screen controls, which a blind person might easily miss.

11. BETTER DESIGN CHOICES • Create two completely separate interfaces, one for use with the screen reader, and another for use without. This can drastically reduce the need to compromise on the design. It also simplifies coding by reducing the need to constantly test the environment and select the appropriate display or action based on the state of the screen reader. • Always test with screen off! • Larger controls and much more space between controls than used with designs for the sighted • Reduce colour, increase contrast • Reduce the amount of information read by the screen reader. If possible, add beginner and pro modes that change the amount of information read.

12. BETTER DESIGN CHOICES • Do everything you can to speed the user’s interaction with the app. Remove steps whenever possible. • Add sound, but carefully • Best used with headphones • Map sounds to 3D space (we use OpenAL) • Use sparingly • Don’t make user learn meanings of random sounds. Sounds should convey meaning and match the physical world as much as is practical. • Add vibration, but sparingly

13. BETTER DESIGN CHOICES • Invert the position of important items. Layouts for the sighted often place important items on the top of the view, but they should be at the bottom for the blind. • Contextual menu display must be optimised to reduce the distance from the control that initiated it. This can cause issues when scrolling leaves a control near the edge of the screen. • Consider adding unique actions to the interface for the blind user. For instance, you might relocate the currently selected item so that it is always where the user taps if they tap an empty part of the screen. • Sleeves which cover the bezel are very useful.

14. INTERESTING APPROACHES THAT SURPRISED US BY WORKING • Drawing - Used within a constrained environment, ‘drawing’ elements onto the screen works well, but introduces issues if the drawn item does not exactly match the finger movements. • Scrolling - When paired with other elements like sound, user-driven scrolling can be effective. However, you must ensure that scrolling ends with complete elements still on the screen.

15. ISSUES WE ENCOUNTERED • There are some serious issues with VoiceOver • An item always selected, there is no ability to have nothing selected, which means that something will always respond. This is not the behaviour when VO is off • You cannot locally disable accessibility. Often things we needed to do within our apps would have been much easier to do ourselves rather than try to work within the constraints of VO • VO intercepts every user interaction before passing it to the app. This makes overriding default gestures impossible, even when they don’t make sense.

16. BENEFITS OF THE COURSES • Educational Community • Heartland Community College • Students • Employers