Tap into Mobile App Testing

2. Tap into Mobile App Testing Andreea Haras 11/09/2013

3. Agenda • Introduction • Mobile Testing vs. Traditional Web-Based Testing • Challenges in Mobile Testing • Guidelines & Best Practices • Conclusion • Q&A

4. About me • Andreea Haras – andreea.haras@gmail.com • 4+ years experience as software tester • Mobile testing enthusiastic • Interested in new technology

5. Why mobile testing?

6. Why mobile testing? • “The mobile market is no longer just communication. It’s internet, music, video, web, social networks, navigation and everything else... The mobile ecosystem now embraces anything a consumer wishes to do either on the move or on a handheld device at home” - Forrester

7. Why mobile testing? • “The mobile market is no longer just communication. It’s internet, music, video, web, social networks, navigation and everything else... The mobile ecosystem now embraces anything a consumer wishes to do either on the move or on a handheld device at home” - Forrester • Applications running on mobile devices (e.g., mobile applications running on smart phones or new generation tablets) are becoming so popular that they are representing a revolution in the IT sector.

8. Why mobile testing? • “The mobile market is no longer just communication. It’s internet, music, video, web, social networks, navigation and everything else... The mobile ecosystem now embraces anything a consumer wishes to do either on the move or on a handheld device at home” - Forrester • Applications running on mobile devices (e.g., mobile applications running on smart phones or new generation tablets) are becoming so popular that they are representing a revolution in the IT sector. • Over 1.6 million mobile applications have been developed, available in Apple App Store and Google Play (www.mobithinking.com)

9. Why mobile testing? • “The mobile market is no longer just communication. It’s internet, music, video, web, social networks, navigation and everything else... The mobile ecosystem now embraces anything a consumer wishes to do either on the move or on a handheld device at home” - Forrester • Applications running on mobile devices (e.g., mobile applications running on smart phones or new generation tablets) are becoming so popular that they are representing a revolution in the IT sector. • Over 1.6 million mobile applications have been developed, available in Apple App Store and Google Play (www.mobithinking.com) • Estimates for downloads of apps in 2013 range from 56 to 82 billion (www.idc.com) and a prediction of 200billion global downloads in 2017 that will be worth US$63.5 billion (www.portioresearch.com)

10. Native, Traditional, Hybrid, Oh My! • Native Apps: coded with a specific programming language. They are fast, reliable, and powerful, but are tied to a mobile platform and downloaded from a central portal by OS (i.e. Apple App Store, Google Play). That means you must duplicate them using the appropriate programming language in order to target another mobile platform. Netflix, Pandora, Shazam.

11. Native, Traditional, Hybrid, Oh My! • Native Apps: coded with a specific programming language. They are fast, reliable, and powerful, but are tied to a mobile platform and downloaded from a central portal by OS (i.e. Apple App Store, Google Play). That means you must duplicate them using the appropriate programming language in order to target another mobile platform. Netflix, Pandora, Shazam. • Traditional Mobile Web Apps: mobile web sites designed to match every web-enabled phone. They’re accessed through the mobile device’s web browser and do not need to be downloaded/installed. They re-direct to the site’s M Site. Amazon, eBay, MSN are good examples.

12. Native, Traditional, Hybrid, Oh My! • Native Apps: coded with a specific programming language. They are fast, reliable, and powerful, but are tied to a mobile platform and downloaded from a central portal by OS (i.e. Apple App Store, Google Play). That means you must duplicate them using the appropriate programming language in order to target another mobile platform. Netflix, Pandora, Shazam. • Traditional Mobile Web Apps: mobile web sites designed to match every web-enabled phone. They’re accessed through the mobile device’s web browser and do not need to be downloaded/installed. They re-direct to the site’s M Site. Amazon, eBay, MSN are good examples. • Hybrid Apps: rely on development frameworks and are written with the same technology used for websites and mobile web implementations. They are hosted or run inside a native container on a mobile device. They ensure cross-platform compatibility and can access the phone’s hardware (camera, GPS, user’s contacts, etc.). Foursquare, Twitter, Wikipedia and Facebook are good examples.

13. Pop Quiz There are seven (yes, seven!) mobile operating systems that are recognized in the industry, other than the top 2 (iOS and Android), can you name one or more of the other five?

15. Can you name the differences between Mobile Testing and Traditional Web-Based Testing?

16. Mobile Testing vs. Traditional Web-Based Testing • Integration testing • Within areas of the Device Hardware • Battery, light, ringer, sleep, wake, usage, connectivity loss or change in speeds (“elevator test”), camera • Integration with other Apps on the device • Facebook, Twitter, etc.

17. Mobile Testing vs. Traditional Web-Based Testing • Integration testing • Within areas of the Device Hardware • Battery, light, ringer, sleep, wake, usage, connectivity loss or change in speeds (“elevator test”), camera • Integration with other Apps on the device • Facebook, Twitter, etc. • Interruption Testing • How the App behaves with user-controlled and automatic interruptions

18. Mobile Testing vs. Traditional Web-Based Testing • Integration testing • Within areas of the Device Hardware • Battery, light, ringer, sleep, wake, usage, connectivity loss or change in speeds (“elevator test”), camera • Integration with other Apps on the device • Facebook, Twitter, etc. • Interruption Testing • How the App behaves with user-controlled and automatic interruptions • Device-specific Gesture Testing • Pinching, Zooming, Shaking, Orientation, Eye Scrolling, Eye Pausing, Tilting, Swiping, Press and Hold, Tapping

20. Pop quiz What mobile testing challenges do you think there are?

21. Challenges in Mobile Testing • The time frames • Variety of Devices & OS Versions • Display density • Different browsers • Network connections • Installation issues

22. Challenges in Mobile Testing • The time frames • Variety of Devices & OS Versions • Display density • Different browsers • Network connections • Installation issues • Simulating real life scenarios • Call interrupts • Low battery • Alarms • Network issues

23. More challenges in Mobile Testing • Usability • Accessibility • Intuitive • Mobile experience

24. More challenges in Mobile Testing • Usability • Accessibility • Intuitive • Mobile experience • Automation • What to automate? • Tools • Effort vs Value

25. More challenges in Mobile Testing • Usability • Accessibility • Intuitive • Mobile experience • Automation • What to automate? • Tools • Effort vs Value • Non functional testing • Performance • Security • Load testing

26. Guidelines & Best Practices • Determining your Mobile Test Matrix • Setting up your Lab: Phones • Setting up your Lab: Tablets • Checklist • Key types of Mobile QA Testing: Manual or Automated Testing? • Things to remember • Test Automation Tool Selection – Few factors to consider

27. Determining your Mobile Test Matrix The most difficult facet of the testing matrix is device fragmentation. There are nearly 4000 separate Android device models! How do you make sure that all users get a consistent experience? Device manufacturer market share (Q1 2013)

28. Determining your Mobile Test Matrix • Android device fragmentation • Source: BGR

29. Determining your Mobile Test Matrix Multiple approaches to build a “Mobile Test Matrix” • Group devices based on the hardware and/or software specification and prioritize them • User base – Build a test matrix based on the device and software version the user uses • Use statistics from vendor market place (especially useful for native apps like games, utilities, etc.) • Risk based and functionality based matrix

30. Determining your Mobile Test Matrix Multiple approaches to build a “Mobile Test Matrix” • Group devices based on the hardware and/or software specification and prioritize them • User base – Build a test matrix based on the device and software version the user uses • Use statistics from vendor market place (especially useful for native apps like games, utilities, etc.) • Risk based and functionality based matrix Example: • Group 1, priority C: Small devices with a small CPU, RAM and low resolution. Older software versions and older browsers. • Group 2, priority B: Mid-range devices with an avg. CPU, RAM (<512 MB), good screen size and resolution. The software is not the latest. • Group 3, priority A: High-end devices with a dual/quad-core CPU, RAM (>512 MB) and a high screen resolution. Latest software versions.

31. Setting up your Lab: Phones • Newest, highest resolution, high density, normal sized phone running latest version of OS. (Samsung Galaxy S4, iPhone 5S) • Oldest, smallest, low density small sized phone running oldest supported version of OS. (iPhone 3, HTC Tattoo) • Middle of the range phone matches highest usage, normal size, high density, normal resolution, highest usage OS (Samsung Galaxy S2) • Phone with Highest Market Share (Samsung Galaxy S3, iPhone 4S) • Budget smartphone - normal size, large screen, mdpi, old OS (Dell Venue)

32. Setting up your Lab: Tablets • XLargesize high end tablet (10”), xhdpi, latest OS. (Nexus 10) • Large size Tablet (5-7”), med density, legacy OS (Kindle Fire) • Phablet(Phone/Tablet Hybrid) (5”) xhdpi, large size, latest OS, strange aspect ratio (LG Intuition, Samsung Note) • iOSrunning iOS 5 or lower (iPad 2) • iOSrunning iOS 6 (iPadmini, iPad 3)

33. Checklist

34. Key types of Mobile QA Testing: Manual or Automated Testing? Functional testing Performance testing  Memory Leakage Testing  Interrupt testing Usability testing Installation testing Upgrade testing

35. Things to remember • Start testing early. Use emulators / simulators • Use the app on different carriers and network connections like 3G, 4G, WiFi, or LTE • Combine testing tools and integrate them into a continuous integration system • When possible, have a group of BETA testers to help with real-world testing • Consider the problem space: go mobile to test mobile • Don’t assume that because it works on one device, it will work on all

36. Test Automation Tool Selection – Few factors to consider • Supported devices • Emulator support • Availability of integrated simulators • Types of connection supported • Multi-device support (this is important to run same test on multiple devices) • Compatibility with existing automation tools • Security - VPN connections, data sharing, etc. • Functionality supported - swipe, drag and drop, zoom, scrolling, pinching, rotating, etc • Image / text recognition • Debugging and test reporting

37. Conclusions • Despite the challenges in mobile application testing, careful selection of target devices, using statistical techniques for test optimization and tools that maximize automation can ensure a cost effective mobile testing process.

38. Conclusions • Despite the challenges in mobile application testing, careful selection of target devices, using statistical techniques for test optimization and tools that maximize automation can ensure a cost effective mobile testing process. • Given the stiff competition in the mobile apps market and variety of apps available for users to download, it is vital that you thoroughly test each and every aspect of the app before releasing it into the market, to ensure that the product meets the quality standard demanded by users.