Mobile Testing: Brought to You by the Letters A & B

#GHC13 Mobile Testing:Brought to You by the Letters A & B Leslie Nguyen October 3, 2013 2013

Presenter: Leslie Nguyen • BS/MS of Computer Science from USC • iOS developer • Cisco • Groupon

Guess which generated more signups? Variation B with image Variation A with video

What does 40.6% mean for 10 million sign ups? • Improvement of 40.6%  2,880,000 more emails • 10% converted to volunteers  additional 288,000 volunteers • Each email tied to an average of $21  additional $60 million in donations

What is A/B Testing?

Show of Hands • Who works on web stuff?

Show of Hands • Who works on web stuff? • Keep your hand up if you have performed A/B testing on the web

Show of Hands • Who works on mobile apps?

Show of Hands • Who works on mobile apps? • Keep your hand up if you have performed A/B testing for these apps

Why Should You Focus on Mobile? These are the 2013 Q2 percentages of revenue from mobile:

Challenges with A/B Testing for Mobile • Long release cycle • Latency is high • What happens when the app is offline?

A/B Testing Made Easy App Server Device ID Bucket Users Configuration Manager A/B Testing Profile Renders UI Analytics Conversion Calculations Stop Test

Problems – First Launch • A/B testing profile downloads asynchronously at launch • What about first launch? • What about A/B testing the first screen?

Solution • Cache the testing profile • Bundle the app with a testing profile

Goal Tracking & Analytics • Client sends analytics of events to the server • Server makes sense of the analytics • Server determines which variation is winning and how long to run the test

What happens if the app is offline?

Solution – Storing Analytics • Client stores results in local phone database • Client uploads the info when the user goes back online

Consistency • It’s important for a user to have a consistent UI experience • Once a user is bucketed, they should remain there for that session • On mobile, the session can last until the end of the test or until the app is updated

What about the slow release cycle?

Solution – Parameterized Tests • Strings or button colors can be wrapped with the configuration manager • This enables values to be configured remotely by server • Elements can be configured at a future time

Strings and Button Colors [ABTest test:@”login” data:^(NSDictionary *data) { button.title = [data objectForKey:@”title”]; button.backgroundColor = [data objectForKey@”color”]; }];

How do you test a UI Layout:Tab Bar vsCarousel?

Solution – Runtime Class Instantiation Class viewClass = [ABTestclassForTest@”Carousel”]; UIViewController *vc = [[viewClassalloc] init];

Conclusions • A/B testing is awesome! • Mobile on the rise • Data driven decisions means better conversions  more revenue • It’s easy to get started

Got Questions? leslie@box.com

Got Feedback? Rate and Review the session using the GHC Mobile App To download visit www.gracehopper.org

Mobile Testing: Brought to You by the Letters A & B