160 likes | 307 Views
Experimentation Platforms: Build or Buy?. RecSys 2014. Mike Lo, PayPal. Hello!. Product Test Ops Merchant Testing Knowledge Mgt and Insights Experimentation Analytics. Mike Lo, Sr. Manager - Experimentation, PayPal. PayPal. Experimentation Program.
E N D
Experimentation Platforms: Build or Buy? • RecSys 2014 • Mike Lo, PayPal
Hello! Product Test Ops Merchant Testing Knowledge Mgt and Insights Experimentation Analytics Mike Lo, Sr. Manager - Experimentation, PayPal PayPal Experimentation Program Manage PayPal’s Experimentation Program Formerly part of eBay’s Program Build or Buy? We did both at PayPal Expectations for today’s talk Focus on Experimentation Platform PayPal’s ‘Testing Story’ Some comparison with eBay program.
Words words words Beta Statistics Ramp ANALYTICS Mobile Agile Big DATA Alpha Random Insights
Becoming Data-Driven is difficult! It doesn’t happen overnight. It took us years.
PayPal = eBay right? What about the existing platform? Technology Adoption eBay-focused Rationale why we didn’t implement the eBay Experimentation Platform PayPal’s tech stack Fix Tracking Can’t wait Quick wins Endorsement Program, not the Product. Built ground up for eBay Tightly coupled with eBay systems Need to decouple features
Outcome? Ran experiments successfully for 12 months… then … Couldn’t Scale Had to cut the cord Result: Implemented eBay Platform
So are you advocating to build rather than buy? No. It’s all about the TESTING NEEDS of your organization.
What happened? • Product Pages • Marketing Pages • Testing different content variants • Testing different feature variants No Product Tests Keep it separate? PayPal’s Needs Like many companies, the need started from Marketing but didn’t end there. Product Pages • Technical Limitations • Is there a clean distinction? • 20 experiments is nice. But how do we get to 2,000?
Why it made sense for PayPal to build instead of buy 3 main reasons PayPal is investing to build 1 Analytical 2 Technical 3 Scale
1 - High Fidelity Analytics Other sites have natural ‘fences’ Our site is an ecosystem Single Enterprise Testing Solution Scientific rigor in the results
2 - Technical agility Agile Product Lifecycle Who does your dev? Look beyond It’s all about going Agile! PayPal’s tech stack Master of your domain Data Site speed performance Rolling Full-Bright Coupled with PDLC Code once, push in minutes Allows MVT
3- Scaling to the next level Volume Sophistication Deeper Insights Utilizing the platform to its fullest potential Thousands of experiments Patented mechanism Utilizing PayPal’s platforms Entire data pipeline Automated design Real-time analytics Data modeling
Beware • 2 potential unforeseen effects of building a robust platform, if left un-checked… • Platform is reduced to a CMS tool • Ramping / Throttling / Release tool
If you build it, will they come? Again, it is not just about the Product… follow through with Adoption!
Questions? • Thanks for listening!