1 / 16

Experimentation Platforms: Build or Buy?

Experimentation Platforms: Build or Buy?. RecSys 2014. Mike Lo, PayPal. Hello!. Product Test Ops Merchant Testing Knowledge Mgt and Insights Experimentation Analytics. Mike Lo, Sr. Manager - Experimentation, PayPal. PayPal. Experimentation Program.

devlin
Download Presentation

Experimentation Platforms: Build or Buy?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Experimentation Platforms: Build or Buy? • RecSys 2014 • Mike Lo, PayPal

  2. Hello! Product Test Ops Merchant Testing Knowledge Mgt and Insights Experimentation Analytics Mike Lo, Sr. Manager - Experimentation, PayPal PayPal Experimentation Program Manage PayPal’s Experimentation Program Formerly part of eBay’s Program Build or Buy? We did both at PayPal Expectations for today’s talk Focus on Experimentation Platform PayPal’s ‘Testing Story’ Some comparison with eBay program.

  3. Words words words Beta Statistics Ramp ANALYTICS Mobile Agile Big DATA Alpha Random Insights

  4. Becoming Data-Driven is difficult! It doesn’t happen overnight. It took us years.

  5. We bought.

  6. PayPal = eBay right? What about the existing platform? Technology Adoption eBay-focused Rationale why we didn’t implement the eBay Experimentation Platform PayPal’s tech stack Fix Tracking Can’t wait Quick wins Endorsement Program, not the Product. Built ground up for eBay Tightly coupled with eBay systems Need to decouple features

  7. Outcome? Ran experiments successfully for 12 months… then … Couldn’t Scale Had to cut the cord Result: Implemented eBay Platform

  8. So are you advocating to build rather than buy? No. It’s all about the TESTING NEEDS of your organization.

  9. What happened? • Product Pages • Marketing Pages • Testing different content variants • Testing different feature variants No Product Tests Keep it separate? PayPal’s Needs Like many companies, the need started from Marketing but didn’t end there. Product Pages • Technical Limitations • Is there a clean distinction? • 20 experiments is nice. But how do we get to 2,000?

  10. Why it made sense for PayPal to build instead of buy 3 main reasons PayPal is investing to build 1 Analytical 2 Technical 3 Scale

  11. 1 - High Fidelity Analytics Other sites have natural ‘fences’ Our site is an ecosystem Single Enterprise Testing Solution Scientific rigor in the results

  12. 2 - Technical agility Agile Product Lifecycle Who does your dev? Look beyond It’s all about going Agile! PayPal’s tech stack Master of your domain Data Site speed performance Rolling Full-Bright Coupled with PDLC Code once, push in minutes Allows MVT

  13. 3- Scaling to the next level Volume Sophistication Deeper Insights Utilizing the platform to its fullest potential Thousands of experiments Patented mechanism Utilizing PayPal’s platforms Entire data pipeline Automated design Real-time analytics Data modeling

  14. Beware • 2 potential unforeseen effects of building a robust platform, if left un-checked… • Platform is reduced to a CMS tool • Ramping / Throttling / Release tool

  15. If you build it, will they come? Again, it is not just about the Product… follow through with Adoption!

  16. Questions? • Thanks for listening!

More Related