Create Your Roadmap to Data-Driven Quality. Seth Eliot Principal Knowledge Engineer, Test Excellence April 15, 2014. About Seth. Digital Media Services. A/B Testing of Services. Petabytes Processed. Services and Cloud and Data-Driven Engineering and.
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Principal Knowledge Engineer, Test Excellence
April 15, 2014
Digital Media Services
A/B Testing of Services
Data-Driven Engineering and...
Road, Car, Gas
Application (service or product)
You and your...
What is it? Why is it important?
“Big Data” is hot, let’s start with that more frogs and kiss them faster and faster” - Mike Moran
“Big Data” is hot, let’s start with that
GQM - 1994
Start at the beginning
What do you need to know about quality?
Quinn, et al, 1999
Does sunscreen increase chance of drowning?
“Downtime” is defined as any period of time when users are unable to send or receive email via all supported mailbox access
Is the application (product, service) there for the user?
Critical work stoppage for user
Power of Production Data
End to end
Customer Experience Improvement Program (CEIP)
Get data from or near production
Real-time service quality
1B API requests per day
Usually easy for services
Client apps may have a deployment capability
Or may make use of feature flags
Bucket- more frogs and kiss them faster and faster” - Mike Moranized data is aggregated and uses less bandwidth
Determine the data necessary to answer your questions
Infrastructure data more frogs and kiss them faster and faster” - Mike Moran
New and Open
Feedback data more frogs and kiss them faster and faster” - Mike Moran
Customer Support Data
Social Media Mining
Data Center X
Data Center Y
Again services have it easy
Many clients are always connected
On prem servers (enterprise) require partnerships
Depends on Type
personally identifiable info (PII)
Depends on Purpose
provide the service
improve the current service
improve a future version service
improve non-associated services
User owned resources: bandwidth
disk, cpu, etc…
end-to-end across clients and services
by user , by session
A 50,000 foot view
HDFS – Hadoop Distributed File System
Power BI in Excel
Real-time monitoring more frogs and kiss them faster and faster” - Mike Moran
Web Apps integrated with OneDrive, FB, web mail, etc
Feedback more frogs and kiss them faster and faster” - Mike Moran
and learn new questions
View Inbox – Page Load Time (PLT) by Browser
As experienced by actual users
500 Million measurements per month
Availability more frogs and kiss them faster and faster” - Mike Moran
Predict 75% of dips 24 hours ahead of time
Netflix prioritizes perceived availability & performance more frogs and kiss them faster and faster” - Mike Moran
Yammer prioritizes more frogs and kiss them faster and faster” - Mike Moranusage
What happens to new user retention when you shorten the signup flow?
It goes down!
Don’t ship that feature
We also added Telemetry
We can see how it works
In Visual Studio 2012 we added asynchronous loading for solutions
The most recent version of this deck can be found at http://setheliot.com