Create your roadmap to data driven quality
Download
1 / 61

Create Your Roadmap to Data-Driven Quality - PowerPoint PPT Presentation


  • 100 Views
  • Uploaded on

Create Your Roadmap to Data-Driven Quality. Seth Eliot Principal Knowledge Engineer, Test Excellence April 15, 2014. About Seth. Digital Media Services. A/B Testing of Services. Petabytes Processed. Services and Cloud and Data-Driven Engineering and.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Create Your Roadmap to Data-Driven Quality' - brittany-price


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Create your roadmap to data driven quality

Create Your Roadmap to Data-Driven Quality

Seth Eliot

Principal Knowledge Engineer, Test Excellence

April 15, 2014


About seth
About Seth

Digital Media Services

A/B Testing of Services

Petabytes Processed

Services and

Cloud and

Data-Driven Engineering and...


…how to create your “roadmap” to DDQ

At destination

Roadmap

Roadmap

Road, Car, Gas

Application (service or product)

Environment

Engineering Processes

DDQ Strategy

You and your...


Data driven quality

Data-Driven Quality

What is it? Why is it important?


Data where have we been
Data, where have we been?

  • The HiPPO

  • Highest Paid Person’s Opinion

  • Engineering Data

    • Test pass/fail results

    • Bug counts

    • Delivery cadence

    • Code coverage

    • Code Churn


Data where are we going
Data, where are we going?

Real Users

  • Engineering data

    • Test results  Scoring engines using Bayesian analysis

  • Production-quality data

    • You may have heard of TiP

    • Not as difficult as you might think

      • Lots of solutions for lots of application types

Production Environments



Virtuous cycle of ddq
Virtuous cycle of DDQ more frogs and kiss them faster and faster” - Mike Moran


The roadmap
The roadmap more frogs and kiss them faster and faster” - Mike Moran

  • Determine your questions

  • Design for production-quality data

  • Select your data sources

  • Use the right data tools

  • Get answers to your questions

  • Learn new questions

  • Repeat


“Big Data” is hot, let’s start with that more frogs and kiss them faster and faster” - Mike Moran

“Big Data” is hot, let’s start with that

Performance

Availability

GQM - 1994

Start at the beginning

?

Usage


Determine your questions

Determine your questions more frogs and kiss them faster and faster” - Mike Moran

What do you need to know about quality?


Why not just get data and look for answers
Why not just get data and look for answers? more frogs and kiss them faster and faster” - Mike Moran

  • Do night-lights cause

  • near-sightedness in children?

Quinn, et al, 1999

Does sunscreen increase chance of drowning?


What questions does exo ask
What questions does EXO ask? more frogs and kiss them faster and faster” - Mike Moran

“Downtime” is defined as any period of time when users are unable to send or receive email via all supported mailbox access


Is it available
Is it available? more frogs and kiss them faster and faster” - Mike Moran

“Dialtone”

  • Often a Pri 0

Is the application (product, service) there for the user?

Critical work stoppage for user


How do users perceive availability
How do users perceive availability? more frogs and kiss them faster and faster” - Mike Moran

  • Application works, but feature does not

  • Occasionally does not render/load properly

  • 違った言語


How is performance
How is performance? more frogs and kiss them faster and faster” - Mike Moran

Huge Impact

Power of Production Data

Real users

Multiple environments

End to end

Scale &

geo-diversity


What do users do
What do users do? more frogs and kiss them faster and faster” - Mike Moran

Customer Experience Improvement Program (CEIP)


Your turn determine your questions
Your turn - more frogs and kiss them faster and faster” - Mike MoranDetermine your questions

  • What type of answer are you looking for?

    • Availability, performance, usage?

    • Prioritize

    • When does availability NOT come first?


Your turn determine your questions1
Your turn - more frogs and kiss them faster and faster” - Mike MoranDetermine your questions

  • What are the key scenarios for the type you selected?

SharePoint

  • What is high priority?


Design for production quality data

Design for production-quality data more frogs and kiss them faster and faster” - Mike Moran

Get data from or near production


Two types of data to acquire
Two types of data to acquire more frogs and kiss them faster and faster” - Mike Moran

  • Active: synthetic

  • Passive: real (RUM)

    • Active data in prod

    • For services only?

    • Client: is the service there?


Staged d ata acquisition mitigates risk
Staged more frogs and kiss them faster and faster” - Mike Morandata acquisition mitigates risk

  • Service

  • Product (client, on-prem server)

Deployment Validation

Service Validation

Real-time service quality

Scale Validation


Staged data acquisition netflix
Staged more frogs and kiss them faster and faster” - Mike Morandata acquisition - Netflix

1B API requests per day

Canary Deployment


Staged data acquisition facebook
Staged Data Acquisition - more frogs and kiss them faster and faster” - Mike MoranFacebook

  • Dogfood

  • In prod, no users (except internal ones)

  • Some servers in Production

  • World-wide deployment

  • Feature flags


Speed of deployment
Speed of deployment more frogs and kiss them faster and faster” - Mike Moran

Usually easy for services

Client apps may have a deployment capability

Or may make use of feature flags

  • EaaSy- Everything is now connected and thus updatable


D ata acquisition for clients
D more frogs and kiss them faster and faster” - Mike Moranata acquisition for clients

Outlook

  • Filtering and aggregation at client

    • Be kind to the client

    • Don’t abuse user resources:

    • Bandwidth

    • Battery

    • Disk


Bucket- more frogs and kiss them faster and faster” - Mike Moranized data is aggregated and uses less bandwidth


D ata acquisition for clients1
D more frogs and kiss them faster and faster” - Mike Moranata acquisition for clients

Outlook

  • Filtering and aggregation at client

    • Be kind to the client

  • Pipeline to collect and process data

    • Make it easy

  • Staged Data Acquisition

Scale Validation


Your turn design for production quality data
Your turn - more frogs and kiss them faster and faster” - Mike MoranDesign for production-quality data

  • What might be your stages for risk mitigated data acquisition?

  • Role of active and passive monitoring?

  • How can you engineer for EaaSy deployment?


Select your data sources

Select your data more frogs and kiss them faster and faster” - Mike Moransources

Determine the data necessary to answer your questions


Infrastructure data more frogs and kiss them faster and faster” - Mike Moran

CPU

Memory

Storage

Network


Application data
Application data more frogs and kiss them faster and faster” - Mike Moran

Failure Rate

Average Time

New and Open

Download Template


Hang and crash d ata
Hang and crash more frogs and kiss them faster and faster” - Mike Morandata

  • Specialized application data

    • Most frequently encountered conditions

    • Bucket the Big Data and find the offenders

    • Get offending code and function calls (stack)

  • But it worked in Dogfood

    • Culprit was a old version add-in

    • Harden against this


Usage data
Usage more frogs and kiss them faster and faster” - Mike Morandata

  • Client-side instrumentation

    • Proprietary

    • Javascript: clicks, hovers [web apps]

    • Get 1x1 GIF: Page Views [web apps]

  • Combine into more complex scenarios

    • How did user get to shopping cart checkout?


Feedback data more frogs and kiss them faster and faster” - Mike Moran


Feedback data
Feedback more frogs and kiss them faster and faster” - Mike Morandata


Feedback data also includes
Feedback more frogs and kiss them faster and faster” - Mike Morandata also includes…

Customer Support Data

Xbox Kinect

Social Media Mining


Active monitoring data
Active more frogs and kiss them faster and faster” - Mike Moranmonitoring data

Data Center X

DependencyService(s)

AM2

Data Center Y

AM3

Service

Client

AM1


Ease of detection
Ease of detection more frogs and kiss them faster and faster” - Mike Moran

Again services have it easy

Many clients are always connected

On prem servers (enterprise) require partnerships

  • EaaSy– Rich near real-time telemetry


Data handling privacy
Data handling - privacy more frogs and kiss them faster and faster” - Mike Moran

  • Transparency and Control

  • Collection and Retention

Depends on Type

anonymous Data

pseudonymous

personally identifiable info (PII)

sensitive PII

Depends on Purpose

provide the service

improve the current service

improve a future version service

improve non-associated services

content personalization

ad targeting


Data handling non service products
Data handling – non-service products more frogs and kiss them faster and faster” - Mike Moran

  • Client and on-prem server considerations

User owned resources: bandwidth

battery

disk, cpu, etc…

Correlations

end-to-end across clients and services

by user , by session


Your turn select your data sources
Your turn - more frogs and kiss them faster and faster” - Mike MoranSelect your data sources

  • Infrastructure data

  • Application data

  • Hang and crash data

  • Usage data

  • Feedback data

  • Active monitoring data

  • Design to handle your data


Use the right data tools

Use the right data more frogs and kiss them faster and faster” - Mike Morantools

A 50,000 foot view


Data storage and processing systems
Data storage and processing systems more frogs and kiss them faster and faster” - Mike Moran


Hadoop in 60 seconds
Hadoop in 60 Seconds more frogs and kiss them faster and faster” - Mike Moran

Hadoop

HDFS – Hadoop Distributed File System

Map-Reduce

ADD

FCA

ECC

1xA

1xC

1xF

2xA

0xB

3xC

2xD

1xE

1xF

1xA

2xD

ADD

FCA

ADDFCAECC

ADD

FCA

ADD

FCA



1xE

2xC

ECC

ECC

ECC


A common data flow
A common data flow more frogs and kiss them faster and faster” - Mike Moran

Power BI in Excel

Cube

SQL DW

Telemetry

System

Programmatic access

Other Visualizations

Real-time

Monitoring


Real-time monitoring more frogs and kiss them faster and faster” - Mike Moran

Web Apps integrated with OneDrive, FB, web mail, etc


Feedback more frogs and kiss them faster and faster” - Mike Moran

Xbox Kinect

Twitter


Your turn use the right data processing tools
Your turn - more frogs and kiss them faster and faster” - Mike MoranUse the right data processing tools

  • DB, DW, Cube, Big-Data platforms

    • Put it all together

  • Do you need real-time monitoring?

  • …sentiment analysis?


Get answers to your questions

Get answers to your questions more frogs and kiss them faster and faster” - Mike Moran

and learn new questions


Outlook com prioritizes performance
Outlook.com prioritizes performance more frogs and kiss them faster and faster” - Mike Moran

View Inbox – Page Load Time (PLT) by Browser

As experienced by actual users

JSI

JavaScript Instrumentation

500 Million measurements per month


Availability more frogs and kiss them faster and faster” - Mike Moran

Prioritizes availability

Predict 75% of dips 24 hours ahead of time

Time


Netflix prioritizes perceived availability & performance more frogs and kiss them faster and faster” - Mike Moran


Yammer prioritizes more frogs and kiss them faster and faster” - Mike Moranusage

What happens to new user retention when you shorten the signup flow?

It goes down!

Don’t ship that feature


Find new questions and repeat
Find new questions and repeat more frogs and kiss them faster and faster” - Mike Moran

Xbox recommends

Try algorithm

Collect data

Adjust algorithm

Collect data

Repeat


Find new questions and repeat1
Find new questions and repeat more frogs and kiss them faster and faster” - Mike Moran

We also added Telemetry

We can see how it works

In Visual Studio 2012 we added asynchronous loading for solutions


Your turn get answers find more questions
Your turn – more frogs and kiss them faster and faster” - Mike MoranGet answers; find more questions

  • Enjoy the rest of STPCon :-)

  • Get back to work

  • Implement your roadmap

  • Get answers to your quality assessment questions


  • Determine your questions more frogs and kiss them faster and faster” - Mike Moran

  • Design for production-quality data

  • Select your data sources

  • Use the right data tools

  • Get answers to your questions

  • Learn new questions

  • Repeat

The most recent version of this deck can be found at http://setheliot.com


Special thanks to these folks
Special thanks to these folks more frogs and kiss them faster and faster” - Mike Moran

David Brooks

Mike Tholfsen

Jodie Draper

Brian Mueller

Tara Roth

Dror Cohen

Nathan Halstead

Lori Oviatt

Monica Catunda

Lynette Skinner

Joe Schumacher

Donny Luu

John Hoegger

Alain Anyouzoa

  • Ravi Vedula

  • Andrea Jesse

  • Bill Hodghead

  • Danny Thayer

  • Joseph Sefair

  • Kitty Thomas

  • Amanda Reinke

  • Heather Lader

Any questions?

Thanks!


Please fill out the survey
Please fill out the survey more frogs and kiss them faster and faster” - Mike Moran

  • Create Your Roadmap to Data-Driven Quality

  • Session 101

  • Seth Eliot

?


ad