Just in time testing
Download
1 / 44

Just-In-Time Testing - PowerPoint PPT Presentation


  • 105 Views
  • Updated On :

Just-In-Time Testing. Robert Sabourin President AmiBug.Com, Inc. Montreal, Canada [email protected] Overview. Some Philosophy Context Drivers Turbulence Skills Testing Ideas Test Triage. Just In Time Testing. Robert Sabourin , Software Evangelist President AmiBug.Com Inc.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Just-In-Time Testing' - everly


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Just in time testing

Just-In-Time Testing

Robert Sabourin

President

AmiBug.Com, Inc.

Montreal, Canada

[email protected]

© Robert Sabourin, 2008


Overview
Overview

Some Philosophy

Context Drivers

Turbulence

Skills

Testing Ideas

Test Triage

© Robert Sabourin, 2008


Just in time testing1
Just In Time Testing

  • Robert Sabourin , Software Evangelist

  • President

  • AmiBug.Com Inc.

  • Montreal, Quebec, Canada

  • [email protected]

© Robert Sabourin, 2008


Fundamental question
Fundamental Question

  • How do you know when you are finished?

Some Philosophy

© Robert Sabourin, 2008


Gerald m weinberg
Gerald M. Weinberg

“Quality is value to some person”

Exploring Requirements Quality Before Design

Dorset House

Some Philosophy

© Robert Sabourin, 2008


Edsger w dijkstra
Edsger W. Dijkstra

“Program testing can be used to show the presence of bugs, but never to show their absence”

Some Philosophy

© Robert Sabourin, 2008


C northcote parkinson
C. Northcote Parkinson

Parkinson’s Law:

“…work expands so as to fill the time available for its completion…”

Some Philosophy

© Robert Sabourin, 2008


Steve Covey

“…begin with the end in mind …

“…first things first …"

Some Philosophy

© Robert Sabourin, 2008


Purpose of testing
Purpose of Testing

Common definition:

To find bugs before our customers do!

Broader definition:

The role of testing is to provide objective input to facilitate business decisions!

Keeps stakeholders aware of all issues or concerns that relate to shipping a product!

Some Philosophy

© Robert Sabourin, 2008


Jit core principals
JIT Core Principals

  • RRR – Keep it Real Recent & Relevant

  • Purposeful Testing – On Time On Quality On Budget are meaningless if you are not on Purpose

  • Fundamental Question of Software Engineering: How do you know you are finished?

  • First Things First: Begin with the end in mind.

  • BTO - Context Drivers: Business Technology Organization

  • Be Prepared – Be ready for anything

  • Testing is Intractable: Tests cannot show software is bug free

  • Collect new test Ideas from many sources always

  • Triage Ruthlessly

  • Decide what not to test on purpose

  • Bugs are concerns about the project

  • There is no try: Do or Do Not

  • It’s all about people – and the occasional bug!

Core Values

© Robert Sabourin, 2008


Context drivers bto
Context Drivers - BTO

  • Business

    • Value

    • To whom?

    • Why?

  • Technology

    • Solutions

  • Organization

    • Corporate Structure

    • Team Structure

    • Roles and Responsibilities

Context

© Robert Sabourin, 2008


Context

© Robert Sabourin, 2008


Just in time testing2
Just-In-Time Testing

Turbulence

Turbulence

© Robert Sabourin, 2008


Just in time testing3
Just-In-Time Testing

Unprepared

Test Skills

© Robert Sabourin, 2008


Just in time testing4
Just-In-Time Testing

Sharpen Testing Skills

Thinker

Detective

Reporter

Diplomat

Negotiator

Cheer Leader

Pragmatist

Test Skills

© Robert Sabourin, 2008


Just in time testing5
Just-In-Time Testing

So what exactly did they throw over the wall?

Test Focus

© Robert Sabourin, 2008


First things first
First Things First

  • Begin with the end in mind

    • Gain Consensus

      • Goals

        • How do we know we are finished?

      • Purpose

        • Why are we doing this project?

        • How will be react to change?

      • Meaning

        • What is a bug?

        • What is a test?

Test Focus

© Robert Sabourin, 2008


Be prepared
Be Prepared

We have precious little time to run tests!

We must always be prepared!

Test Triage

© Robert Sabourin, 2008


Time

Test Triage

© Robert Sabourin, 2008


Yoda

"No! Try not, Do. Or do not.

There is no try."

Test Triage

© Robert Sabourin, 2008


Testing ideas
Testing Ideas

  • Collect all testing ideas you can find!

    • List

    • Sort

    • Organize

    • Shuffle

Test Triage

© Robert Sabourin, 2008


Testing ideas1
Testing Ideas

  • How to find them?

    • Does system do what it is suppose to do?

    • Does the system do things it is not supposed to?

    • How can the system break?

    • How does the system react to it’s environment?

    • What characteristics must the system have?

    • Why have similar systems failed?

    • How have previous projects failed?

Test Triage

© Robert Sabourin, 2008


Testing ideas2
Testing Ideas

  • Collect testing ideas

  • From testing ideas build a series of testing objectives

    • Each can be assigned as work to testers

    • Each can include all, part of, or multiple testing ideas

Capture testing ideas

© Robert Sabourin, 2008


Testing ideas3
Testing Ideas

  • I often use Index Cards

    • Unique id

    • One testing idea per card

    • Colour indicates source

    • Shuffled and reviewed

    • Organized and reorganized

    • Sorted, grouped, prioritized and collected

Capture testing ideas

© Robert Sabourin, 2008


Test idea sources
Test Idea Sources

  • Capabilities

  • Failure Modes

  • Quality Factors

  • Usage Scenarios

  • Creative Ideas

  • States

  • Data

  • Environments

  • White Box

  • Taxonomies

Capture testing ideas

© Robert Sabourin, 2008


Testing ideas4
Testing Ideas

  • Investigative approaches

    • We become truffle snorting pigs and try to find useful information in all evidence we discover

    • We can even get good ideas from out of date sources

Capture testing ideas

© Robert Sabourin, 2008


Testing ideas5
Testing Ideas

  • Requirements

    • Use cases

    • Functional requirements

    • Quality factors

    • Constraints

    • Written requirements

    • Implicit requirements

Capture testing ideas

© Robert Sabourin, 2008


Testing ideas6
Testing Ideas

  • Failure Modes

    • What can break?

    • Reaction to invalid input?

    • How does software behave in constrained environment?

      • Memory

      • Disk Space

      • Network Bandwidth

      • CPU capacity

      • Shared resources

    • Stress, Load, Volume

Capture testing ideas

© Robert Sabourin, 2008


Capture testing ideas

Quality Factors Importance

Different Application Types

© Robert Sabourin, 2008


Testing ideas7
Testing Ideas

  • Usage Scenarios

    • Identify classes of users

    • Identify how users will use system

    • Describe scenarios

    • Use Story board or similar approaches

    • Identify variations

Capture testing ideas

© Robert Sabourin, 2008


Testing ideas8
Testing Ideas

  • Creative approaches

    • Action verbs

    • Mind Maps

    • Soap Operas

    • Lateral Thinking

Capture testing ideas

© Robert Sabourin, 2008


State models
State Models

Testing Ideas

power up

service

needed

idle

inserting

coins

reset button

coin inserted

coin return

cup removed

no cups

OR no coffee

OR sensor jam

right amount

entered

coin return

make

coffee

user

choose

Capture testing ideas

button pushed

© Robert Sabourin, 2008


Testing ideas9
Testing Ideas

  • Data

    • Flow

    • Structure

    • Create

    • Update

    • Change

Capture testing ideas

© Robert Sabourin, 2008


Testing ideas10
Testing Ideas

  • Environment

    • Hardware

    • Software

    • Operating systems

    • Locales

    • Browsers

    • Plug-ins

    • Co-dependent software

Capture testing ideas

© Robert Sabourin, 2008


Testing ideas11
Testing Ideas

  • White Box

    • Design

    • Internal structure

    • Code

Capture testing ideas

© Robert Sabourin, 2008


Testing ideas12
Testing Ideas

  • Bug taxonomies

    • Collections of possible bugs

    • Appendix A of Testing Computer Software, Kaner, Falk, Nguyen

    • Boris Biezer Taxonomy Otto Vinter manages

    • Shopping cart taxonomy Giri Vijayaraghavan

Capture testing ideas

© Robert Sabourin, 2008


Which test
Which test?

  • Impact estimation

    • For each test idea guesstimate:

      • benefit of implementation

      • consequence of implementation

      • benefit for not implementing

      • consequence of not implementing

    • How credible is the information?

Triage testing ideas

© Robert Sabourin, 2008


Which test1
Which test?

Test Idea Rejection – What If?

  • If the cost/benefit does not make business sense then consider implementing:

    • part of the test, could that lead to part of the benefit at a more reasonable cost?

    • more than the stated test, would that generate more benefit?

    • a different test than the stated idea, could that generate more benefit for less cost?

Triage testing ideas

© Robert Sabourin, 2008


Which test is next
Which test is next?

  • Questions

    • Given state of project, state of business, state of technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next?

    • How much effort are we willing to spend continuing to test this project?

    • Can we ship yet?

Triage testing ideas

© Robert Sabourin, 2008


Which test is next1
Which test is next?

  • Magic crystal ball

    • If it existed how would you use it?

    • What question would you ask?

    • What question would it ask?

Triage testing ideas

© Robert Sabourin, 2008


Deciding what not to test
Deciding what not to test?

  • Time pressure

    • Should we skip a test?

    • If test failed could system still be of value to some stakeholder?

    • If test was skipped could important bugs have been otherwise found?

Triage testing ideas

© Robert Sabourin, 2008


Test triage
Test Triage

  • Test Triage Meeting

    • Review Context

      • Business

      • Technical

    • Information since last triage

      • Test results

      • Bug results

      • New testing ideas

Triage testing ideas

© Robert Sabourin, 2008


Test triage1
Test Triage

  • Allocate Testing Assignments to Testers

    • Make sure testers know context

    • Best thing to test

    • Best person to test it

    • Best people to explore it

    • Best lead

    • Are subject matter experts required?

    • Blend scripted & exploratory testing

Triage testing ideas

© Robert Sabourin, 2008


Bottom line
Bottom Line

  • My experience is that it is better to omit a test on purpose than to skip it because you ran out of time or forgot about it!

  • Systematically collecting, evaluating and triaging testing ideas helps me decide what not to test - at least for now?

Get Started Right

© Robert Sabourin, 2008


ad