Test requirements the basis of testing
Download
1 / 48

- PowerPoint PPT Presentation


  • 401 Views
  • Updated On :

Test Requirements: The Basis of Testing. David Capocci, CQA, CSTE Sr. QA Systems Analyst SAFECO Corporation [email protected] Agenda. Defining Test Requirements (TR) What, Why, Where Fitting TR’s into the testing picture What’s within our testing process Generating TR’s

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about '' - iria


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Test requirements the basis of testing

David Capocci, CQA, CSTE

Test Requirements:The Basis of Testing

David Capocci, CQA, CSTE

Sr. QA Systems Analyst

SAFECO Corporation

[email protected]


Agenda

David Capocci, CQA, CSTE

Agenda

  • Defining Test Requirements (TR)

    • What, Why, Where

  • Fitting TR’s into the testing picture

    • What’s within our testing process

    • Generating TR’s

  • Organizing & Decomposing TR’s

  • Test Requirements Hierarchy Samples

  • Setting the stage for measuring test coverage


Background

David Capocci, CQA, CSTE

Background

  • “Test Requirements Hierarchy” is a term coined from Rational’s SQA Team Test software.

  • The principle of identifying, organizing, and measuring test requirements is universal to many test processes and methodologies

  • Much of this in-service is distilled from:

    • Rational methodologies (we are an SQA Team Test house after all)

    • QAI Workbench model

    • Seminar topics from

      • 19th annual International Software Testing Conference

      • Star ‘98 West Conference

    • Ed Kit’s “Software Testing in the Real World”


Test requirements

David Capocci, CQA, CSTE

Test Requirements

  • What exactly is a Test Requirement?

  • Why identify Test Requirements?

  • Where does a Test Requirement come from?

Defining TR’s: What, Why, Where


What exactly is a test requirement

David Capocci, CQA, CSTE

What exactly is a Test Requirement?

  • Identifies the “WHAT” of testing

    • What needs to be tested, AND

    • What are you going to validate about it

  • Includes both normal and error conditions

  • Covers business rules, functionality, non-functional standards

  • Do NOT have case specific data values assigned to them yet (data appears in test cases, the “How” of testing) examples…

  • Defining TR’s: What, Why, Where


    Example 1 testing the inserting of a record to a table

    David Capocci, CQA, CSTE

    “Validate that you can insert an entry”

    “Validate that insertion fails if entry already present”

    “Validate that insertion fails if table already full”

    “Validate that you can insert an entry to an empty table (initial)”

    These are test requirements NOT tests because they do not describe the data element being inserted

    The data is irrelevant at this level, it will appear in the test cases used to cover these test requirements

    “Validate you can insert ‘John Doe’” is a test case not a test requirement

    Example 1: Testing the inserting of a record to a table

    Test Requirements Identified (among others):

    Defining TR’s: What, Why, Where


    Why identify test requirements

    David Capocci, CQA, CSTE

    Why identify Test Requirements?

    • It’s what QC “owns” in our workbench: Requirements-based or Function-based testing

    • It’s the basis for establishing the completion of testing

    • Helps determine the scale of the testing effort

    • Governs the types of resources you will need

    • Serves to identify automation strategies you can use

    • Becomes a roadmap for your testing effort

    • Can be a tool for leverage and dialog with developers and business analysts

    • Dev Team can sign off on them (Verification!)

    Defining TR’s: What, Why, Where


    Where does a tr come from

    David Capocci, CQA, CSTE

    Where does a TR come from?

    • Traditional: Business Requirements, functionality, internal logic…

      • Marketing specs, Functional specs, Technical specs

    • Reality:

      • “Interview Analysis”, Non-Functional Checklists (standards & compliance), Use Cases (from business scenarios and users), Discovery during testing, any other deliverables from previous workbenches (diagrams, modeling, flowcharts, etc.)

    Defining TR’s: What, Why, Where


    How do test requirements relate to the test plan

    David Capocci, CQA, CSTE

    How do Test Requirements relate to the Test Plan?

    • Traditionally, the Test Plan has represented “what” was going to be tested, even including test cases.

    • Paradigm is shifting: Test Plan should relate what your testing process (and deliverables) will be for a particular project.

    • A Test Plan should build confidence in the testing process for a project, making approaches clear.

    • A Test Plan can include the Test Requirements

    • However, if the Test Requirements are too lengthy, they should become their own document: A Test Requirements Hierarchy

    Defining TR’s: What, Why, Where


    Agenda1

    David Capocci, CQA, CSTE

    Agenda

    • Defining TR’s: What, Why, Where

    • Fitting TR’s into the testing picture

      • What’s within our testing process

      • Generating TR’s

    • Organizing & Decomposing TR’s

    • Test Requirements Hierarchy Samples

    • Setting the stage for measuring test coverage


    Drilling down where test requirements fit into the picture

    David Capocci, CQA, CSTE

    Drilling down: Where test requirements fit into the picture

    Business

    Requirement

    1

    Test

    Generates

    Requirement

    M

    1

    Test

    Executes/Runs

    Scenarios/

    M

    Generates

    M

    Cases

    1

    Test

    Procedure/

    Script

    Fitting TR’s into the testing picture


    Drilling down

    David Capocci, CQA, CSTE

    Drilling Down

    First, Let’s look

    at this relationship:

    What’s within our

    testing process

    Business

    Requirement

    Test

    Requirement

    Test

    Scenarios/

    Then we’ll look

    at this relationship:

    Gernerating TR’s

    from what feeds into

    our testing process

    Cases

    Test

    Procedure/

    Script

    Fitting TR’s into the testing picture


    Atm example practice writing test requirements

    David Capocci, CQA, CSTE

    Business Requirements:

    - “ATM must do withdrawals”

    - “Withdrawals are between $20-$300”

    - “Withdrawals are in $20 multiples”

    Group Exercise!

    1. Limit the scope to these 3 requirements.

    2. What will you validate (test for)?

    3. Are there any implied requirements that may not be written out?

    ATM Example: Practice Writing Test Requirements

    What’s within our testing process


    Example 2 testing withdrawals on an atm

    David Capocci, CQA, CSTE

    “Validate that a withdrawal option is available”

    "Validate that a withdrawal of a multiple of $20, between $20-$300 can be done"

    "Validate that <$20 is not allowed"

    "Validate that >$300 is not allowed"

    "Validate that $20 multiples >$300 is not allowed"

    "Validate that non-$20 multiples between $20-$300 not allowed"

    "Validate strange numeric amounts/combinations not allowed (all zero's, all 9's, 20.0000)"

    “Validate that the withdrawal received is equal to the amount requested”

    "Validate that a valid withdrawal amount must be below the account balance”

    These are test requirements NOT tests because they do not describe the data element being used (like $20, $40, $60, $1)

    The data is irrelevant at this level, it will appear in the test cases used to cover these test requirements

    Example 2: Testing Withdrawals on an ATM

    Test Requirements Identified (among others):

    What’s within our testing process


    Test scenarios cases for validate that a withdrawal of a multiple of 20 between 20 300 can be done

    David Capocci, CQA, CSTE

    Test Scenarios/Cases for -“Validate that a withdrawal of a multiple of $20, between $20-$300 can be done”

    What’s within our testing process


    Test procedure script for previous example

    David Capocci, CQA, CSTE

    Step 1: Insert Card

    Step 2: Enter PIN

    Step 3: Select Withdraw option

    Step 4: Enter dollar amount

    Step 5: Validate amount received

    Do until EOF ‘until end of data file

    Input data record

    Senddata CARDINFO to “Cardfield”

    Senddata “Enter”

    Senddata PIN to “PINFfield”

    Senddata “Enter”

    Senddata “W” to “SelectionField”

    Senddata AMOUNT to “DollarField”

    Senddata “Enter”

    If ErrorMsg > 0 then print ErrorMsg

    Print “DollarAMTgiven”

    Loop

    Test Procedure & Script for previous example

    Procedure:

    Script: (in pseudo-code )

    Think

    Manual !

    Think

    Automated !

    What’s within our testing process


    Agenda2

    David Capocci, CQA, CSTE

    Agenda

    • Defining TR’s: What, Why, Where

    • Fitting TR’s into the testing picture

      • What’s within our testing process

      • Generating TR’s

    • Organizing & Decomposing TR’s

    • Test Requirements Hierarchy Samples

    • Setting the stage for measuring test coverage


    The workbench concept

    David Capocci, CQA, CSTE

    Rework

    Product Input

    Check

    Product Output

    DO

    DO

    EntranceCriteria

    Exit Criteria

    Standards

    Tools

    The Workbench Concept

    Our workbench is called “Generating Test Requirements”

    Generating TR’s


    Entrance criteria for business requirements to generate test requirements

    David Capocci, CQA, CSTE

    Entrance Criteria for BusinessRequirements to generate Test Requirements

    • Visible ?

    • Clear? (unambiguous)

    • Complete?

    • Consistent? (conflicting requirements must be prioritized)

    • Reasonable? (achievable)

    • Measurable? (quantifiable)

    • Modifiable? (will it change or is it stable?)

    • Traceable? (the source is known)

    • Dependent requirements identified?

    • Testable? (given current environment, resources, skills)

    Generating TR’s


    Exit criteria for test requirements

    David Capocci, CQA, CSTE

    Exit Criteria for Test Requirements

    • Can another tester create test cases/scenarios from these?

    • Does a Test Requirement specify what is being tested and what about it we are validating? (Clear?)

    • Are the Test Requirements…

      • Complete?

      • Consistent? (conflicting requirements must be prioritized)

      • Reasonable? (achievable)

      • Measurable? (quantifiable for measuring test coverage)

      • Modifiable? (will it change or is it stable?)

      • Traceable? (the source is known)

      • Testable? (given current environment, resources, skills)

    • Do the test requirements cover the complete scope of the project?

    • Are all the test requirements verified and signed off by the Development Team?

    Generating TR’s


    When creating test requirements do

    David Capocci, CQA, CSTE

    When creating Test Requirements (“Do”)...

    • Use “action” verbs & words

      • “Validate that…”

      • “Check for…”

      • “Test that…”

  • Trace them back to the source

  • Remember that different applications arrange in different ways

    • Think of MF, batch, C/S, web, e-commerce, GUI, etc.

    • Use “testing considerations” checklists that generally cover what kinds of things should be considered when testing your specific situation

  • Make your Test Requirements document a “living document”

  • Generating TR’s


    David Capocci, CQA, CSTE

    Also...

    • Maintain a level of balance between too much & too little

      • Too High level: won’t be useful, vague, can’t generate test cases from it.

      • Too low level: Over-process, over documentation, no productivity

      • General rule: 5-7 levels deep in an outline format

  • Organize them!

    • Outline/Hierarchy format recommended

    • Look at how the BA breaks down the project into areas

    • Look at how the PA breaks down the project into areas

    • Organize by functional areas

    • Organize by “types” of testing (Function vs. System vs. Non-Functional)

  • Generating TR’s


    Agenda3

    David Capocci, CQA, CSTE

    Agenda

    • Defining TR’s: What, Why, Where

    • Fitting TR’s into the testing picture

      • What’s within our testing process

      • Generating TR’s

    • Organizing& Decomposing TR’s

    • Test Requirements Hierarchy Samples

    • Setting the stage for measuring test coverage


    Distinguishing the types of testing

    David Capocci, CQA, CSTE

    Distinguishing the types of testing….

    IX. Stress Tests (breaking point: memory, resources)

    X. Resource Usage Tests

    XI. Documentation Tests

    XII. Compatibility Tests

    XIII. Recovery Tests

    XIV. Serviceability Tests

    and others…

    *III - XIV are all “Systems-based tests”

    I. Function-Based Tests

    II. User Interface Tests

    III. Security Tests

    IV. Installation Tests

    V. Configuration Tests

    VI. Performance Tests (Response)

    VII. Load Tests (simultaneous users, lots of small transactions)

    VIII. Volume Tests (Big transactions)

    Organizing TR’s


    Organizing by functional areas

    David Capocci, CQA, CSTE

    Organizing by Functional areas….

    • Most testers perform function-based, or requirements-based tests

      • Tests on functionality that the system will provide

      • Usually scenario/case based

      • includes normal and alternate (invalid) cases

    Organizing TR’s


    Organizing by functional areas1

    David Capocci, CQA, CSTE

    Organizing by Functional areas….

    • Most testers also perform User Interface Style Tests

      • These are generally separate from the functionality that the software will provide

      • Usually encompass the architectural standards & compliance (like Windows Design Standards)

      • Also includes tests of navigation, menus, admin functions (like printing, saving)

    Organizing TR’s


    Agenda4

    David Capocci, CQA, CSTE

    Agenda

    • Defining TR’s: What, Why, Where

    • Fitting TR’s into the testing picture

      • What’s within our testing process

      • Generating TR’s

    • Organizing &DecomposingTR’s

    • Test Requirements Hierarchy Samples

    • Setting the stage for measuring test coverage


    Remember this drilling down

    David Capocci, CQA, CSTE

    Remember this?…Drilling down

    Fitting TR’s into the testing picture


    Decomposing drilling down within a test requirement

    David Capocci, CQA, CSTE

    Business

    Function

    Tasks within the

    Function

    Transactions to

    perform a task

    Data Entry Types

    for transactions

    Field Validation

    Decomposing: Drilling down within a Test Requirement

    Business

    Requirement

    Test

    Requirement

    Keep the

    function-based

    perspective

    in mind!

    Test

    Scenarios/

    Cases

    Test

    Procedure/

    Script

    Fitting TR’s into the testing picture


    Test requirement decomposition

    David Capocci, CQA, CSTE

    Test Requirement Decomposition

    Decomposing TR’s


    Test requirement decomposition1

    David Capocci, CQA, CSTE

    Test Requirement Decomposition

    High level Functional Areas:

    usually from

    “Functional Spec” type

    docs, or BA work

    Business

    Function

    Tasks within the

    Function

    Transactions to

    perform a task

    Lower level

    Functional Areas:

    usually from

    “Technical Spec” type

    docs regarding

    internal logic,

    or PA work

    Data Entry Types

    for transactions

    Field Validation

    Decomposing TR’s


    Test requirement decomposition2

    David Capocci, CQA, CSTE

    Test Requirement Decomposition

    • You can start on the high level functional areas early into the project & build in lower level areas as they become available

    • Any level can contain a test requirement which can also be made up of (or broken down into) lower level test requirements

    Decomposing TR’s


    Business function level

    David Capocci, CQA, CSTE

    Business Function Level

    • Try to identify “groups” of functions or functions connected by similar themes

      file management functions, printing functions, help functions, car rental functions, reservation functions, ticket purchase functions, claim reporting functions

    • Be sure all areas of the system are covered. If something is left out or doesn’t fit into a group, it becomes its own group.

    • It may be easier to identify functional areas by “window” instead of by function.

    Trans Data

    Type

    Field

    Validation

    Function

    Task

    Transaction

    Decomposing TR’s


    Business function level1

    David Capocci, CQA, CSTE

    Business Function Level

    • At this level, the idea is seeing the connections, integration, and interactions of the system’s functionality.

    • May not necessarily be identifying a test requirement at this level as much as just identifying the functional area.

    Trans Data

    Type

    Field

    Validation

    Function

    Task

    Transaction

    Decomposing TR’s


    Task level

    David Capocci, CQA, CSTE

    Task Level

    • Break down each Function area into the tasks within the function

    • For complex tasks, it is possible to break them down further into sub-tasks

    • Some Business Functions can not decompose into further tasks (example: Check Writing function)

    Trans Data

    Type

    Field

    Validation

    Task

    Function

    Transaction

    Decomposing TR’s


    Transaction level

    David Capocci, CQA, CSTE

    Transaction Level

    • From this level down, we start to address the internal things that occur to make the functions and tasks happen

    • Identify any logical transactions that ties the task to the database or any other transactions necessary to perform the task.

    • Identify any data processing, calculation, data formatting type transactions

      Note: A screen or window may cause the execution of several different logical transactions

    Trans Data

    Type

    Field

    Validation

    Transaction

    Function

    Task

    Decomposing TR’s


    Transaction data type level

    David Capocci, CQA, CSTE

    Transaction Data Type Level

    • Identify which of the four types the transaction can become: Add, Change, Delete, Inquire

    • It is entirely possible that a transaction can be multiple types.

    • If a transaction is only one type, you can wrap this level up into the higher level.

    Trans Data

    Type

    Field

    Validation

    Function

    Task

    Transaction

    Decomposing TR’s


    Field validation level

    David Capocci, CQA, CSTE

    Field Validation Level

    • Most testers like to jump directly to this level. It’s the most obvious at times.

    • Field validation covers all edits & cross edits on fields and data.

    • Be careful of the detail you document at this level. Remember the separation between test requirement and test case.

    Field

    Validation

    Trans Data

    Type

    Function

    Task

    Transaction

    Decomposing TR’s


    Field validation level1

    David Capocci, CQA, CSTE

    Field Validation Level

    • Not all test requirements (especially at this level) fit well to be documented in this manner.

      • Example: Testing all the stated properties of windows objects

      • Use “Validate that the properties of all the objects in this window match the properties listed on the Object Properties Reference Table in Appendix B upon initialization of the window”

      • Don’t list each property check as a separate test requirement if it can be summarize under one test requirement

      • This is a judgement call YOU make for your given project.

    Field

    Validation

    Trans Data

    Type

    Function

    Task

    Transaction

    Decomposing TR’s


    Example 3 rental car application

    David Capocci, CQA, CSTE

    Example 3: Rental Car Application

    1. Validate that a Rental can occur.

    1.1 Check Customer policy coverage

    1.2 Query Car availability

    1.3 Query Car rates

    1.4 Open a Rental ticket

    1.4.1 Validate that a customer record can be entered

    1.4.2 Validate that credit card approval is obtained

    1.4.3 Validate that status on the car record is changed

    from “waiting” to “rented”

    2. Billing Function

    3. Reservation Function

    Let’s look at

    the lower levels

    for this one

    Then we’ll try it

    on this one

    Decomposing TR’s


    Example 3 rental car application1

    David Capocci, CQA, CSTE

    Example 3: Rental Car Application

    1. Validate that a Rental can occur.

    1.4 Open a Rental ticket

    1.4.1 Validate that a customer record can be entered

    1.4.1.1 Validate that a new customer can be added to the customer table

    1.4.1.1.1 Validate that the first name is all alpha

    1.4.1.1.2 Validate that the age is > 21.

    1.4.1.1.3 Validate that the phone number is numeric

    1.4.1.1.4 Validate area code is an existing area code number.

    1.4.1.2 Validate changing an existing customer

    Decomposing TR’s


    Example 3 rental car application2

    David Capocci, CQA, CSTE

    Example 3: Rental Car Application

    1. Validate that a Rental can occur.

    1.4 Open a Rental ticket

    1.4.2 Validate that credit card approval is obtained

    …fill in the lower level test requirements!

    First, Identify any sub-areas (further tasks, or even

    separate transactions within this)

    Then, Identify the lowest level field validation test

    requirements (think about what is typically

    involved with credit card authorizations)

    Decomposing TR’s


    What did you come up with

    David Capocci, CQA, CSTE

    What did you come up with?

    1. Validate that a Rental can occur.

    1.4 Open a Rental ticket

    1.4.2 Validate that credit card approval is obtained

    _________________________________________

    _________________________________________

    _________________________________________

    _________________________________________

    _________________________________________

    _________________________________________

    _________________________________________

    _________________________________________

    Decomposing TR’s


    Possible test requirements

    David Capocci, CQA, CSTE

    Possible Test Requirements...

    Function

    Task

    Transaction

    1. Validate that a Rental can occur.

    1.4 Open a Rental ticket

    1.4.2 Validate that credit card approval is obtained

    1.4.2.1 Validate the expiration date is a valid future date

    1.4.2.2 Validate the expiration date is not within 1 month of

    expiring.

    1.4.2.3 Validate that the CC# is 12 digits

    1.4.2.4 Validate that the $ amount is <= credit balance available

    1.4.2.5 Validate that an authorization # is received.

    Decomposing TR’s


    Agenda5

    David Capocci, CQA, CSTE

    Agenda

    • Defining TR’s: What, Why, Where

    • Fitting TR’s into the testing picture

      • What’s within our testing process

      • Generating TR’s

    • Organizing & Decomposing TR’s

    • Test Requirements Hierarchy Samples

    • Setting the stage for measuring test coverage


    Trh samples

    David Capocci, CQA, CSTE

    TRH Samples

    Let’s look at a few examples of how test requirements can be organized:

    • Sample 1: Excerpt from a Personal Finance app

    • Sample 2: actually organizing Function requirements, but good representative of the concept

    • Sample 3: QBS, from Rational’s sample project (adds types of testing into the hierarchy)

    • Sample 4: another view of the Personal Finance app

    • Sample 5: A mainframe batch system


    Test coverage measures

    David Capocci, CQA, CSTE

    Test Coverage Measures

    • Test Requirements are the “what” of testing & are the basis for establishing the completion of testing

    • TR’s give us the point of measurement for test coverage

    • Each TR should receive a Priority, Risk, and Weight

    • Each TR should be tracked for Verification () and Validation (%)

    Test Coverage Measures


    Summary recap

    David Capocci, CQA, CSTE

    Summary & Recap

    • Defined: What, Why, Where

    • Described: How TR’s “fit” into the big picture

    • Decomposed: Organizing TRs & Generate TR’s into a measurable format

    • Demonstrated: Sampling some TRH

    • Determined: The Measurement Connection


    ad