1. Why is Test Planning so Hard? Elizabeth D’Angelo, PhD
CGI Group Inc.
2. 2 Confidential Agenda Background
Test Planning: Three Issues
Test Planning: Three Solutions
Practical Tips to Help you in Test Planning
3. 3 Confidential Reasons Why Test Planning is Difficult
4. 4 Confidential Test Planning Issue 1: Knowledge (1) Right Mix is Critical!
Requirements: Implicit vs. explicit knowledge (people vs. docs)
Thorough knowledge of applications and flows
Breath vs. depth
5. 5 Confidential Test Planning Issue 2: Project Management (1) Test Planning is Project Management within Project Management!
It is NOT optional
Be prepared to articulate WHY is needed
Plan – Prepare – Execute motto
Take time to do proper planning
“Vanilla flavour” plan is useless
6. 6 Confidential Test Planning Issue 2: Project Management (2)
7. 7 Confidential Test Planning Issue 3: The “Gruyere” paradigm How many holes are in this cheese?
What is actually delivered vs. what was planned to be delivered?
Are all the features truly complete?
8. 8 Confidential Test Planning Solution 1: The Expert (1) KNOW the WHAT…
9. 9 Confidential Test Planning Solution 1: The Expert (2) Become the testing guru
Help your team to become the testing experts
But there are also cheaper ways:
Lunch and Learn sessions
Organize workshops with presentations
Test case design
Create a testing library
10. 10 Confidential Test Planning Solution 1: The Expert (3) Build your credibility by becoming the industry/application expert
Read ALL documentation
Review the requirements, design, architecture
Talk to users, developers
Don’t be afraid of asking questions
Ensure to communicate your status properly
Assume responsibility when needed
11. 11 Confidential Test Planning Solution 1: The Expert (4) And there is more… Soft Skills!
Communications (defect reporting)
Negative thinking or “What can go wrong”
Discipline and Tenacity
12. 12 Confidential Test Planning Solution 1: The Planner (1) KNOW the HOW
Discuss under/over testing
Who will be doing what
What needs to be in place before test starts
Test case design guidelines
Metrics to be collected
13. 13 Confidential Test Planning Solution 1: The Planner (2) You will reap what you sow
Planning is the most essential activity!
PLAN, PLAN, PLAN ….
You cannot prepare what you have not planned for
You cannot execute what you have not prepared for!
14. 14 Confidential Test Planning Solution 1: The Planner (3) Make your Master Test Plan the Bible
Document ALL your assumptions
Think of ALL risks
Use ALL information available to you
Establish in/out scope INCLUDING
Brainstorm and define what will be tested when!
15. 15 Confidential Test Planning Solution 3: The Weather-Man (1) Look at signs to forecast quality issues
Previous history (application, business function, etc.)
Get to know the team
Collect and store metrics ? you will need them!
Clearly communicate deviations from plan based on unexpected quality issues
16. 16 Confidential Test Planning Solution 3: The Weather-Man (2) Establish and Communicate Forecast for your Plan:
SLAs for defect fixes
17. Confidential Practical Tips to Help you in Test Planning
18. 18 Confidential ALWAYS Use Risk-Based Approach! (1) Spend your effort where is worth it,…Unless you have unlimited time/budget/resources ?
Risk-based approach will allow you to test
More comprehensibly what really matters
First what is more important
If needed, know where to cut testing
NOT an isolated task, you need to socialize approach and get agreement on risk areas
19. 19 Confidential ALWAYS Use Risk-Based Approach! (2) Technical risks assessment should include:
New and immature technology,
Brand new system or enhancement
New and difficult to understand business requirements,
Complexity of business logic,
Complexity of technology,
Size of the project,
Skill mix on the project,
Number of users (including customer facing or back office considerations) historical data when appropriated (the risk is higher for modified functions that have frequently failed in the past, i.e., Do you have any historical data of defects by area?)
20. 20 Confidential ALWAYS Use Risk-Based Approach! (3) Business Impact assessment should include:
Importance of features from client/customer perspective
Criticality of features from the business perspective
Frequency of use
21. 21 Confidential ALWAYS Use Risk-Based Approach! (4) As a rule of thumb:
Front end (end user)
High frequency, High volume of users
Wrong information displayed
Medium frequency and groups of users
Used by a few users
22. 22 Confidential Risk-Based Approach Process Organize test risk assessment workshop with business and technical stakeholders
Identify and list requirements, business functions
Evaluate technical and business impact (rate 1-3)
Evaluate probability of failure (rate 1-3)
23. 23 Confidential Risk-Based Approach: Business/Requirements Map
24. 24 Confidential Cover all aspects of testing When planning… make sure to cover the overall process:
25. 25 Confidential Schedule Templates Detailing All Activities (1) Have an overall schedule template that you can easily fill with specific data
26. 26 Confidential Schedule Templates Detailing All Activities (2)
27. 27 Confidential
28. 28 Confidential Forecasting with Metrics – Failure Rate/Env Document what is the expected failure rate
Existing application/delivery team: use previous data
Be as granular as possible
By Functional Area
If you don’t have data, rule of thumb is around 30% (i.e., 30% of test cases will fail during test execution)
Incorporate retesting activities to schedule
29. 29 Confidential Forecasting with Metrics – Defect Age SLAs Negotiate and document SLAs for defect fixing, then measure
How long defects remain in New state (i.e., not assigned?)
How long defects remain in Assigned state (Are too many defects assigned to the same resource and therefore there is no progress?
How long defects remain in Fixed state (i.e., before being closed by Test Team)? (Is testing team following up on defect fix verification? Are there competitive priorities (execution vs. verification)?
A high number of defects in Need Info state could indicate that there are concerns with the requirements, lack of details on the defect (to reproduce).
30. 30 Confidential Conclusions DO
PLAN, PLAN, PLAN …. AND THEN PLAN!
Get buy-in into risk-based approach
Create templates to help you be efficient
Use historical information/metrics to predict future
Share your documents, status and metrics so stakeholders understand challenges
Hope for the best, but plan and prepare for the worst!
Work in isolation
Over complicate your methodology
Think you don’t need to understand domain
Do ad-hoc testing (unless it is planned)
Keep issues, departures from assumptions to yourself
Be naďve and plan for the best!
31. THANKS!! Questions??
32. 32 Confidential Contact Information