Product risk analysis in practice. Kees Blokland. NO RISK NO TEST. Why risk based testing?. Exhaustive testing. Because we cannot test all possible ways through the software. Why risk based testing?. Testers must make choices. Test strategy. What to test and what n ot to test
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Product risk analysis in practice Kees Blokland NO RISK NO TEST
Exhaustive testing Because we cannot test all possible ways through the software Why risk based testing?
Testers must make choices Test strategy What to test and what notto test What to test thoroughly, what to test superficially What to test first, what to test later What to test once, what to test each software build What to test by senior testers, what by junior testers What to test scripted, what to test exploratory, … just to name a few
Testers must make choices We base these choices on the level of risk What to test and what notto test What to test thoroughly, what to test superficially What to test first, what to test later What to test once, what to test each software build What to test by senior testers, what by junior testers What to test scripted, what to test exploratory, … just to name a few
Risk defined WIKIPEDIA The probability or threat of quantifiable damage, injury, liability, loss, or any other negative occurrence that is caused by external or internal vulnerabilities, and that may be avoided through pre-emptive action.
V = I x R Voltage Current Resistance Electrical engineering: Ohm’s law
R = L x I Risk Likelihood Impact Testing: Test’s law on software failures NO RISK NO TEST
BMW needs to call back 10.000 cars in China due to a software defect Doors open spontaneously Gear switches from parked state and car starts rolling by itself from the hill source: computable.nl
Huge traffic jam due to system failure Huge traffic jams due to failing information boards source: computable.nl
Basic process of Product Risk Analysis • Preparation • Risk identification • Risk classification
1. Preparation • Divide system in components payment • New? • Changed? • Unchanged? preview login download upload templates shopping card
1. Preparation • Divide system in components • New / changed / unchanged • Identify attributes • Functionality / performance / security / usability / …
1. Preparation example
1. Preparation • Divide system in components • New / changed / unchanged components • Identify attributes • Functionality / performance / security / usability / … • Compose product risk assessment team • Testers / developers / users / designers / business analysts / product owners / …
2. Risk identification • Identify product risks in lines of text like: “This screen does not show the right values, so that all customers get the wrong invoice.” “The process is not ready in time, which means …” “A visitor has access to classified information, so …” • No risk found: empty cell • Combine comparable risks • Component or attribute missing: add it! • No consensus? split risks in separate items
2. Risk identification example
3. Risk classification • Determine likelihood of failure Frequency of use Inexperienced developers New tools High time pressure Complex functions Many changes Many interfaces Error prone functions Input needed from developers/testers L M H
3. Risk classification • Determine likelihood of failure • Determine impact Losing income Losing customers Societal Damage Claims by users Repair costs Damage to company image Compensation effort and cost Development and test cost Clients / Users / Business perspective L M H
3. Risk classification example • Determine likelihood of failure • Determine impact • Look up risk class
3. Risk classification example
Managing a risk analysis meeting • Limit meeting duration (2-3 hours) • Start with most interesting topics • Keep to the purpose of the meeting: product risks • project risks action item • Incomplete specifications action item “Performance is a risk, but we miss acceptance criteria” • Small releases: meeting not always required Always communicate!
How to map risks on tests? Low risk Medium risk Component Test + System Test High risk + System Integr. T. Test levels + Acceptance Test Performance Special test Security Implicit All Usability Superficial Specific Techniques Junior Thorough Medior Testers Formal Senior Model based Multiple Free format Session based Exploratory testing Session based by experts
How to map risks on tests? - continued Tested once Manual regression test Test rounds Multiple Automated regr. test Each - Experts Informal Review Specific Formal Focus Junior - Medior Testers Code review Pairing Senior Pairing with testers
What to test first and what later? First execute tests on high risk components / features / aspects Most serious defects Then execute tests on medium risk components / features / aspects Time Less serious defects Low severity defects In the end execute tests on low risk components / features / aspects Project deadline Low severity defects undiscovered
Remarks on • PRA in agile / scrum • PRA and outsourcing • PRA and large organizations
PRA in agile / scrum • Order of development (including test) is based on • Business value • Risk • Risks are also translated into acceptance criteria Business epic/feature PRA at epic/feature level Acceptance criteria User story PRA in sprint Test cases Acceptance criteria
PRA and outsourcing • Does supplier know impact of failing software???? • Does supplier know where to focus testing??? • Risk profile = basis for project estimation 50% higher risk Fixed price? 35% average risk 20% lower risk % testing effort Communication!
PRA and large organizations • Break PRA down in multiple levels: • Prioritize features in release level PRA • More high level • More focus on business value / impact • Product Risk Analysis on each feature • More detailed level • More focus on design and build A Online payment B Book preview B Mobile view C Website stylerefresh payment preview login download upload templates shopping card
Conclusion • Product Risk Analysis is a communication process • Translate risk into a clear test strategy • Make the process fit to the context No exact science Product Risk provides the key to maximum test efficiency and minimum residual risk NO RISK NO TEST!
谢谢! Thank you! email@example.com