Modified Session Based testing on Gaia

1. Modified Session Based testing on Gaia Naoki’s Method

2. Preface • Step 1 : Open your mind. • Step 2 : Breathe in. • Step 3 : Breathe out. • This does not replace automation of any type • This doc and supplementary pdfs are available at http://people.mozilla.com/~nhirata/SBT/ • So please stop reading this, and pay attention to the discussion

3. Sections: • Section 1: boring terminology stuff • Section 2: How we can apply this to Gaia • Section 3: Demo

4. Section 1: • What is QA? • What’s exploratory testing? • What’s session based testing? • Why session based testing?

5. What is QA? • To me, it’s about risk management and then some • Unknown amount of bugs with unknown severity to find and have fixed in a limited amount of time • QA has to lower the risk of impact to the end user before the time of ship. • If it was a set number of bugs, we could probably apply one or two methods. Since it’s unknown, there’s no real good one solution… we employ various techinques • Pick your battles well and try to plan ahead

6. What’s exploratory testing? • “Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design and test execution. Cem Kaner, who coined the term in 1983,[1] now defines exploratory testing as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”” - http://en.wikipedia.org/wiki/Exploratory_testing

7. Plain English • What does the word explore mean to you? • Now apply that to software.

8. Why Exploratory? • Kaner: Exploratory testing better than scripted testing • http://searchsoftwarequality.techtarget.com/news/1323432/Kaner-Exploratory-testing-better-than-scripted-testing My own thought : 1) If 1+x and x =1 and x never changed… why wouldn’t the actual answer be 2? What would cause you to run it over and over again? 2) How much time do you spending testing reading and trying to understand not only the STRs but the results? Once you get to the point of familiarity, do you have tester fatigue? 3) If you do the same task over and over… are you thinking? People should be paid to think, computers should be able to do repetitive tasks.

9. What is Session-Based Testing? • “Session-based test management (SBTM) is a technique for managing exploratory testing. Two of the major criticisms of exploratory testing are that it's hard to make progress visible, and it's hard to know what kind of coverage you might have after exploratory testing has been completed. Session-based test management is one answer to those problems, since it provides a metric for measuring progress (sessions) and takes coverage into account.” - http://searchsoftwarequality.techtarget.com/tip/Using-session-based-test-management-for-exploratory-testing • This was introduced by James and Jon Bach

10. WAT? • Structure and metrics for unstructured approach to testing • Basically: • Set aside 1 hr of straight testing given mission goal • Area tested and your name • Jot down notes and bugs at the end • Give a summary/account of what you found in an outline fashion • Example : http://www.satisfice.com/sbtm/sample_session.htm

11. Um, You said metrics? • The session metrics are the primary means to express the status of the exploratory test process. They contain the following elements: • Number of sessions completed • Number of problems found • Function areas covered • Percentage of session time spent setting up for testing • Percentage of session time spent testing • Percentage of session time spent investigating problems • http://www.satisfice.com/sbtm/index.shtml • Generate with a Pivot Chart: • Bug density per each target area. • Average bug yield per sessions • Bug trend over a given period of time (Example: Total # of sessions and total # of bugs week by week for the past 3 month).  This is another way to pulse check on how good/bad the product is doing. • Who ran the most sessions/who filed the most bugs • http://blogs.msdn.com/b/testing123/archive/2009/03/11/session-based-testing.aspx

12. Difference between this and bug bash • http://blogs.msdn.com/b/testing123/archive/2009/03/11/session-based-testing.aspx

13. Supplemental PDFs and readings • http://staqs.com/docs/sbtm/ • James Lindsay’s “Adventures in Session-Based Testing” • Jon Bach’s “Session Based Testing” • Jon Bach’s “How to Manage and Measure Exploratory Testing” • Jon Bach’s “How to Measure Ad Hoc Testing” (this was his earlier paper) • http://www.mddionline.com/article/applying-session-based-testing-medical-software

14. Tools: • http://staqs.com/docs/sbtm/#2 • http://sessiontester.openqa.org/ • http://openqa.org/ • https://sites.google.com/site/sessionbasedtester/

15. Sections 2: • How Session Based Testing is applied • Some modifications necessary • Where it doesn’t apply • Advantage

16. Change your mindset • 1) This is the opportunity to make things better! • 2) Change your mindset from this is working right to… there are bugs and I will find them • 3) Human Cognitive Behavior: http://bigthink.com/ideas/20583 • Pay attention not just to the task at hand but everything around it. • 4) You are working with the Devs not against them • ie make it easy for them to reproduce and figure out where the issues are. • The easier it is for them to figure things out the more inclined they are to fix things earlier.

17. How Session Based Testing is applied • Have an outline of the positive testings that can be done • Since nothing was mapped for Gaia and it is revolving, it’s easiest to use the requirements as a base • Take that outline and use each bullet point as a mission. • Use the concept of states. • A function has a precondition a postcondition and a change. In the same way, this will leave the application in a state • Reset the device if you lose track of the state you are in.

18. What to test • Remember your QA interview question? • What would you test for or look for when testing a mug? • That’s right…. You’re applying the interview question onto the application. Imagine that. An interview question that applies to the job. [/snark]

19. Generic Testing Outline • Positive Test Cases • MVC: • UI • Dialogs • Test for friendly error messages • Functionality • Data • Special Case Conditions • Integration Testing • Multiple/Cross Feature • Network testing • Requirements • Safety • Installation • Compatibility • Usability • Hardware • Negative Test Cases • Boundary Conditions • Initialization • Deletion • Screen size • SD card space • Memory • Race conditions • Real World • Stress test • Performance Testing • Security • Accessibility • Color blind • Blind • L10n • Date/time • Number • Double byte • Hi-ascii • Words fitting ui • Proper translation

20. Risk Management • You don’t have all the time in the world to cover this… what do you do? • Don’t Panic • Test the requirements and explore from there. • Find the base line by exploring as much as you can by testing all the requirements • Look at the code check-ins and get a gist of all the changes • If you have coding experience, think… “how would I build it and where would I most likely %$#%#$ up?” • Then work on the “critical path” and defining it. • Put yourselves in the shoes of the end user! • “Optimize Quality for Business Outcomes” by Golze, Li, and Prince (from Mercury Publishing)

21. Some modifications necessary • I haven’t bothered taking notes on time • People often use metrics in the wrong fashion • This isn’t a measure on QA time; this is a measure on quality of software • I take roughly 2 to 3 hrs. in a session because I write down bugs at the time of testing, and it requires reproducibility • I think in terms of states and do testing around the mission as well as changes • If you wonder why I painstakingly labeled all the open bugs… now you know why. It’s to figure out which areas are the buggiest. • Having the bugs as the source of truth, you can just look at the gaia issues page and see what area needs the most work. • Because of this it’s important to make sure that all the gaia bugs are up to date and still a source of an issue. Ie verify that these bugs are still bugs.

22. Where it doesn’t apply • Regression testing • Smoke Tests Bottom Line: Automation is still needed to help cover these things! Sometimes regression bugs occur because of an oversight, or unexpected impact change. You still need to do overall testing somehow at the very least in a major cycle.

23. Why not verification testing? • Change is where you find variation • Test around the fix! • You might find more bugs!

24. Advantage • Regression testing is basically the scripted testing once it’s already run once. • It takes less time to setup session based testing, because there’s less reading, modifying, updating based on change. It’s great for a starting project or feature and you find bugs faster due to the less overhead. • Regression testing is better suited for automation, the downside is the overhead cost of creation and maintenance • Note: Unit test automation should still be done to cut down on the number of bugs/cost of fixing the bugs.

25. Section 3: Demo • Look at the system requirements • Take some time… • Test… • Look at a verification • Take some time… • Test around it… • Look at a group of code changes • See what changed… • Test around it…