Automatic Error Prevention

1. Automatic Error Prevention JTest C++Test By Shimrit Tzur-David

2. Outlines • Automatic Error Prevention overview • JTest Static Analysis - Coding Standards • JTest Unit Testing • JTest White-Box Testing • JTest Black-Box Testing • JTest Regression Testing • C++Test Static Analysis • Freeware Tools • Summery

3. Automatic Error Prevention overview • The key for developing reliable software on time and on budget is twofold: • Reduce the opportunity for errors by following coding standards. • Thoroughly test each class, as soon as it is developed to prevent small mistakes from growing into widespread, difficult-to-pinpoint problems.

4. Motivation

5. What is Automatic Error Prevention ? • Methodology for improving software quality and reliability. • It uses information gained from software testing, measurement, and monitoring to progressively improve the development process.

6. What is AEP ? – Cont. • Fixing problems where they occur in the manufacturing process, not only eliminates many quality problems in the finished product, but also promotes the ultimate goal of improving the quality of the manufacturing process. • By fixing the process itself, it's possible to prevent the same types of errors from occurring over and over again.

7. Five simple steps for AEP • Detect an error • Isolate the cause of the error • Locate the point in production that created the error • Implement practices to prevent the error from reoccurring • Monitor for improvement

8. Example • Inspectors on an auto assembly line discovered that seat bolts are not being tightened properly. • The cause: the bolts do not exactly fit the tool used to tighten them. • The corrective action: provide the proper fitting tool. • Monitoring the process is accomplished by closely inspecting the seat bolts for tightness.

9. Error Prevention vs. Error Detection • Error detection is the process of finding and fixing errors after an application is built; the flawed process that generated those errors is left uncorrected. • In the seat example, error detection would have simply tightened the seat bolts at the end of the assembly line. This action left the root of the problem embedded in the manufacturing process.

10. Testing Categories • There are few tests which can be performed in order to significantly reduce the number of errors in our code: • Static Analysis • White-Box Testing • Black-Box Testing • Regression Testing

11. Traditional coding standards • Rules which apply to constructs within the class under test. • A traditional coding standard might test whether or not a file’s source code contains a construct that has a high probability of resulting in an error. • For example, one traditional coding standard checks that we use “equals” instead of “==” when comparing strings.

12. Global coding standards • Rules that ensure that projects use fields, methods, and classes wisely. • They program correctness, ease of understanding, and ease of maintenance. • A global coding standard might check that a project does not contain logical flaws and unclear code (such as unused or overly-accessible fields, methods, and classes). These problems increase the probability of an error being introduced into the code, and might also make the code less object-oriented.

13. Unit testing • Focused test on the smallest possible unit of the software application. • By performing unit testing, we can catch errors as early as possible during development, when errors are easier to find and correct. • We are able to test parts of a project without waiting for the other parts to be available. • We will be able to test internal conditions that are not easily reached by external inputs in the larger integrated systems.

14. White-Box Testing • White-box testing doesn't test that the class behaves according to the specification, but instead, it ensures that the class doesn't crash and that it behaves correctly when passed unexpected input. • White-box involves looking at the class code and trying to find out if there are any possible class usages that will make the class crash (in Java this is equivalent to throwing an uncaught runtime exception). • White-box testing is a process that checks code using test data derived from the class's internal structure.

15. Black-Box Testing • Black-box testing checks that the class behaves according to the specification, meaning that the class produces the correct output for each input. • The tester does not know anything about how the program is coded. He simply takes any output, and compares it to what he thinks is the desired output. • We can view the program as a "black-box", whose behavior can be determined by studying its inputs and the related outputs.

16. Regression Testing • Regression testing checks that the class behavior doesn't break when the code is modified. • Regression testing checks that the class continues to perform correctly after it has been modified. • In regression we’re going back over our previous tests to ensure that the bug we previously found has been fixed and that no new bugs have been introduced.

17. Outlines • Automatic Error Prevention Overview • JTest Static Analysis - Coding Standards • JTest Unit Testing • JTest White-Box Testing • JTest Black-Box Testing • JTest Regression Testing • C++Test Static Analysis • Freeware Tools • Summery

18. JTest • Parasoft's Jtest testing tool enables statically and dynamically analyzing Java code. • It is an Automated Error Prevention product that automates Java unit testing and coding standard compliance. • Jtest checks whether code follows over 400 coding standard rules (plus any number of custom rules) and can automatically correct violations of over 200 rules. • With the click of a button, developers can identify and prevent problems such as uncaught runtime exceptions, functional errors, memory leaks, performance problems, and security vulnerabilities.

19. JTest static analysis • Static analysis can be thought of as consisting of both an automatic code review and automatic coding standards enforcement. • Adhering to coding standards is an effective error prevention strategy. Coding standards are rules that ensure that code is written in a way that makes it less error-prone.

20. JTest static analysis – Cont. • By adhering to these standards, we significantly reduce the opportunities to introduce errors in the code. • Adhering to coding standards enforcement also makes code more readable and easier to maintain. • Jtest statically analyzes each class by parsing its Java source code and comparing it to a set of coding rules. • Jtest can automatically correct many of the coding standard errors it identifies.

21. Example • To check whether a class follows the default set of Jtest coding standards: • Select the class resource, and click the Play button in the toolbar. • By default, when you click this button, Jtest will check coding standards as well as perform unit testing.

22. The source file

23. The standard tab • A Jtest summary panel opens after the test completes. The Standards tab of this panel displays the following information about the coding standard test run: • Elapsed time for the coding standards checking. • Number of files checked. • Number of failed runs. • Number of errors found. • Number of suppressed errors. • Number of rules violated.

24. Errors found window

25. Rules violation Text label ‘case10’ maybe typo for ‘case 10’ The Rule: Avoid using text labels in "switch" statements. case 2 is missing either “break”, return, or /* falls through */ The Rule: Avoid a "switch" statement with a bad "case". Literal constant is used: 20 The Rule: Avoid using literal constants. Exceptional constants: -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 0.0, 0L, 0.0F, 0.0D, 0xF, 0xD Literal constant is used: -2 The Rule: Avoid using literal constants. Exceptional constants: -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 0.0, 0L, 0.0F, 0.0D, 0xF, 0xD

26. Automatic fixing of coding standard errors • Jtest can automatically correct many of the coding standard errors it identifies.

27. After Automated fixing

28. Outlines • Automatic Error Prevention overview • JTest Static Analysis - Coding Standards • JTest Unit Testing • JTest White-Box Testing • JTest Black-Box Testing • JTest Regression Testing • C++Test Static Analysis • Other Tools • Summery

29. JTest unit testing • Unit testing involves testing software code starting with its smallest functional point, which is typically a single class or method. • The objective of unit testing should be to verify the code's functionality and construction/robustness. • Ideally, unit testing should start applying the following types of tests as soon as a Java class compiles: • Construction testing (also known as white-box testing) • Functional testing (also known as black-box testing) • Regression testing

30. JTest unit testing – Cont. • Jtest creates a jtest project ‘ProjectName.jtest’ with the same packages hierarchy as in the original project. • For each class ‘ClassName.java’ in the original project, it creates a class ‘ClassNameTest.java’ in the jtest project. • ClassNameTest.java contains test methods for each method in ‘ClassName.java’. • For the method ‘methodName’ in ClassName.java, we will have in ClassNameTest.java the methods: testMethodName1, testMethodName2, until testMethodNameN, where N can be any number greater than 0.

31. JTest unit testing – Cont. • Each testMethodName method in ClassNameTest.java does two things: • White-box testing - It generates methodName in ClassName.java with different parameters. • Black-box testing - After the execution of methodName, it uses the validation method assertEquals to ensure that the output of the function is as expected. • There are few more methods in each ClassNameTest.java for the tests start-up and shut-down and a main method to run the tests.

32. JTest unit testing – Cont.

33. Outlines • Automatic Error Prevention overview • JTest static analysis - Coding Standards • JTest unit testing • JTest White-Box Testing • JTest Black-Box Testing • JTest Regression Testing • C++Test static analysis • Freeware tools • Summery

34. JTest White-Box Testing • White-box involves looking at the class code and trying to find out if there are any possible class usages that will make the class throwing an uncaught runtime exception. • Jtest executes the class under test using a symbolic virtual machine. • While executing each bytecode, it checks if there are solutions to the equations that will make the bytecode throw an uncaught runtime exception. If it finds a solution it reports an error for it.

35. Example

36. Errors found window

37. Errors /* NullPointerException */ publicvoid testStartsWith1() throws Throwable { boolean RETVAL = Simple.startsWith(null, null); } /* StringIndexOutOfBoundsException */ publicvoid testStartsWith4() throws Throwable { boolean RETVAL = Simple.startsWith(“”, “0”); } /* NullPointerException */ publicvoid testStartsWith3() throws Throwable { boolean RETVAL = Simple.startsWith(null, “0”); }

38. Few Words on Design by Contracts… • In order to fix white box testing errors, Jtest uses the principle of DBC. • A program is correct if it performs according to its specification. • The principal idea of DBC is that a class and its clients have a contract with each other: The client must guarantee certain conditions before calling a method specialized on the class (the preconditions), the class guarantees certain properties after the call (the postconditions). • If the pre- and postconditions are included in a form that the compiler can check, then any violation of the contract between caller and class can be detected immediately.

39. Automatic fixing white-box testing • Jtest can automatically correct many of the white box testing it identifies.

40. After the fixing

41. Outlines • Automatic Error Prevention overview • JTest static analysis - Coding Standards • JTest unit testing • JTest White-Box Testing • JTest Black-Box Testing • JTest Regression Testing • C++Test static analysis • Freeware tools • Summery

42. JTest Black-Box Testing • Black-box testing checks that the class produces the correct output for each input. • In general, any black-box testing tool requires the user to specify a set of inputs to try, along with the correct outputs for those inputs. The tool then runs the inputs automatically and checks that the correct outputs are generated. • Jtest automatically generates a set of sophisticated inputs; Jtest analyzes the class bytecodes and provides a minimal set of inputs that produce as much coverage as possible for the class.

43. JTest Black-Box Testing – Cont. • Jtest executes all of the inputs and provides the actual outcomes for those inputs. • The developer can see the outcomes and verify them with one click of a button. • when Jtest performs subsequent tests on this class, it will notify us if a different outcome is produced.

44. Example

45. Errors found window

46. Errors Expected: <false> but was: <true> Expected: <true> but was: <false>

47. Create user-defined test case • Let’s assume that we want to add our own test case to the method map, and to check what’s happening when we execute it with ’34’ as input. • All we need to do is to add to class SimpleTest.java the following method: • public void testMap() { assertTrue(Simple.map(34) == -2); } • And run the unit test again.

48. Outlines • Automatic Error Prevention overview • JTest static analysis - Coding Standards • JTest unit testing • JTest White-Box Testing • JTest Black-Box Testing • JTest Regression Testing • C++Test static analysis • Freeware tools • Summery

49. JTest Regression Testing • When Jtest first runs a unit test, it takes a "snapshot of the current functionality of each method tested, and records its behavior in a JUnit-format test case. • Jtest provides automatic regression testing. Even if we don't specify what the correct outcomes are, Jtest remembers the outcomes from previous runs and compares them every time the class is tested and reports an error for any outcome that changes.

50. Example • Let’s assume that we’re changing the add method to reduce i2 from i1, it can be intentional, but it can also be a typo.