Debugging and Testing

(Chapter 8) Debugging and Testing

Testing • Testing is the process of running a program with the intent of finding bugs • Requires running the program on sample data • Might require writing extra code to emulate missing parts of the system • Requires the programmer to be "egoless" • Often performed by programmers other than the developers

Types of Testing • Static: Done without actually executing program • Code inspections • Walkthroughs • Dynamic: Done by executing program or its parts • Module or unit testing • Integration testing • System testing

Module or Unit Testing • Tests individual classes or small sets of classes • Generally done by the programmer responsible for the classes • Bottom-up process • May require the use of a throw-away test driver program

Test Driver for FarmerStateInfo int main() { Side farmer = EAST; Side wolf = EAST; Side goat = WEST; Side cabbage = WEST; FarmerState s = new FarmerStateInfo(farmer, wolf, goat, cabbage); s->display(); cout << "Testing safety:" << endl; if ( s->isSafe() ) cout << "Safety test wrong" << endl; else cout << "Safety test OK" << endl; farmer = WEST; s = new FarmerStateInfo(farmer, wolf, goat, cabbage); s->display(); cout << "Testing safety:" << endl; if ( s->isSafe() ) cout << "Safety test OK" << endl; else cout << "Safety test wrong" << endl; }

Test Driver Output 67% farmer ||F ||W G|| C|| Testing safety: Safety test OK F|| ||W G|| C|| Testing safety: Safety test OK 68%

Notes on the Output • What gets tested: • The constructor • The display() method • Safety of the cabbage, both when it's vulnerable and when it's not • The safety of the goat when it's not vulnerable • What does not get tested: • The safety of the goat when it is vulnerable • Note: the isSafe() method had to be made temporarily public

Testing State-Changing Methods int main() { FarmerState s = new FarmerStateInfo(WEST, WEST, WEST, WEST); s->display(); State newst = FarmerStateInfo::self(s); cout << "Action farmer-takes-self " << endl; if ( newst != NULL ) newst->display(); else cout << "not allowed" << endl << endl; newst = FarmerStateInfo::wolf(s); cout << "Action farmer-takes-wolf " << endl; if ( newst != NULL ) newst->display(); else cout << "not allowed" << endl << endl; . . . newst = FarmerStateInfo::cabbage(s); cout << "Action farmer-takes-cabbage " << endl; if ( newst != NULL ) newst->display(); else cout << "not allowed" << endl << endl; }

State-Changing Test Output 69% farmer F|| W|| G|| C|| Action farmer-takes-self not allowed Action farmer-takes-wolf not allowed Action farmer-takes-goat ||F W|| ||G C|| Action farmer-takes-cabbage not allowed 70%

Testing FarmerProblemInfo and State Equality int main() { Problem p = new FarmerProblemInfo(); State start = p->getStartState(); State final = p->getFinalState(); cout << "Testing state display:" << endl; start->display(); final->display(); cout << "Testing state equality:" << endl; if ( start->equals(final) ) cout << "Equality check wrong" << endl; else cout << "Equality check OK" << endl; if ( start->equals(start) ) cout << "Equality check OK" << endl; else cout << "Equality check wrong" << endl; }

Equality Test Output 71% farmer Testing state display: F|| W|| G|| C|| ||F ||W ||G ||C Testing state equality: Equality check OK Equality check OK 72% Note that the test driver could be used for other problems just by changing the constructor call.

Testing expand() int main() { Problem p = new FarmerProblemInfo(); State start = p->getStartState(); start->display(); Item testItem = new ItemInfo(start, NULL, NULL, 0, 0); ItemArray children = testItem->expand(p); cout << "Num children: " << testItem->getNumChildren() << endl; for (Integer i = 0; i < testItem->getNumChildren(); i++) { State nextState = children[i]->getState(); nextState->display(); } }

expand() Test Output 73% farmer F|| W|| G|| C|| Num children: 1 ||F W|| ||G C|| 74%

Integration Testing • Putting together the results of module testing • Top-down approach • Example: testing the SolverInfo class tests the integration of ProblemInfo and QueueInfo • Often includes a primitive graphical user interface (GUI) so that: • system has look and feel of final product • separate drivers not needed • Might require stubs or dummy routines for code to be added later

System Testing • Attempts to find problems in a complete program • Done when • program is first completed, and • as the program evolves (maintenance testing) • In industry, often done by an independent group of programmers • Requires creation of a test suite that: • is used over and over as program evolves • attempts to identify all combinations of inputs and actions that could cause program to fail

Testing and Proof of Correctness • It is not possible to test a nontrivial system with all possible inputs and outputs • So testing cannot formally prove that a system is correct, only give some level of confidence in it • Formal proofs of dynamic program correctness are not possible either, because executing programs are subject to physical influences • Formal proofs of abstract algorithm correctness are possible though difficult, and require machine assistance

Debugging • The process of fixing problems (bugs) that testing discovers • Debugginghas two parts: • finding the cause of a bug (can be 95% of debugging) • fixing the bug • A good programmer becomes adept at using a run-time debugger

Types of Program Bugs • Syntactic bugs not caught by the compiler Example: for (... ; ... ; ...); { ... } • Design bugs: easy to find, can be difficult to fix • Logic bugs: can be difficult to find, easy to fix

Types of Bugs (cont'd) • Interface bugs: • When assumptions made by a method caller are different than those of the callee. • Can be difficult to find, easy to fix • Memory bugs: • Freeing or overwriting storage that is later used • Especially difficult to find • Example: IntArray A; // = new Integer[4]; ... A[0] = 10; A[1] = 20: ...

Fixing Bugs • Fixing might entail correcting a single line of code • Fixing might entail redesign, recoding, and further testing: • For example, incorrect circularity check in expand() • Fixing a bug might cause other bugs to appear • It has been observed that code in which a bug has been found is more likely to still contain bugs

Defensive Programming • The easiest way to do debugging is to not have bugs • Failing that, one should strive for locality: Ensure that bugs that are identified are local problems found where and when they occur • Locality is the goal of defensive programming • Defensive programming begins with defensive design

Defensive Design • First goal: Simplicity of Design Example: Take the time and trouble to make a repeated piece of code a method, even if small • Also: Simplicity of Class Interfaces Example: refactoring the abstract StateInfo and ProblemInfo classes • Another goal: Program for Errors • Anticipate exceptions even when they are not expected • Example: add(Item) and remove() should check for fullness and emptiness even though currently all callers check first

Program Robustness Consider this add method for FrontQueueInfo: void FrontQueueInfo::add(Item item) { front++; items[front] = item; numItemsAdded++; } If add is called and the queue is full, a probable segmentation fault will occur. As written, this code is brittle. A robust program will not crash and burn when something unexpected happens.

Relying On the Caller One approach: try to ensure that every call protects itself: FrontQueue q = new FrontQueue(n); . . . if ( !q->full() ) { q->add(item); } else { <deal with full queue> } However, as the program evolves, it is possible that a call will be added that is not this protective.

Defensive Coding Using assert add itself could have some kind of defense against an incorrect call: void FrontQueueInfo::add(Item item) { assert( !full() ); front++; items[front] = item; numItemsAdded++; } If the queue is full, this will result in a helpful message and a program halt. Although this makes it easier to debug, the program is still brittle.

Defensive Coding Through Exception Handling An exception is an unexpected error condition: • Array index out of bounds • Divide by zero • Heap memory exhausted • Keyboard interrupt signal • Abort signal • Conditions specific to application

C++ Approaches to Exception Handling • Try to guarantee that an exception condition does not exist through an assertion, and generate an unrecoverable error if the assertion is false. • Deal with an exception condition through the use of a handler function that is called when a system-defined exception signal is raised. • Deal with general exception conditions by unwinding the stack with catches and throws.

Defensive Coding Example: Safe Arrays typedef int Integer; typedef Integer * IntegerArray; main () { IntegerArray a = new Integer[5]; IntegerArray b = new Integer[5]; for (Integer i = 0; i < 10; i++) a[i] = i; for (Integer i = 0; i < 10; i++) b[i] = i; for (Integer i = 0; i < 10; i++) cout << a[i] << " "; cout << endl; for (Integer i = 0; i < 10; i++) cout << b[i] << " "; cout << endl; } expected output: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 actual output: 0 1 2 3 4 5 6 7 0 1 0 1 2 3 4 5 6 7 8 9 problem: accessing array elements that are out of bounds

A Safe Array VectorInfo Class #include <assert.h> typedef int Integer; typedef Integer * IntegerArray; typedef class VectorInfo * Vector; class VectorInfo { private: IntegerArray p; Integer size; public: VectorInfo(); VectorInfo(Integer n); ~VectorInfo(); Integer& element(Integer i); };

Notes On VectorInfo • One class attribute is a dynamic array • One class attribute is the maximum array size • One constructor will implement a default array size • The element method will check for index out of bounds • assert.h is included

Exception Handling Using assert VectorInfo::VectorInfo() { size = 10; p = new Integer[size]; } VectorInfo::VectorInfo(int n) { assert ( n >= 1 ); size = n; p = new Integer[size]; assert (p != 0); // in case heap used up } Integer& VectorInfo::element(Integer i) { assert (i >= 0 && i < size); // line 40 return (p[i]); }

Main Program main () { Vector a = new VectorInfo(5); Vector b = new VectorInfo(5); for (int i = 0; i < 10; i++) a->element(i) = i; for (int i = 0; i < 10; i++) b->element(i) = i; for (int i = 0; i < 10; i++) cout << a->element(i) << " "; cout << endl; for (int i = 0; i < 10; i++) cout << b->element(i) << " "; cout << endl; }

Main Program Output VectorInfo2.cc:40: failed assertion `i >= 0 && i < size' Abort Although this code aborts, it is preferable because it gives the programmer useful information. From the point of view of the user, it is still brittle.

Exception Handling with signal.h • The signal.h file provides a standard mechanism for handling system-defined exceptions. Some: #define SIGHUP 1 /* hangup */ #define SIGINT 2 /* interrupt */ #define SIGILL 4 /* illegal instruction */ #define SIGFPE 8 /* floating point exception */ #define SIGBUS 10 /* bus error */ #define SIGSEGV 11 /* segmentation violation */ A programmer can arrange to have a handler function called when an exception arises using: signal(signal, handler);

Signal Handler Function • Returns void and takes an integer (signal number) as argument: • void handler(Integer signal); • If part of a class, must be static • Installed by associating it with a signal: • void signal(Integer signal, void (*handler) (Integer)) • Automatically invoked when an exception of the associate signal type occurs • Can be intentionally invoked by explicitly raising the associated signal: • void raise(Integer signal);

Signal Example: Keyboard InterruptMain Program #include <signal.h> ... void cntrl_c_handler(int sig); // handler prototype main() { int i = 0, j; cout << "COUNT TO J MILLION, Enter j: "; cin >> j; j *= 1000000; signal(SIGINT, cntrl_c_handler); // set interrupt ``trap'' while (i < j) // interrupt with ctl-C during this loop ++i; cout << " HIT " << j/1000000 << " MILLION" << endl; }

Keyboard Interrupt Handler void cntrl_c_handler(int sig) { char c; cout << "KEYBOARD INTERRUPT"; cout << "\ntype y to continue: "; cin >> c; if (c != 'y') exit(0); signal(SIGINT, cntrl_c_handler); // reset "trap" // and return }

Signal Example Output 6% test COUNT TO J MILLION, Enter j: 100 [control-C from keyboard] KEYBOARD INTERRUPT type y to continue: y [control-C from keyboard] KEYBOARD INTERRUPT type y to continue: y HIT 100 MILLION 7%

Raising An Exception Under Program Control #include <signal.h> ... void cntrl_c_handler(int sig);// handler prototype main() { int i = 0, j; cout << "COUNT TO J MILLION, Enter j: "; cin >> j; j *= 1000000; signal(SIGINT, cntrl_c_handler); // set interrupt ``trap'' while (i < j) // interrupt with ctl-C during this loop ++i; if (i % 1000000 == 0) // generate interrupt after raise(SIGINT); // each million cout << " HIT " << j/1000000 << " MILLION" << endl; }

Signal Example Output 9% test COUNT TO J MILLION, Enter j: 4 KEYBOARD INTERRUPT type y to continue: y KEYBOARD INTERRUPT type y to continue: y KEYBOARD INTERRUPT type y to continue: y KEYBOARD INTERRUPT type y to continue: n 530% Note that the program does not continue to completion.

User-Defined Signals Two signal codes are designed to be raised by programmers in application-specific circumstances: . . . #define SIGUSR1 16 /* user defined signal 1 */ #define SIGUSR2 17 /* user defined signal 2 */ . . .

User-Defined Signal Example Recall the safe array constructor: VectorInfo::VectorInfo(int n) { assert ( n >= 1 ); size = n; p = new Integer[size]; assert (p != 0); // in case heap used up } If the heap is exhausted (p==0), this code will abort. Suppose the programmer can do some garbage collection to reclaim parts of the heap.

User-Defined Signal Example (cont'd) #define SIGHEAP SIGUSR1 #include <assert.h> typedef int Integer; typedef Integer * IntegerArray; typedef class VectorInfo * Vector; class VectorInfo { private: IntegerArray p; Integer size; public: VectorInfo(); VectorInfo(Integer n); ~VectorInfo(); Integer& element(Integer i); static void vectorHeapHandler(Integer sig); };

User-Defined Signal Example (cont'd) VectorInfo::VectorInfo(int n) { assert ( n >= 1 ); size = n; p = new Integer[size]; if (p == 0) { // invoke handler to do raise(SIGHEAP); // garbage collection p = new int[size]; // try again to create array } } main () { signal(SIGHEAP, VectorInfo::vectorHeapHandler); vect a(5); vect b(5); . . . } void VectorInfo::vectorHeapHandler(int sig) { // possible action to reclaim heap storage }

Limitations of Signal Handling for C++ It would be nice to handle the other VectorInfo exceptions in this way, for example: #define SIGHEAP SIGUSR1 #define SIGSIZE SIGUSR2 class VectorInfo { private: IntegerArray p; Integer size; public: VectorInfo(); VectorInfo(Integer n); ~VectorInfo(); Integer& element(Integer i); static void vectorHeapHandler(Integer sig); static void vectorSizeHandler(Integer sig); };

Limitations of Signal Handling for C++ (cont'd) VectorInfo::VectorInfo(int n) { if ( n < 1 ) { raise(SIGSIZE); // bad vector size signal } size = n; p = new Integer[size]; if (p == 0) { raise(SIGHEAP); p = new int[size]; } }

Limitations (cont'd) main () { signal(SIGHEAP, VectorInfo::vectorHeapHandler); signal(SIGSIZE, VectorInfo::vectorSizeHandler); vect a(5); vect b(5); . . . } void VectorInfo::vectorSizeHandler(int sig) { cout << "Size error. Going with default." size = 10; p = new int[size]; } Unfortunately, this will not work since vectorSizeHandler must be static. It therefore has no access to size and p.

Exception Handling with catch/throw • A more sophisticated and general form of exception handling • Not intended to handle asynchronous exceptions defined in signal.h • Works by causing dynamic, non-local exit from runtime stack frames ("unwinding" the stack)

Normal Runtime Stack Handling call call main() { . . . foo(); . . . } void foo() { . . . bar(); . . . } void bar() { . . . wowie(); . . . } return return return call void zowie() { . . . . . . } void wowie() { . . . zowie(); . . . } call return

Normal Runtime Stack Handling (cont'd) In zowie, the runtime stack looks like: zowie wowie bar foo main Each normal return causes the stack to be popped: zowie wowie wowie bar bar bar foo foo foo foo main main main main main

Debugging and Testing