1 / 32

Unit Testing

Unit Testing. What, why and how ? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se. My View on Unit Testing. Part of something bigger – testing on many levels Fast and early feedback! Tight feedback loop, results within seconds Early implies lower cost Confidence!

mairi
Download Presentation

Unit Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Unit Testing What, why and how? Johnny Bigert, 2012-10-25 johnny@johnnybigert.se

  2. My View on Unit Testing • Part of something bigger – testing on many levels • Fast and early feedback! • Tight feedback loop, results within seconds • Early implies lower cost • Confidence! • Less fear of refactoring • Better code! • Testability requires decoupling and usable APIs • Reliability! • Possible to remove all non-determinism

  3. Definitions and Terminology Unit test, functional test, system test – what does it mean?

  4. Definitions and Terminology • There are no definitive answers • Different dimensions of automated tests: • Level – Unit, component, integration, system • API – Developer test, external test • Characteristic – Functional, non-functional • Acceptance testing, regression testing

  5. What Level Are We Testing On? • Unit – The smallest unit of program code, normally a single function, class with methods or data structure with functions • Component – A set of units integrated together, normally a module or subsystem comprising an isolated set of classes or files. • Integration – Several layers of a stack integrated, normally a stand-alone application (e.g. client or server) • System – Several stand-alone applications integrated (e.g. clients and servers), normally the full system

  6. What API Are We Using For Testing? • Developer tests – Function calls to an internal or external interface, normally test and production code running in the same process • External tests – Invokation of a non-programming interface, normally running in separate processes or machines. For example, UI (simulated user input), network (HTTP etc), IPC.

  7. What Aspect Are We Testing? • Functional – The behavior of the system, normally expressed as a step-by-step instruction with some alternative paths • Non-functional – The characteristics of the system, normally expressed as measurements and numbers. For example, latency, round-trip time, memory/CPU/bandwidth consumption, scalability, availability.

  8. Functional “Component test” – personal favorite. Provides stability just like a “functional test” (test code less brittle than unit tests). Provides flexibility, reliability and fast feedback just like a “unit test” (quick to write, easy to mock out problematic parts, results within milliseconds).

  9. Non-Functional

  10. Choose How to Test • When choosing, consider • Cost (of writing, maintaining, execution resources etc.) • Feedback loop (time) • Reliability • Trade-off example: Mocking or using system x? • Mocking: cost of writing and maintaining, risk of modeling wrong behavior. • Using: cost/risk of non-determinism, cost of test resources and longer execution times.

  11. Regression And Acceptance Testing • Definition: “regression” or “regression bug” • Something that was once considered “done” but does not fulfill requirements anymore • Acceptance testing: the use of a test to determine if a feature fulfills the requirements • Acceptance tests becomes part of the regression test suite once a feature is “done”

  12. How to Unit Test Necessities of effective unit and component testing

  13. Dependencies • Assumption: we want to achieve fast feedback and reliable tests • Note, there are other kinds of tests that also have merit • Remove non-determinism: we need to be in control • Randomness, time, file system, network, databases, multi-threading • Remove dependencies to non-deterministic components

  14. Dependencies • What kind of problematic dependencies to you have in your code base?

  15. Getting Rid of Dependencies, C++ • Example: Randomness • class A { public: void foo() {boolcoinflip = rand() % 2 == 0; if(coinflip) … }}; • Problem: dependency to non-deterministic component (randomness)

  16. Getting Rid of Dependencies, C++ • C++ solution: introduce interface and use dependency injection • class IRandomness {virtual boolcoinflip() = 0;};

  17. Getting Rid of Dependencies, C++ • class A { public: A(IRandomness &r) : randomness(r) {} // ctor inject void foo() { if(randomness.coinflip()) … }private:IRandomness &randomness;};

  18. What Is a Mock Object? • Conforms to interface • Programmable behavior • For example, “if someone calls your send function with argument ‘hello’, return 7” • Can verify function invocations • Mock frameworks available • E.g. gmock (Google) • Much more elegant in reflective languages

  19. Getting Rid of Dependencies, C++ • Test code: • void testFoo() {RandomnessMock random; // inherits IRandomness A a(random); a.foo(); …}

  20. Getting Rid of Dependencies, C++ • Production code: • void main() {Randomness random; // inherits IRandomness A a(random); …}

  21. Bi-Directional Dependencies Application (inherits) sendData INetworkReceiver INetworkSender onDataReceived (inherits) Network

  22. Getting Rid of Dependencies, C • Function pointers = hard-to-read code • Link-time tricks • Link with other translation unit defining e.g. rand()

  23. Getting Rid of Dependencies, C • RandMock m; // mixed C/C++ test codeRandMock &getRandMock() { return m; // used by test case to access mock}int rand() {m.rand(); // forward to mock object} • Drawbacks: hard to understand test case code, one test executable per component to test = harder to maintain • Or clean C solution

  24. Code Coverage • Don’t focus on high levels of code coverage per se • Necessary, but not sufficient • Instead, focus on covering the most important user scenarios • After that, use code coverage to determine which code has not been covered • Analyze to understand what user scenarios are missing

  25. Unit Test Frameworks Test runners/frameworks • GTest • UnitTest++ • CppUnit • C Unit Testing Framework Mock frameworks • GMock • Hippo • Mockpp • Amop

  26. Vision Agree on what to aim for

  27. Fast Feedback Example From a previous job: • Proprietary hardware/software/OS platform • Very expensive hardware = bottleneck in testing • Emulator to test software locally on Linux machine • Ran unit tests in emulator, turn-around time = 15 mins • Unacceptable

  28. Fast Feedback Example • Main problem: emulator start-up time • Challenge: proprietary types used (e.g. string classes) • Solution: fake proprietary types (string with std::string, many others just empty implementations) • Ran unit tests on local machine,turn-around time < 15 seconds (from 15 mins)

  29. Fast Feedback Example • Invest to speed up feedback • Invest to remove testability obstacles • Unit testing should be easy, fast and fun

  30. Development Excellence • Defect prevention • Continuous improvement - feedback from bugs in order to improve software development process • Relentless testing • Regression detection • Continuous integration running all regression tests often • Focus on both functional and non-functional requirements • Always releasable • Keep CI builds green • Keep the product working

  31. Continuous Integration • Staged • Stage 1 – build, unit tests, code coverage and functional smoke tests – 10 mins • Stage 2 – extensive functional tests, smoke non-functional tests – 1 hour • Stage 3 – full functional and non-functional regression tests , static and dynamic analysis – 12 hours • Run stages per • Commit (fine-grained tracability, expensive), or • Regularly (1. commit, 2. hourly, 3. nightly)

  32. Code Analysis Tools • Static analysis (run on source code) • Lint – free, local analysis only, lots of false positives • Klocwork – commercial, global analysis • Coverity – expensive, global analysis, good precision • Dynamic analysis (need representative data) • IBM Rational Purify – commercial, memory leaks, double frees, buffer overflow, use of uninitialized memory. Instrumentation. • Valgrind – free, memory problems, heap, usage, cache usage, call graphs, threading problems. No instrumentation.

More Related