Dealing with Leaks

1. Dealing with Leaks © 2003 Mark Bartosik, www.leakbrowser.com

2. What is a resource leak ? • Repeated allocation of a resource without timely deallocation Probable resources: • memory virtual memory, non-paged pool memory • address space • OS objects / handles files, threads, events, semaphores, sockets, windows, timers, etc. • disk space • application level / non OS recourses e.g. database records © 2003 Mark Bartosik, www.leakbrowser.com

3. Unreferenced resources int global = 20; void foo() { int b = 22; HANDLE h =CreateEvent(....); int * int_ptr = int(1000); int * array_ptr = new int[1024]; } © 2003 Mark Bartosik, www.leakbrowser.com

4. Unreferenced heap memory int global = 20; void foo() { int b = 22; HANDLE h =CreateEvent(...); int * int_ptr = int(1000); int * array_ptr = new int[1024]; execution is here } int main() { int a = 21; foo(); return EXIT_SUCCESS; } © 2003 Mark Bartosik, www.leakbrowser.com

5. Unreferenced heap memory int global = 20; void foo() { int b = 22; HANDLE h =CreateEvent(...); int * int_ptr = int(1000); int * array_ptr = new int[1024]; } int main() { int a = 21; foo(); execution is here return EXIT_SUCCESS; } © 2003 Mark Bartosik, www.leakbrowser.com

6. Unbounded resource consumption std::map<unsigned, transaction_t> transactions; void do_transaction(P1_t p1, P2_t p2) { transactions[id] = transaction_t(p1,p2); } © 2003 Mark Bartosik, www.leakbrowser.com

7. C++, avoiding the problem • encapsulate ownership within objects or smart pointersbstr_t CComPtr shared_ptr auto_ptr scoped_ptr auto_handle std::string, avoid BSTR, char* • only use raw pointers to implement tightly encapsulated objects,e.g. when implementingshared_ptr • never use raw pointers to transfer ownership • never use raw pointers with business logicexception: indication optional objects - use of NULL • Use a garbage collector • Problem:No language enforcement -- trusts the programmer © 2003 Mark Bartosik, www.leakbrowser.com

8. VB, avoiding the problem • Stay within Visual Basic 5/6 environment (avoid C / OS APIs). • Release objects (with events) in “unload” methodsset some_object = nothing • Forms reference objects (with events) and the objects can reference the form. This is a cyclic reference. • VB7 is implemented on .NET only and garbage collected. © 2003 Mark Bartosik, www.leakbrowser.com

9. Java, .NET, avoiding the problem • Stay within JVM or .NET VM (avoid C / OS APIs).You have no destructors, and finalizers are restrictive. • Cyclic references -- no problem they are garbage collected. • Don’t get lax, watch out for unbounded resource consumption. © 2003 Mark Bartosik, www.leakbrowser.com

10. Garbage collection - the solution • pause the entire program • find every object that can be reached from the static storage or thread stacks • mark those objects with an “age” • delete all older objects, in any order you like © 2003 Mark Bartosik, www.leakbrowser.com

11. Garbage collection - the solution • pause the entire program • find every object that can be reached from the static storage or thread stacks • mark those objects with an “age” • delete all older objects, in any order you like © 2003 Mark Bartosik, www.leakbrowser.com

12. Garbage collection - the penalties • Pause the entire program! When? For how long? • Find EVERY reachable object? Page in every object from disk! Must run in physical memory to be efficient. Working set will be same as used memory! • Delete all the garbage, in what order? Any order! So how do I test deterministically? So I cannot use finalizers? • These may or may not be acceptable compromises • Remember it only solves half the problem! © 2003 Mark Bartosik, www.leakbrowser.com

13. Workarounds • Watch dogs Who watches the watch dog? What can the watch dog do? Bark! • COM+ Memory and lifetime limits can be configured for COM servers • Heap limits Maximum heap size can be configured • WRITE GOOD CODE! • FIND THE LEAKS! © 2003 Mark Bartosik, www.leakbrowser.com

14. Available tools • Bounds checker • Purify • Heap Agent • Great Circle • Lint • Parasoft • C++ debug heap © 2003 Mark Bartosik, www.leakbrowser.com

15. Typical restrictions • Crashes itself on large scale programs • Crashes large scale programs • Only reports unreferenced memory not unbounded allocations • Only reports malloc, free etc • Does not report OS API leaks like SysAllocString • Only supports C++ • Requires recompilation • Reports many false positives • Requires many hours to run • Grinds target program to a halt © 2003 Mark Bartosik, www.leakbrowser.com

16. perfmon © 2003 Mark Bartosik, www.leakbrowser.com

17. Identifying the problem • Monitor resourcesperfmon “process private bytes” “process handle count” “process thread count”taskmgr VM Size, Handle count, Thread count, GDI Objects • Exercise the program • Coverage test -- does it leak? • Use case tests -- when does it leak (most)? © 2003 Mark Bartosik, www.leakbrowser.com

18. Quantifying the problem • Must stay out of the red • Must consistently reach the green, without crashing • The restart period might be... 1 hour 1 working day 1 working week 1 year 1000 transactions never! • Max resources might be... 100 KB 2 GB © 2003 Mark Bartosik, www.leakbrowser.com

19. Needle in a haystack • 3 processes • 250 component DLLs • 3500 source files • 1,500,000 lines of source code • 100 KB of leaks per transaction • exercising 15% of the codegreat only 225,000 lines of source to check! • 10,000 allocations per transaction • caused by probably no more than half a dozen lines of code I have hidden a few leaks in the 1,500,000 lines of D3000 code.We are going to find them NOW! © 2003 Mark Bartosik, www.leakbrowser.com

20. Ignore one time initialization • Ignore the first few transactions • They might cause one-time initialization leaks, but who cares? © 2003 Mark Bartosik, www.leakbrowser.com

21. Cycle and observe everything • Choose a small prime number • Perform this number of transactions or cycles • Track all allocations and deallocations © 2003 Mark Bartosik, www.leakbrowser.com

22. Cycle again observing deallocations • Allocations made in the last cycle, may not be deleted until the next • Perform additional cycles, only tracking deallocations strtype a = b; strtype & strtype::operator=(const strtype & rhs) { if (m_len < rhs.m_len) { char * temp = new char[rhs.m_len]; delete m_p; m_p = temp; } m_len = rhs.m_len; memmove(m_p, rhs.m_p, m_len); return *this; } © 2003 Mark Bartosik, www.leakbrowser.com

23. Leak Browser learns • Now generate a report of the leak • Leak Browser has just “learnt”, what you are interesting in • Next time Leak Browser will examine these few areas of code in greater detail (requiring more CPU) • Clear the leaks • Repeat 7 monitoring cycles • Repeat some additional cycles • Regenerate the report © 2003 Mark Bartosik, www.leakbrowser.com

24. Not everything is a leak • Background tasks may provide “background noise”for example, incoming news or price updates • Caches may delay some deallocations • Queues and logs may include records of the last 1000 transactions © 2003 Mark Bartosik, www.leakbrowser.com

25. Capabilities • Resource profiling - finding the bloat • Fast over 10,000 allocations per second can be tracked • Tracks about 300 key APIs, not just malloc & free • Finds both unreferenced leaks and unbounded leaks • Does not require code recompilation • Is not intrusive with large applications • Remote installation and debugging • It works! © 2003 Mark Bartosik, www.leakbrowser.com

26. How does it work? • Traps key APIs • API is redirected into Leak Browser stub • Stack trace is recorded with allocation details • In-memory database is maintained • If no PDB further instrumentation can be done © 2003 Mark Bartosik, www.leakbrowser.com

27. Challenges • Must be thread safe but... • Cannot hold a thread lock and call the OS • Cannot allocate memory • Restrictions on using thread local storage (it may not be allocated yet, or it may have been deallocated) • Has only micro-seconds to trace the stack • Standard Microsoft algorithm takes seconds • Cannot afford to call the OS • Must cope with optimized code • for stack tracing, and instrumentation • must modify hundreds or thousands of functions without side effects © 2003 Mark Bartosik, www.leakbrowser.com

28. Limitations • Need to be able to cycle the application • No true garbage detection (yet) • .NET interop only (so far) • No JVM support - none planned • Slow display -- too much HTML • Occasional crashes, but more reliable than the rest © 2003 Mark Bartosik, www.leakbrowser.com

29. Summary • Use garbage collection where appropriate • Use best practice for the programming language of choice • Monitor resource usage during development • Are you within your customer acceptable restart period? • If not • Leak Browserwww.leakbrowser.com © 2003 Mark Bartosik, www.leakbrowser.com