Advanced Kernel Debugging

1. Advanced Kernel Debugging Andre Vachon Software Development Lead Windows Product FeedbackMicrosoft Corporation

2. Goals, Non-Goals • Goals • Give you insights into what the debugger can do • Will be referring you to the docs often • Cover important debugger features and changes • Non-Goals • Teach you how to debug your component • Technology sessions and whitepapers have in depth discussions and information about troubleshooting various OS technologies • Provide details about specific debugger commands • The documentation is the best source of information of all the debugger commands and features • These slides should not be used as debugger reference material

3. Session Outline • Debugger Installation and configuration • WinDbg UI • Remote debugging options • Dump Files • Dump file debugging • 64-bit debugging • Debugger core engine changes • Debugger commands • Debugger extensions

4. Debugger Teams Current Focus • Reliability • Managed code debugging • User-mode only technology today • Symbol server • !analyze • Customers satisfaction • Tell us what you need

5. Debugger Install • Always get the latest release • Latest releases are the most reliable, with most fixes • http://www.microsoft.com/whdc/ddk/debugging • The debugger in the DDK and SDK are out-of-date, only there for convenience • We support all target OS versions back to Windows NT 4.0 SP3 • We support all host OS versions back to Windows NT 4.0 SP3 • Both user mode and kernel mode debugging • Windows 9x kernel debugging is not supported • The package contains • Docs – debugger.chm – this is your friend! • Information about all debugger commands, extensions and tools • All the debuggers (WinDbg, CDB, NTSD, KD) and supporting DLLs • Extensions for all OS versions - right versions are loaded automatically • Variety of other debugger related tools • ADPlus, GFlags, KDbgCtrl, Logger, etc.

6. Common Issues • Local variables are wrong • Turn off compiler optimizations • Breakpoint issues • Bp sets a breakpoint on an address; Bu sets a breakpoint on a symbol • Use the “bu” command if you want to save breakpoints in the workspace, or want them to work across module load\unload\load • “I found a debugger problem” • Report it! We don’t release until all known bugs are fixed • 1394 debugging won’t connect • Disable 1394 host controller on the target • COM port debugging won’t connect • Disable legacy USB support in the BIOS • USB 2.0 debugging • Initial trial implementation is now available - requires a special USB cable

7. WinDbg Docked Windows • WinDbg windows can be floating or docked • Docked windows shrink and grow with the WinDbg frame they belong to • Docked window positions and sizes stay relative to each other as the frame changes • Undocked (a.k.a., floating) windows are always on top of the WinDbg window • WinDbg supports multiple docks • Individual WinDbg windows can be part of any dock • Can be moved to any dock at any time • New windows • We recommend all windows be docked • Use multiple docks on a multimon system • Ctrl-Tab iterates through all windows in all docks

8. WinDbg Tabbed Windows • WinDbg windows that are part of a dock can be tabbed • Tabbed windows are overlaid • They are resized as a group when an edge is dragged • When a tabbed window is dragged to another location, it is moved individually

9. WinDbg Windows Menus • Each window has its own menu • Menu is accessed by • Left click on the menu button (next to the close button) • Right click on the title bar of a window • Right click on the tab of a tabbed window • Disassembly Window menu • Highlight instruction from current source line – used to match all the assembly to a source line, providing the equivalent of a .cod file • Show source line \ source file for current instruction – display source information for each assembly line

10. WinDbg Demo • Docked windows • Multiple docks • Tabbed source windows • Highlight assembly instructions for current source line

11. Debugger Remoting • Very valuable if you have testing done in multiple remote locations • Provides a complete remoting experience • Can be enabled at startup or dynamically • -server, -remote, or .server to enable a remote client to connect • -premote to enable a process server the host can connect to • Works with mixed debugger: WinDbg connecting to CDB • See docs for extensive descriptions and details • Supports debugging across firewalls • Use debugger proxies

12. Targets • Kernel mode • Target is the machine being debugged. Multiple “transports” supported • COM port or 1394 • Works on all platforms • USB 2.0 support in the works • Trial binaries available in the current release • USB 2.0 debugging will have similar limitation to 1394 debugging • BIOS, PCI bridge, etc issues • Additionally requires special USB device (peer-to-peer networking device) • Local KD • Only useful for basic data inspection • No breakpoint support (no single machine debugging) • User mode • Machine where the process being debugged runs • Most often, host and target are the same • To separate the host from the target, use a “process server”, dbgsrv.exe • See docs for complete details

13. Host • Defined to be • The machine where the core debugger engine runs • The machine that needs access to the symbols • Windbg.exe, kd.exe, ntsd.exe, and cdb.exe can be host debuggers • They can also be configured to be remote debuggers • The host can connect to the target in many ways • windbg –k [other param] to connect to a kernel target • -k kdsrv: to attach to a KD connection server • windbg – p | pe | pn | pt to connect to a user mode target on the same machine • windbg –premote to connect to a process server

14. Remote Client And Proxies • The host must enable remoting by using .server or -server • Can be started dynamically • Multiple protocol supported, including • Named Pipe, TCP, SSL • The remote side connects using –remote • Example: cdb –remote tcp:server=1.1.1.1,port=1639 • All commands are sent from the remote to the host, executed on host, and output sent back to the target • WinDbg source file lookup (using .lscrpath) and other UI only commands (.cls, .open) are the only exception • Process server \ KD Connection server • Makes low-level target data remotely available for the host debugger • Proxy \ repeater • Machine running a proxy tool, sitting between a remote and a host, or between a host and target

15. User Mode Debugging: Non-Invasive Attach • In a normal debugging scenario, the debugger “attaches” to a process • OS feature that connects the debugger and debuggee • OS sends events to the debugger • Module loads, exceptions, etc. • Attaches do not always work • A debugger may already be attached to the process • Loader lock hang can cause attach to fail • Non-invasive attach is the debugger controlling the process externally • All threads are suspended • All data can be inspected • No standard events will be sent (no breakpoints)

16. User Mode Debugging: Multi-Process\System Debugging • The debugger can attach and debug multiple processes or systems simultaneously • .attach, .create can be used to attach to\create additional processes • Use ‘|s’ to switch between processes • .opendump can be used to open multiple dumps • .attach –premote can attach to a new process server • Multiple dumps or process servers are treated as different “systems” • Use ‘||s’ to switch between systems

17. Kernel Dump Files • OS can generate 3 different types of dumps • Full (all physical memory) • Kernel (pages mapped in the kernel address space) • Minidump (64 K) • Header, basic OS information, PRCB • OS Module list (loaded and unloaded) • Faulting Process, Thread, Stack and context • Data pages pointed to by the context and bugcheck params • The kernel can also store additional data in minidumps by calling IoAddTriageDumpDataBlock() • Future addition of SMBIOS information • Kernel minidumps only store data – no image information • The debugger requires access to images to debug a minidump • OS also provides callbacks to drivers so they can store additional state (added to all dumps)

18. Dump File Generation • Operating system generates the crash time at the time of the crash • Raised IRQL – no code can be paged in • Special path in the disk drivers to support writing dump files • Dump files get written to the page file – guaranteed, locked down sectors on disk • Type of dumps generated by the OS : • Windows XP client: minidump is the default • Windows Server 2003: kernel dump is the default • Default can be reconfigured by the end-user • When Full or Kernel dumps are generated by the OS, upon reboot a minidump is created from the original dump • Called “Dump conversion” • Equivalent to using “.dump /m” in the debugger • OCA error reports always contain the minidump

19. Debugging A Minidump • Debuggers: • Kernel minidumps require using KD or WinDbg • Both WinDbg and VS supports debugging user mode minidumps • Step 1: Get the images • A minidump contains minimal data. The code streams are not included, except for the code page pointed to by eip or bugcheck params • All Microsoft kernel mode code for recent OS is on the Internet symbol server • You need to the latest debugger to get access to the images • Step 2: Debugger maps the images and merges the images with the data stored in the minidump • Debug info from the images is used to look for the symbols • If you have the wrong image, wrong symbols will be loaded • Step 3: Get symbols • Symbol server is again the best solution

20. Symbol Loading • http://msdn.microsoft.com/download/symbols is the location where we try to get every public symbol file in Microsoft indexed • We work with many groups across Microsoft to get all the files in this unique location • We store compressed files on the server • The debugger needs to use a local cache to store the symbol files • The debugger automatically populates the local cache specified in the debugger • Full access to the Internet symbol server requires debugger 6.2.13 or later • New EULA must be accepted • Managing multiple versions of symbols and images can be hard • Store all your binaries in your own symbol server • Designed to store all versions of images and symbols you create. • Based on timestamps for images and GUID for PDBs • SymStore – tool to build a symbol server • Easily scripted • Run it as part of the build process

21. Source Code Loading • KD \ NTSD \ CDB • Source code is loaded on the host machine • .scrpath to set the path • WinDbg • WinDbg can load its own version of the source code • Use .lsrcpath when connecting remotely to get local source access • Source code need not be loaded on the same machine • .srcpath | .lsrcpath \\<machine_name>\share\<root_of_source_tree> • The debugger engine does path matching to find the actual files in the source tree, if the paths are close to identical to where the source code was built • If the paths are significantly different, include each directory where source is located

22. Source Server • New feature in the debugger to automatically load the correct version of the source code automatically • Set your srcpath to ‘srv*’ • How does it work? • The build process must runs tools to encode the version of the source code into a PDB • When debugging, the debugger extracts the source code information from the PDB • The debugger calls the source control system to extract a copy of that source file on your machine • Requires the debugger machine have the source control tools installed • Requires the machine and user account have source access • Further documentation in the debugger package

23. !analyze • Debugger extension designed to categorize bugs • Automated analysis • Simplifies analysis of known problems • Understand various states of the OS • Provides starting point to analyze complex problems • Extracts commonly used debugging information • Applies common triaging techniques and well known analysis steps • Results of the analysis are • “Bucket ID” • Unique string representing the bug • An Owner for the problem, extracted from triage.ini • In verbose mode, using !analyze -v • Detailed list of all the data found during the analysis

24. !analyze Algorithm • Use bugcheck parameters to extract basic information • Scan the stack for special functions such as Trap0E • Analyze frames on the final stack to determine most likely culprit • Different weights are assigned to routines • Highest weight frame found on the stack is treated as the culprit • If stack does not yield an interesting frame, analyze raw stack data • Check for presence of memory or pool corrupting drivers • Hardware problems are quite common – no way to find these from a dump file today • Check for corrupted code streams (possible bad RAM) • Check for other possible problems, such as invalid call sequences • Bad CPU, chipset problems • Bad disk

25. 64-Bit Debugging – What’s The Same? • WinDbg UI is identical • All debugging features work on both 32-bit and 64-bit • All debugger commands and extensions work on both 32-bit and 64-bit • Exceptions are processor specific debugger commands and extensions • MSRs, !pcr • Stepping, breakpoints work identically • Dump files work identically • Symbols work identically • Symbols are specific to the compiler and linker version • All general OS data structures should be identical • Again processor specific data structures (CONTEXT, PTE) will differ • Some macros that use processor specific features are different

26. 64-Bit Debugging – What’s Different ? • Pointers are 64 bits instead of 32 • Be careful when casting pointers. • User dq to dump quadwords • Assembly languages are unique to each processor • x64 assembly is very similar to x86 • IA64 is totally different • Instructions are bundled 3 at a time, predicates, etc … • Calling conventions are unique to each processor • x64 passes first 4 params in registers • IA64 supports rotating registers • Source level debugging should hide all these differences • The OS tries to abstract all processor and platform specific features • Looking at PTEs and other processor specific structures internal to the OS will expose differences

27. KDbgCtrl.exe • New tool, works on Windows Server 2003 and above • Tool to manipulate kernel debugging options on the target machine; Examples • du - Disable kernel debugger user exception handling • d - Disable kernel debugger • sdb <size> - Set kernel DbgPrint buffer size • See docs for complete details • Windows Server 2003 and later • Newer features only work on the latest OS

28. Core Debugger Changes • Unicode support • Changed dbghelp.dll and dbgeng.dll to be fully Unicode • Symbols and file names can be in Unicode • WinDbg UI is unicode (CDB, KD remain ANSI) • Improved stack walking • New support for “Debugger programs”

29. Managed Code Support • Requires Windows Longhorn build • Symbols looked up by address • Correct source lines looked up by address • Managed frames display locals • Local variables are correct for primitive types and the raw objref for complex types • ?? can get the address of a local by name • Symbols will be resolved to addresses for jitted code and static members of loaded types • x can be used with managed modules and symbols • Managed exception info is displayed for second-chance CLR exceptions • F9 in WinDbg can be used to set breakpoints • sx now has a 'clr' option for controlling the exception used by the CLR for managed exceptions • Many more features to come in future version of WinDbg and longhorn

30. Debugger Commands • Read the docs for information on all commands • .formats: Converts numbers to timestamps • bm: sets breakpoints on all commands that match the wildcard • bp /p <EPROCESS/PEB> • .enable_unicode: Show ushorts as chars • .enable_long_status: Show signed int as hex • .frame: Switch the current frame on a stack • .dump: Generates a dump file of the target machine by reading data from the target • Kernel full dumps should only be done over 1394 • .pcmd: allows addition to the debugger prompt • example .pcmd –s !thread

31. Interesting Debugger Commands • kP – Shows all function parameters, each on one line • As accurate as DV • Different algorithm as kb 0:000> kP ChildEBP RetAddr 0006fb34 77f6492b ntdll!DbgBreakPoint 0006fc90 77f55349 ntdll!LdrpInitializeProcess( struct _CONTEXT * Context = 0x0006fd30, void * SystemDllBase = 0x77e60000, struct _UNICODE_STRING * UnicodeImageName = 0x0006fce8, unsigned char UseCOR = 0xb4 '', unsigned char ImageFileOptionsPresent = 0x00 '') +0xdf3 0006fd1c 77f75d87 ntdll!LdrpInitialize( struct _CONTEXT * Context = 0x01000108, void * SystemArgument1 = 0x001a0019, void * SystemArgument2 = 0x77f630de) +0x186 00000000 00000000 ntdll!KiUserApcDispatcher+0x7

32. Interesting Commands • lm [options]: detailed information about a module (replaces !drivers) kd> lmv mntos* start end module name 00400000 00618080 ntoskrnl (private pdb symbols) c:\localsym\ntoskrnl.pdb\DD3B00F5D69B4D1D8DADA85166188F862\ntoskrnl.pdb Loaded symbol image file: ntoskrnl.exe Mapped memory image file: c:\localsym\ntoskrnl.exe\404D5566218080\ntoskrnl.exe Image path: \\winbuilds\release\XPSP2\2094\usa\x86fre\bin\ntoskrnl.exe Timestamp: Mon Mar 08 21:25:58 2004 (404D5566) Checksum: 0021AF90 ImageSize: 00218080 File version: 5.1.2600.2094 Product version: 5.1.2600.2094 File flags: 0 (Mask 3F) File OS: 40004 NT Win32 File type: 1.0 App File date: 00000000.00000000 Translations: 0409.04b0 CompanyName: Microsoft Corporation ProductName: Microsoft« Windows« Operating System InternalName: ntoskrnl.exe OriginalFilename: ntoskrnl.exe ProductVersion: 5.1.2600.2094 FileVersion: 5.1.2600.2094 (xpsp_sp2_rc1.040308-1920) FileDescription: NT Kernel & System LegalCopyright: Microsoft Corporation. All rights reserved.

33. Interesting Commands • ln: Shows source and symbolic information about the specified code address kd> ln 4bf339 d:\xpsp_rc1\base\ntos\mm\sysload.c(459) Source Depot: depot:2003 //depot/xp/base/ntos/mm/sysload.c#1 (004bf339) ntoskrnl!MmLoadSystemImage | (004bf53f) ntoskrnl!CmpQueryKeyName Exact matches: ntoskrnl!MmLoadSystemImage(UNICODE_STRING*, _UNICODE_STRING*, UNICODE_STRING*, unsigned long, void**, void**) • !address: Shows general information about an address kd> !address 77f75a58 77f50000 : 77f51000 - 00070000 Type 01000000 MEM_IMAGE Protect 00000020 PAGE_EXECUTE_READ State 00001000 MEM_COMMIT Usage RegionUsageImage FullPath ntdll.dll

34. Debugger Commands • .kdfiles • Use the debugger to transfer updated versions of drivers over debug port to replace the version currently installed on the target • On each OS Mod load operation, the target checks with the debugger if an updated driver is present • Normally only works for regular drivers (loaded by MmLoadSystemImage). You need to use the special boot loader with debugger enabled to change boot drivers • wt: single step trace of a function and subfunctions • Numerous improvements to make the output more useful • Option to control depth of tracing, increasing performance • .call: Execute functions in the target process\system • BE VERY CAREFUL with this command. It can easily cause deadlocks in the target process • .call should not be used as a replacement for debugger extension

35. Aliases • Aliases are character strings that are automatically replaced with other character strings • User-named aliases are set and are named by the user • Fixed-name aliases are set by the user, and are named $u0, $u1, ..., $u9 • Commands to manipulate user named aliases • as sets an alias • ad clears an alias • al lists all aliases • Commands to manipulate fixed named aliases • r sets an alias • Aliases are always stored as strings • String replacement is done before any evaluation is performed on the command \ expression

36. Debugger Expressions • Pseudo registers: Special symbols recognized by the debugger which contain specific, common values • Examples: $thread – current thread, $peb – current peb (user mode), $ra – return address • ?? *@$teb will dump the current TEB • r? can be used to assign types information to a pseudo register ($t0) • ?? : C++ expression evaluator • Supports fully typed expressions – pointers, classes, arrays • Special handling for some preprocessor tokens CONTAINING_RECORD 0:000> ?? LdrDataTableEntry struct _LDR_DATA_TABLE_ENTRY * 0x6fc28 0:000> ?? ((ntdll!_LDR_DATA_TABLE_ENTRY *)0x6fc28)->FullDllName struct _UNICODE_STRING +0x000 Length : 0x30 +0x002 MaximumLength : 0x7ffe +0x004 Buffer : 0x00010000 "=::=::\“ 0:000> ?? LdrDataTableEntry->FullDllName.Buffer[5] unsigned short 0x3a

37. Debugger Command Programs • A debugger command program is a small program consisting of debugger commands along with control flow tokens such as .if, .for, and .while • This a type of scripting. NOTE: in the debugger docs the term “script” refers to list of commands stored in a file which are passed to the debugger • Lets you build complex commands without having to write a whole debugger extension • A debugger command program can be executed from the command prompt, or stored in a file • Complete documentation about all the supported tokens and programming rules can be found in the docs

38. Example Command Program $$ Walking the LDR_DATA_TABLE_ENTRY List $$ Get module list LIST_ENTRY in $t0. r? $t0 = &@$peb->Ldr->InLoadOrderModuleList $$ Iterate over all modules in list. .for (r? $t1 = *(ntdll!_LDR_DATA_TABLE_ENTRY**)@$t0; (@$t1 != 0) & (@$t1 != @$t0); r? $t1 = (ntdll!_LDR_DATA_TABLE_ENTRY*)@$t1->InLoadOrderLinks.Flink) { $$ Get base address in $Base. as /x ${/v:$Base} @@c++(@$t1->DllBase) $$ Get full name into $Mod. as /msu ${/v:$Mod} @@c++(&@$t1->FullDllName) .block { .echo ${$Mod} at ${$Base} } ad ${/v:$Base} ad ${/v:$Mod} }

39. Scripting • List of commands that can be passed as input to the debugger • Designed for common series of commands to be run in automation scenarios • Commands are read line by line, as input is needed by the debugger • Script don’t allow for control flow (loops branching) • Because of this, they do allow for embedded ‘g’ ‘p’ ‘t’ operations • To load and run a script • Use –c command line option • Use $<, $><, $$<, and $$>< • Example of Scripting • cdb –remote <params> –c “.remote_exit; .logopen foo;!analyze –v;.logclose;.shell copy foo \\share;.reboot”

40. Debugger Extensions • New debugger extensions • New !address (still under development): Given an address, described what type of memory it is (pool, code, heap, etc.) • New !chkimg: Analyzes code streams for possible corruption • Requires access to the original image • Compares the in-memory image to the disk image • New 0x10 flag to !process to get accurate user mode symbols for all processes • Significant updates to the longhorn version of ndiskd • Numerous minor extensions fixes and changes based on developer feedback • FYI If you load your own debugger extensions • The version command shows the list of loaded debugger extensions • Commands are searched for in each DLL, in order • Be careful of duplicate command names

41. Writing Debugger Extensions • Samples, headers, and documentation all part of the debugger package • Not installed by default. You must do a custom install and select the optional SDK component. • WDbgExts.h is the old debugger extension interface • Still widely used, but limited in functionality • DbgEng.h is the full-blown debugger API • Hundreds of methods to control all aspects of debugging • See documentation (debugext.chm) for full details • Separate presentation on writing WinDbg debugger extensions

42. Call To Action • Read the debugger docs! • Send us feedback – about the tool or the docs • Look at your OCA failures • These are REAL customer problems • Fix your pool corruption problems • Tell us about the bugs you fix, so we can update !analyze and point customers to your driver updates • Attend the OCA Crash Analysis session, following this session • Attend the WinDbg Ask the Experts sessions

43. Resources • Debugger URL and download site • http://www.microsoft.com/whdc/ddk/debugging • Debugger e-mail – for debugger bug reports and feature requests • We try to fix all the bugs people report • We do not provide general debugging support on this alias • Debugger newsgroup • Microsoft.public.windbg • Good place for general debugging issues • Windbgfb @ microsoft.com