Kdump: The kexec approach to crashdumps in Linux

1. Kdump: The kexec approach to crashdumps in Linux Hariprasad Nellitheertha, Linux Technology Center, ISL Linux-Bangalore/2004 Bangalore, India To replace the title / subtitle with your own: Click on the title block -> select all the text by pressing Ctrl+A -> press Delete key -> type your own text To replace the title / subtitle with your own: Click on the title block -> select all the text by pressing Ctrl+A -> press Delete key -> type your own text

2. Agenda Introduction Linux and crash dumps Need for kdump Design of kdump Advantages & Limitations Status & To dos A pot file is a Design Template file, which provides you the �look� of the presentation You apply a pot file by opening the Task Pane with View > Task Pane and select Slide Design � Design Templates. Click on the word Browse� at bottom of Task Pane and navigate to where you stored BlueOnyx Deluxe.pot (black background) or BluePearl Deluxe.pot (white background) and click on Apply. You can switch between black and white background by navigating to that pot file and click on Apply. Another easier way to switch background is by changing color scheme. Opening the Task Pane, select Slide Design � Color Schemes and click on one of the two schemes. All your existing content (including Business Unit or Product Names) will be switched without any modification to color or wording. Start with Blank Presentation, then switch to the desired Design Template Start a new presentation as Blank Presentation You can switch to Blue Onyx Deluxe.pot by opening the Task Pane with View > Task Pane and select Slide Design � Design Templates. Click on the word Browse� at bottom of Task Pane and navigate to where you stored BlueOnyx Deluxe.pot (black background) and click on Apply. Your existing content will take on Blue Onyx�s black background, and previous black text will turn to white. You should add your Business Unit or Product Name by modifying it on the Slide Master You switch to the Slide Master view by View > Master > Slide Master. Click on the Title Page thumbnail icon on the left, and click on the Business Unit or Product Name field to modify it. Click on the Bullet List Page thumbnail icon on the left, and click on the Business Unit or Product Name field to modify it. Click on Close Master View button on the floating Master View Toolbar You can turn on the optional date and footer fields by View > Header and Footer Suggested footer on all pages including Title Page: Presentation Title | Confidential Date and time field can be fixed, or Update automatically. It appears to the right of the footer. Slide number field can be turned on as well. It appears to the left of the footer.A pot file is a Design Template file, which provides you the �look� of the presentation You apply a pot file by opening the Task Pane with View > Task Pane and select Slide Design � Design Templates. Click on the word Browse� at bottom of Task Pane and navigate to where you stored BlueOnyx Deluxe.pot (black background) or BluePearl Deluxe.pot (white background) and click on Apply. You can switch between black and white background by navigating to that pot file and click on Apply. Another easier way to switch background is by changing color scheme. Opening the Task Pane, select Slide Design � Color Schemes and click on one of the two schemes. All your existing content (including Business Unit or Product Names) will be switched without any modification to color or wording. Start with Blank Presentation, then switch to the desired Design Template Start a new presentation as Blank Presentation You can switch to Blue Onyx Deluxe.pot by opening the Task Pane with View > Task Pane and select Slide Design � Design Templates. Click on the word Browse� at bottom of Task Pane and navigate to where you stored BlueOnyx Deluxe.pot (black background) and click on Apply. Your existing content will take on Blue Onyx�s black background, and previous black text will turn to white. You should add your Business Unit or Product Name by modifying it on the Slide Master You switch to the Slide Master view by View > Master > Slide Master. Click on the Title Page thumbnail icon on the left, and click on the Business Unit or Product Name field to modify it. Click on the Bullet List Page thumbnail icon on the left, and click on the Business Unit or Product Name field to modify it. Click on Close Master View button on the floating Master View Toolbar You can turn on the optional date and footer fields by View > Header and Footer Suggested footer on all pages including Title Page: Presentation Title | Confidential Date and time field can be fixed, or Update automatically. It appears to the right of the footer. Slide number field can be turned on as well. It appears to the left of the footer.

3. Crash Dumps Snapshot of system At the time of failure Manually triggered Dump contents Snapshot of memory pages Register values Dumping could be disruptive or non-disruptive System rebooted after disruptive dumps Typically, dump is written out to disk or across network

4. Approaches to Reliable Crash Dumping Stand-Alone Dumpers E.g.: mini kernel dump Dump-specific driver interfaces Support for every target device needed Multitude of devices/adapters in Linux Maintenance is a big problem E.g.: Diskdump, Netdump Mini kernel dumper Runs at a separate privilege level E.g.: Adeos

5. The need for kdump Limitations of existing solutions Filtering not possible with stand-alone dumpers Need to maintain device specific code for dump drivers Dump write-out in the failed kernel Complex code base Invasive into the core kernel Installation and set-up is cumbersome

6. Design Goals of kdump Simple and minimally invasive into the kernel code Highly reliable Needs to dump under any situation Available on most architectures Maintenance must be easy Flexibility in terms of dump contents and targets Full dump or kernel-pages only dump Dump to disk or across the network Ease of use Installation, set-up and dump write-out must be easy to perform

7. Kernel-to-kernel bootloaders Running kernel acts as loader for the new kernel System directly jumps from one kernel to another Skips BIOS or firmware stage Reboots are extremely fast Memory can be preserved across reboots Since BIOS is skipped, it is left to the OS to retain or erase memory Finds application in crash dumping tools Memory, preserved across reboots, is essentially a dump

8. Kexec Allows a Linux kernel to boot another kernel Kexec can load and boot any ELF binary image Two components User space � kexec-tools Kernel patch Load new kernel kexec -l <kernel-image> �append=<options> Exec new kernel kexec -e

10.

11. Memory Preserving Reboot First kernel reserves a section of memory Performed very early during first kernel's boot Second kernel boots into this region Upon panic, �backup� necessary data Send NMI IPI to each processor NMI handler performs two actions Snapshot register values Halt the processor Backup memory that's used by second kernel 0-640K Perform kexec reboot

13. Post Reboot Second kernel boots with very little memory Uses the reserved memory region Size configurable at compile time Flag passed to second kernel Indicates crash has occurred Rest of memory treated as a high memory device Not mapped in the page tables Access requires a pte to be created

15. /proc View During proc fs initialization, /proc/vmcore is created Only if reboot is due to a crash Binary file whose size equals size of dump File contents are dump pages Dump treated as an ELF �Core� format file Similar to how kcore treats kernel memory Program header section maps dump pages Notes section maps register snapshots Dump write-out using existing tools #cp /proc/vmcore dump.0

16. Linear View Provides an unformatted view of the dump Interface is /dev/oldmem Device needs to be created Major number 1, minor number 12 #mknod /dev/oldmem c 1 12 Quick and easy way to read selective portions Can be read using �dd� #dd if=/dev/oldmem of=dump.0 bs=<> count=<> skip=<>

18. Reading Dump Pages Done when user reads proc file or oldmem Page frame number calculated Based on user supplied file-offset Fix virtual address for the dump page Create a temporary pte Copy dump page to user buffer Ensure correct ordering of dump pages Read requests for initial region returned with contents from backup region

19. Benefits Minimal processing in the failed kernel Mostly limited to backing up memory Dump processing deferred to second kernel Simple code base in both kernel and user space Kernel � doesn't deal with write-out or myriad of targets User space � existing tools such as cp, gdb can be used Does not need a dedicated dump device Installation, Set-up and Configuration extremely simple

20. Benefits (contd) Dump write-out very easy Any file copy tool will suffice Dump can be written out to disk or across network Network dumping can use standard protocols No special catcher programs needed Dumping across sub-nets not an issue GDB can be used to analyse dumps Minimal debugging can be performed on /proc/vmcore itself No need for dump write-out in such cases

21. Limitations Additional reboot required Custom kernel may be needed if downtime is a concern Non-disruptive dumps not possible Needs adaptation for embedded environments Memory constraints may restrict applicability Will be tied to availability of fast-reboot features on different architectures

22. Status Currently available for Intel x86 platform Patches available in the -mm tree Minimal analysis can be performed on the dump Stack tracing, register contents Examining memory Work in progress to add several features

23. TODOS Selective Dumping Kernel-pages only dump Support for NUMA and discontiguous memory Device shutdown prior to reboot Enable Red Hat's crash tool to recognize this format Port to different architectures X86_64, ppc64 and ia64

24. Usage Obtain latest -mm patch Build two kernels with kexec/kdump enabled Second kernel to have different kernel load address From first kernel, load dumping kernel Exact command available in Documentation/kdump.txt Enable kdump echo 1 > /proc/kdump Upon panic, system reboots Write-out dump file cp /proc/vmcore dump.x

25. Disclaimers and Trademarks This work represents the view of the author and does not necessarily represent the view of IBM. IBM and IBM(logo) are trademarks of International Business Machines Corporation in the United States, other countries, or both. Intel is a trademark of Intel Corporation in the United States, other countries or both. Red Hat is a trademark of Red Hat, Inc. In the United States, other countries or both. Linux is a registered trademark of Linus Torvalds in the U.S., other countries, or both. Other company, product, or service names may be trademarks or service marks of others.