LWM-Linux/11 - Linux Kernel and Modules/Kernel Debugging.md

Kernel Debugging

1. Preparation

Set up a development environment:

• Install a Linux distribution (e.g., Ubuntu, Fedora)
• Install essential development tools: gcc, make, binutils, etc.
• Download kernel source code from kernel.org

Configure kernel for debugging:

• Enable CONFIG_DEBUG_INFO in kernel configuration
• Enable CONFIG_KGDB for remote debugging
• Enable other relevant debug options (e.g., CONFIG_DEBUG_KERNEL)

Build the kernel with debug symbols:

make menuconfig
make -j$(nproc)

2. Kernel Debugging Techniques

Printk Debugging:

• Use printk() function to add log messages
• Set appropriate log levels (KERN_INFO, KERN_DEBUG, etc.)
• View logs using dmesg or /var/log/kern.log

Kernel Oops and Panic Analysis:

• Analyze oops messages in kernel logs
• Use addr2line to translate addresses to source code lines
• Examine call traces to identify the source of the problem

KGDB (Kernel GNU Debugger):

• Configure a serial connection or network for remote debugging
• Set up a GDB client on the host machine
• Connect to the target machine running the kernel
• Set breakpoints, step through code, and examine variables

Ftrace (Function Tracer):

• Enable Ftrace in kernel configuration
• Use trace-cmd tool to collect and analyze function call data
• Examine function graph to understand kernel execution flow

Systemtap:

• Write custom scripts to instrument the kernel
• Collect detailed performance data and debug information
• Analyze complex scenarios without modifying kernel source

Kernel Probes (Kprobes):

• Insert dynamic breakpoints in the kernel
• Gather debugging and performance information
• Use jprobes for function entry probing and kretprobes for return probes

3. Advanced Debugging Techniques

Memory Debugging:

• Use KASAN (Kernel Address Sanitizer) to detect memory errors
• Enable CONFIG_DEBUG_PAGEALLOC to catch use-after-free bugs
• Utilize SLUB debug features for slab allocator issues

Lockdep (Lock Dependency) Checker:

• Enable CONFIG_PROVE_LOCKING and CONFIG_DEBUG_LOCKDEP
• Detect potential deadlocks and incorrect locking behavior
• Analyze lock dependency graphs

RCU (Read-Copy-Update) Debugging:

• Enable CONFIG_RCU_TRACE for RCU tracing
• Use rcutorture module to stress-test RCU implementations
• Analyze RCU grace period and callback execution

Kernel Live Patching:

• Use kpatch or livepatch to apply runtime fixes
• Debug issues related to live patching mechanisms
• Verify patch consistency and function replacement

4. Tools and Utilities

Crash utility:

• Analyze kernel core dumps
• Examine kernel data structures and memory
• Useful for post-mortem analysis of kernel panics

Perf:

• Profile kernel and user space performance
• Identify hotspots and performance bottlenecks
• Analyze hardware events and software tracepoints

eBPF (extended Berkeley Packet Filter):

• Write eBPF programs for advanced tracing and debugging
• Use bpftrace for one-liners and short scripts
• Develop custom eBPF tools for specific debugging needs

Kdump:

• Configure and enable kernel crash dump mechanism
• Capture memory dumps on kernel panics
• Analyze dumps using crash utility or GDB

5. Best Practices

Use git for version control:

• Maintain a clean development history
• Utilize git bisect for identifying problematic commits

Maintain a test environment:

• Use virtual machines or dedicated hardware for testing
• Set up automated testing frameworks (e.g., KernelCI)

Collaborate with the community:

• Subscribe to relevant mailing lists (e.g., LKML)
• Share and discuss issues on appropriate forums
• Submit well-formatted patch series for review

Document your findings:

• Maintain detailed notes of debugging sessions
• Create reproducible test cases for bugs
• Write clear bug reports and patch descriptions

6. Debugging Real-world Scenarios

Boot issues:

• Use early printk for early boot problems
• Analyze initcall debugging output
• Utilize kernel command line parameters for troubleshooting

Device driver debugging:

• Use debugfs to expose driver information
• Implement IOCTL commands for debugging
• Utilize kernel's tracepoints in the driver code

File system issues:

• Enable verbose mount options
• Use debugfs to examine file system internals
• Analyze block layer tracing for I/O related problems

Network stack debugging:

• Use network sniffer tools (e.g., tcpdump, Wireshark)
• Enable netconsole for remote kernel debugging
• Analyze netfilter and netlink debugging information