mohitmishra786/low-level-dev-skills / skills/kernel/kernel-debugging
kernel-debugging
UnverifiedLinux kernel debugging skill for kgdb, kdb, ftrace, kprobes, and crash analysis. Use when setting up kgdb over serial, using ftrace and trace-cmd, inserting kprobes, enabling dynamic debug, or analyzing kdump with crash. Activates on queries about kgdb, kdb, ftrace, kprobes, dyndbg, kdump, or kernel dmesg levels.
Kernel Debugging
Purpose
Guide agents through debugging the Linux kernel: kgdb over serial or USB, kdb built-in debugger commands, ftrace and trace-cmd, kprobes and kretprobes, dynamic debug (dyndbg), crash dump analysis with crash and makedumpfile, dmesg log levels, and /proc/sys runtime tuning for debugging.
When to Use
- Kernel panic or oops with unclear stack trace
- Live debugging a kernel module or driver with kgdb
- Tracing function calls without recompiling (ftrace, kprobes)
- Enabling verbose driver logging with dyndbg
- Analyzing production crash dumps from kdump
- Tuning kernel runtime parameters for debug sessions
Workflow
1. dmesg and log levels
# Kernel ring buffer
dmesg -T -l err,warn
dmesg -w # follow live
# Log level (0=emerg .. 7=debug)
cat /proc/sys/kernel/printk
# current default minimum boot-default
# 4 4 1 7
# Temporarily increase verbosity
echo 8 > /proc/sys/kernel/printk
Kernel printk levels: KERN_EMERG through KERN_DEBUG. Driver dev_dbg() requires dyndbg or DEBUG define.
2. kgdb over serial
Kernel config: CONFIG_KGDB, CONFIG_KGDB_SERIAL_CONSOLE, CONFIG_MAGIC_SYSRQ.
[!WARNING] Debugger entry stops the target kernel and can disrupt its workloads. Identify the exact test target and obtain explicit user approval for the interruption before entering kgdb or kdb; otherwise stop and ask.
# Boot parameter for live debugger entry (ttyS0, 115200)
kgdboc=ttyS0,115200
With the target's kgdboc I/O driver configured and keyboard SysRq enabled, use SysRq-g for live entry.
If /proc/sys/kernel/sysrq disables keyboard SysRq, the privileged /proc/sysrq-trigger interface still works.
Only after the exact target and interruption are explicitly approved, run this in a root shell on that target:
echo g > /proc/sysrq-trigger
For boot-time entry, build kgdboc into the kernel and place kgdbwait after its configuration:
kgdboc=ttyS0,115200 kgdbwait
This stops the kernel during boot, under the same target identification and interruption approval requirements.
Resume it with GDB's continue (or kdb's go) before using SysRq-g for later live entry.
See the kernel kgdb/kdb documentation for transport setup and entry methods.
GDB host side:
# Connect to remote serial
gdb vmlinux
(gdb) set serial baud 115200
(gdb) target remote /dev/ttyUSB0
(gdb) bt
(gdb) list *panic+0x20
USB variant: CONFIG_KGDB_KDB with kgdboc=usb on supported hardware.
3. kdb commands
When CONFIG_KGDB_KDB enabled, kdb provides in-kernel shell:
kdb commands (at SysRq break)
├── bt — stack backtrace
├── lsmod — loaded modules
├── md <addr> — memory display
├── ps — process list
├── id <cpu> — CPU registers
├── go — continue execution
└── cpu <n> — switch CPU context
With CONFIG_KGDB, CONFIG_KGDB_KDB, CONFIG_MAGIC_SYSRQ, and a configured kgdboc console, enter kdb using SysRq-g only after the target and interruption are explicitly approved as above.
If it enters kgdb mode and GDB is not connected, type $3#33 in one pass on the configured debugger console
to switch to kdb; do not use backspace or delete.
SysRq-k is the Secure Access Key (SAK): it kills programs on the current virtual console, not debugger entry.
See the kernel SysRq command reference.
4. ftrace
# Enable function tracer
echo function > /sys/kernel/debug/tracing/current_tracer
echo 1 > /sys/kernel/debug/tracing/tracing_on
# Filter to specific function
echo schedule > /sys/kernel/debug/tracing/set_ftrace_filter
echo ':*probe_*' >> /sys/kernel/debug/tracing/set_ftrace_notrace
# Capture and read
cat /sys/kernel/debug/tracing/trace_pipe
# Disable
echo 0 > /sys/kernel/debug/tracing/tracing_on
echo nop > /sys/kernel/debug/tracing/current_tracer
# trace-cmd (userspace recorder)
trace-cmd record -p function -l schedule,do_page_fault
trace-cmd report
trace-cmd.dat # kernelshark GUI
5. kprobes and kretprobes
# Dynamic kprobe via debugfs
echo 'p:myprobe do_sys_open %ax %si' > /sys/kernel/debug/tracing/kprobe_events
echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
cat /sys/kernel/debug/tracing/trace
# kretprobe
echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events
# Cleanup
echo 0 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
echo '-:myprobe' > /sys/kernel/debug/tracing/kprobe_events
In-kernel kprobe (module):
#include <linux/kprobes.h>
static struct kprobe kp = {
.symbol_name = "do_sys_open",
.pre_handler = my_pre_handler,
};
static int __init mod_init(void)
{
return register_kprobe(&kp);
}
6. Dynamic debug (dyndbg)
# Enable all debug messages for a module
echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control
# Enable specific file:line
echo 'file drivers/i2c/i2c-core.c +p' > /sys/kernel/debug/dynamic_debug/control
# Query current settings
grep mydriver /sys/kernel/debug/dynamic_debug/control
Boot-time: dyndbg="module mydriver +p" on kernel command line.
7. kdump and crash analysis
# Setup kdump (crash kernel reserved memory)
# /etc/default/grub: crashkernel=256M
sudo kdump-config show
# After crash, vmcore in /var/crash/
ls /var/crash/*/vmcore
# Analyze with crash utility
crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/vmcore
crash> bt
crash> log
crash> ps
crash> kmem -i
crash> mod
# Compress vmcore
makedumpfile -c -d /proc/vmcore /tmp/vmcore.lzo
8. /proc/sys runtime tuning for debugging
# Panic on oops (reproduce in VM)
sysctl kernel.panic_on_oops=1
# Soft lockup detection
sysctl kernel.softlockup_panic=1
# Watchdog
sysctl kernel.nmi_watchdog=1
# Slab debugging
# Boot: slub_debug=P,pagealloc
9. Debugging decision tree
Kernel issue type?
├── Panic/oops → dmesg; crash vmcore; CONFIG_DEBUG_INFO_BTF
├── Logic bug in driver → dyndbg; ftrace function_graph
├── Performance regression → perf record -g -a; trace-cmd
├── Intermittent → kprobes on suspect path
└── Module crash → kgdb; try_module_get audit
Common Problems
| Symptom | Cause | Fix |
|---|---|---|
| kgdb won't connect | Wrong tty/baud | Match kgdboc to serial adapter |
| ftrace empty trace | tracing_off or nop tracer | echo function > current_tracer; enable tracing_on |
| kprobe registration fails | Missing CONFIG_KPROBES or inlined function | Use tracepoint or static_key |
| dyndbg has no effect | CONFIG_DYNAMIC_DEBUG disabled | Rebuild kernel with option |
| crash can't read vmcore | Wrong vmlinux debug symbols | Install linux-image-$(uname -r)-dbg |
| SysRq doesn't work | kernel.sysrq disabled | echo 1 > /proc/sys/kernel/sysrq |
Related Skills
skills/low-level-programming/linux-kernel-modules— module development being debuggedskills/kernel/device-drivers— driver-specific probe/IRQ issuesskills/kernel/kernel-internals— subsystem knowledge for interpreting tracesskills/debuggers/gdb— GDB commands shared with kgdbskills/profilers/linux-perf— perf for kernel hotspotsskills/observability/ebpf— non-invasive kernel tracing alternative
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 318k | Updated | Claude CodeCodex | Unverified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 318k | Updated | Claude CodeCodex | Unverified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 318k | Updated | Claude CodeCodex | Unverified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |