mohitmishra786/low-level-dev-skills / skills/kernel/kernel-debugging
kernel-debugging
VerifiedLinux kernel debugging skill for kgdb, kdb, ftrace, kprobes, and crash analysis. Use when setting up kgdb over serial, using ftrace and trace-cmd, inserting kprobes, enabling dynamic debug, or analyzing kdump with crash. Activates on queries about kgdb, kdb, ftrace, kprobes, dyndbg, kdump, or kernel dmesg levels.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 7.7 KB |
SKILL.md
View on GitHub---
name: kernel-debugging
description: Linux kernel debugging skill for kgdb, kdb, ftrace, kprobes, and crash analysis. Use when setting up kgdb over serial, using ftrace and trace-cmd, inserting kprobes, enabling dynamic debug, or analyzing kdump with crash. Activates on queries about kgdb, kdb, ftrace, kprobes, dyndbg, kdump, or kernel dmesg levels.
---
# Kernel Debugging
## Purpose
Guide agents through debugging the Linux kernel: kgdb over serial or USB, kdb built-in debugger commands, ftrace and `trace-cmd`, kprobes and kretprobes, dynamic debug (`dyndbg`), crash dump analysis with crash and makedumpfile, dmesg log levels, and `/proc/sys` runtime tuning for debugging.
## Triggers
- Kernel panic or oops with unclear stack trace
- Live debugging a kernel module or driver with kgdb
- Tracing function calls without recompiling (ftrace, kprobes)
- Enabling verbose driver logging with dyndbg
- Analyzing production crash dumps from kdump
- Tuning kernel runtime parameters for debug sessions
## Workflow
### 1. dmesg and log levels
```bash
# Kernel ring buffer
dmesg -T -l err,warn
dmesg -w # follow live
# Log level (0=emerg .. 7=debug)
cat /proc/sys/kernel/printk
# current default minimum boot-default
# 4 4 1 7
# Temporarily increase verbosity
echo 8 > /proc/sys/kernel/printk
```
Kernel `printk` levels: `KERN_EMERG` through `KERN_DEBUG`. Driver `dev_dbg()` requires `dyndbg` or `DEBUG` define.
### 2. kgdb over serial
Kernel config: `CONFIG_KGDB`, `CONFIG_KGDB_SERIAL_CONSOLE`, `CONFIG_MAGIC_SYSRQ`.
> [!WARNING]
> Debugger entry stops the target kernel and can disrupt its workloads.
> Identify the exact test target and obtain explicit user approval for the interruption before entering kgdb or kdb; otherwise stop and ask.
```bash
# Boot parameter for live debugger entry (ttyS0, 115200)
kgdboc=ttyS0,115200
```
With the target's `kgdboc` I/O driver configured and keyboard SysRq enabled, use SysRq-`g` for live entry.
If `/proc/sys/kernel/sysrq` disables keyboard SysRq, the privileged `/proc/sysrq-trigger` interface still works.
Only after the exact target and interruption are explicitly approved, run this in a root shell on that target:
```bash
echo g > /proc/sysrq-trigger
```
For boot-time entry, build `kgdboc` into the kernel and place `kgdbwait` after its configuration:
```bash
kgdboc=ttyS0,115200 kgdbwait
```
This stops the kernel during boot, under the same target identification and interruption approval requirements.
Resume it with GDB's `continue` (or kdb's `go`) before using SysRq-`g` for later live entry.
See the [kernel kgdb/kdb documentation](https://docs.kernel.org/process/debugging/kgdb.html) for transport setup and entry methods.
GDB host side:
```bash
# Connect to remote serial
gdb vmlinux
(gdb) set serial baud 115200
(gdb) target remote /dev/ttyUSB0
(gdb) bt
(gdb) list *panic+0x20
```
USB variant: `CONFIG_KGDB_KDB` with `kgdboc=usb` on supported hardware.
### 3. kdb commands
When `CONFIG_KGDB_KDB` enabled, kdb provides in-kernel shell:
```text
kdb commands (at SysRq break)
├── bt — stack backtrace
├── lsmod — loaded modules
├── md <addr> — memory display
├── ps — process list
├── id <cpu> — CPU registers
├── go — continue execution
└── cpu <n> — switch CPU context
```
With `CONFIG_KGDB`, `CONFIG_KGDB_KDB`, `CONFIG_MAGIC_SYSRQ`, and a configured `kgdboc` console, enter kdb using SysRq-`g` only after the target and interruption are explicitly approved as above.
If it enters kgdb mode and GDB is not connected, type `$3#33` in one pass on the configured debugger console
to switch to kdb; do not use backspace or delete.
SysRq-`k` is the Secure Access Key (SAK): it kills programs on the current virtual console, not debugger entry.
See the [kernel SysRq command reference](https://docs.kernel.org/admin-guide/sysrq.html).
### 4. ftrace
```bash
# Enable function tracer
echo function > /sys/kernel/debug/tracing/current_tracer
echo 1 > /sys/kernel/debug/tracing/tracing_on
# Filter to specific function
echo schedule > /sys/kernel/debug/tracing/set_ftrace_filter
echo ':*probe_*' >> /sys/kernel/debug/tracing/set_ftrace_notrace
# Capture and read
cat /sys/kernel/debug/tracing/trace_pipe
# Disable
echo 0 > /sys/kernel/debug/tracing/tracing_on
echo nop > /sys/kernel/debug/tracing/current_tracer
```
```bash
# trace-cmd (userspace recorder)
trace-cmd record -p function -l schedule,do_page_fault
trace-cmd report
trace-cmd.dat # kernelshark GUI
```
### 5. kprobes and kretprobes
```bash
# Dynamic kprobe via debugfs
echo 'p:myprobe do_sys_open %ax %si' > /sys/kernel/debug/tracing/kprobe_events
echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
cat /sys/kernel/debug/tracing/trace
# kretprobe
echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events
# Cleanup
echo 0 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
echo '-:myprobe' > /sys/kernel/debug/tracing/kprobe_events
```
In-kernel kprobe (module):
```c
#include <linux/kprobes.h>
static struct kprobe kp = {
.symbol_name = "do_sys_open",
.pre_handler = my_pre_handler,
};
static int __init mod_init(void)
{
return register_kprobe(&kp);
}
```
### 6. Dynamic debug (dyndbg)
```bash
# Enable all debug messages for a module
echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control
# Enable specific file:line
echo 'file drivers/i2c/i2c-core.c +p' > /sys/kernel/debug/dynamic_debug/control
# Query current settings
grep mydriver /sys/kernel/debug/dynamic_debug/control
```
Boot-time: `dyndbg="module mydriver +p"` on kernel command line.
### 7. kdump and crash analysis
```bash
# Setup kdump (crash kernel reserved memory)
# /etc/default/grub: crashkernel=256M
sudo kdump-config show
# After crash, vmcore in /var/crash/
ls /var/crash/*/vmcore
# Analyze with crash utility
crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/vmcore
crash> bt
crash> log
crash> ps
crash> kmem -i
crash> mod
```
```bash
# Compress vmcore
makedumpfile -c -d /proc/vmcore /tmp/vmcore.lzo
```
### 8. /proc/sys runtime tuning for debugging
```bash
# Panic on oops (reproduce in VM)
sysctl kernel.panic_on_oops=1
# Soft lockup detection
sysctl kernel.softlockup_panic=1
# Watchdog
sysctl kernel.nmi_watchdog=1
# Slab debugging
# Boot: slub_debug=P,pagealloc
```
### 9. Debugging decision tree
```text
Kernel issue type?
├── Panic/oops → dmesg; crash vmcore; CONFIG_DEBUG_INFO_BTF
├── Logic bug in driver → dyndbg; ftrace function_graph
├── Performance regression → perf record -g -a; trace-cmd
├── Intermittent → kprobes on suspect path
└── Module crash → kgdb; try_module_get audit
```
## Common Problems
| Symptom | Cause | Fix |
|---------|-------|-----|
| kgdb won't connect | Wrong tty/baud | Match `kgdboc` to serial adapter |
| ftrace empty trace | tracing_off or nop tracer | `echo function > current_tracer`; enable tracing_on |
| kprobe registration fails | Missing CONFIG_KPROBES or inlined function | Use tracepoint or static_key |
| dyndbg has no effect | CONFIG_DYNAMIC_DEBUG disabled | Rebuild kernel with option |
| crash can't read vmcore | Wrong vmlinux debug symbols | Install `linux-image-$(uname -r)-dbg` |
| SysRq doesn't work | `kernel.sysrq` disabled | `echo 1 > /proc/sys/kernel/sysrq` |
## Related skills
- `skills/low-level-programming/linux-kernel-modules` — module development being debugged
- `skills/kernel/device-drivers` — driver-specific probe/IRQ issues
- `skills/kernel/kernel-internals` — subsystem knowledge for interpreting traces
- `skills/debuggers/gdb` — GDB commands shared with kgdb
- `skills/profilers/linux-perf` — perf for kernel hotspots
- `skills/observability/ebpf` — non-invasive kernel tracing alternative
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |