Skip to content

mohitmishra786/low-level-dev-skills / skills/kernel/kernel-debugging

kernel-debugging

Unverified

Linux kernel debugging skill for kgdb, kdb, ftrace, kprobes, and crash analysis. Use when setting up kgdb over serial, using ftrace and trace-cmd, inserting kprobes, enabling dynamic debug, or analyzing kdump with crash. Activates on queries about kgdb, kdb, ftrace, kprobes, dyndbg, kdump, or kernel dmesg levels.

Kernel Debugging

Purpose

Guide agents through debugging the Linux kernel: kgdb over serial or USB, kdb built-in debugger commands, ftrace and trace-cmd, kprobes and kretprobes, dynamic debug (dyndbg), crash dump analysis with crash and makedumpfile, dmesg log levels, and /proc/sys runtime tuning for debugging.

When to Use

  • Kernel panic or oops with unclear stack trace
  • Live debugging a kernel module or driver with kgdb
  • Tracing function calls without recompiling (ftrace, kprobes)
  • Enabling verbose driver logging with dyndbg
  • Analyzing production crash dumps from kdump
  • Tuning kernel runtime parameters for debug sessions

Workflow

1. dmesg and log levels

# Kernel ring buffer
dmesg -T -l err,warn
dmesg -w   # follow live

# Log level (0=emerg .. 7=debug)
cat /proc/sys/kernel/printk
# current  default  minimum  boot-default
# 4        4        1        7

# Temporarily increase verbosity
echo 8 > /proc/sys/kernel/printk

Kernel printk levels: KERN_EMERG through KERN_DEBUG. Driver dev_dbg() requires dyndbg or DEBUG define.

2. kgdb over serial

Kernel config: CONFIG_KGDB, CONFIG_KGDB_SERIAL_CONSOLE, CONFIG_MAGIC_SYSRQ.

[!WARNING] Debugger entry stops the target kernel and can disrupt its workloads. Identify the exact test target and obtain explicit user approval for the interruption before entering kgdb or kdb; otherwise stop and ask.

# Boot parameter for live debugger entry (ttyS0, 115200)
kgdboc=ttyS0,115200

With the target's kgdboc I/O driver configured and keyboard SysRq enabled, use SysRq-g for live entry. If /proc/sys/kernel/sysrq disables keyboard SysRq, the privileged /proc/sysrq-trigger interface still works. Only after the exact target and interruption are explicitly approved, run this in a root shell on that target:

echo g > /proc/sysrq-trigger

For boot-time entry, build kgdboc into the kernel and place kgdbwait after its configuration:

kgdboc=ttyS0,115200 kgdbwait

This stops the kernel during boot, under the same target identification and interruption approval requirements. Resume it with GDB's continue (or kdb's go) before using SysRq-g for later live entry.

See the kernel kgdb/kdb documentation for transport setup and entry methods.

GDB host side:

# Connect to remote serial
gdb vmlinux
(gdb) set serial baud 115200
(gdb) target remote /dev/ttyUSB0
(gdb) bt
(gdb) list *panic+0x20

USB variant: CONFIG_KGDB_KDB with kgdboc=usb on supported hardware.

3. kdb commands

When CONFIG_KGDB_KDB enabled, kdb provides in-kernel shell:

kdb commands (at SysRq break)
├── bt          — stack backtrace
├── lsmod       — loaded modules
├── md <addr>   — memory display
├── ps          — process list
├── id <cpu>    — CPU registers
├── go          — continue execution
└── cpu <n>     — switch CPU context

With CONFIG_KGDB, CONFIG_KGDB_KDB, CONFIG_MAGIC_SYSRQ, and a configured kgdboc console, enter kdb using SysRq-g only after the target and interruption are explicitly approved as above. If it enters kgdb mode and GDB is not connected, type $3#33 in one pass on the configured debugger console to switch to kdb; do not use backspace or delete. SysRq-k is the Secure Access Key (SAK): it kills programs on the current virtual console, not debugger entry. See the kernel SysRq command reference.

4. ftrace

# Enable function tracer
echo function > /sys/kernel/debug/tracing/current_tracer
echo 1 > /sys/kernel/debug/tracing/tracing_on

# Filter to specific function
echo schedule > /sys/kernel/debug/tracing/set_ftrace_filter
echo ':*probe_*' >> /sys/kernel/debug/tracing/set_ftrace_notrace

# Capture and read
cat /sys/kernel/debug/tracing/trace_pipe

# Disable
echo 0 > /sys/kernel/debug/tracing/tracing_on
echo nop > /sys/kernel/debug/tracing/current_tracer
# trace-cmd (userspace recorder)
trace-cmd record -p function -l schedule,do_page_fault
trace-cmd report
trace-cmd.dat  # kernelshark GUI

5. kprobes and kretprobes

# Dynamic kprobe via debugfs
echo 'p:myprobe do_sys_open %ax %si' > /sys/kernel/debug/tracing/kprobe_events
echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
cat /sys/kernel/debug/tracing/trace

# kretprobe
echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events

# Cleanup
echo 0 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
echo '-:myprobe' > /sys/kernel/debug/tracing/kprobe_events

In-kernel kprobe (module):

#include <linux/kprobes.h>

static struct kprobe kp = {
    .symbol_name = "do_sys_open",
    .pre_handler = my_pre_handler,
};

static int __init mod_init(void)
{
    return register_kprobe(&kp);
}

6. Dynamic debug (dyndbg)

# Enable all debug messages for a module
echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control

# Enable specific file:line
echo 'file drivers/i2c/i2c-core.c +p' > /sys/kernel/debug/dynamic_debug/control

# Query current settings
grep mydriver /sys/kernel/debug/dynamic_debug/control

Boot-time: dyndbg="module mydriver +p" on kernel command line.

7. kdump and crash analysis

# Setup kdump (crash kernel reserved memory)
# /etc/default/grub: crashkernel=256M
sudo kdump-config show

# After crash, vmcore in /var/crash/
ls /var/crash/*/vmcore

# Analyze with crash utility
crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/vmcore

crash> bt
crash> log
crash> ps
crash> kmem -i
crash> mod
# Compress vmcore
makedumpfile -c -d /proc/vmcore /tmp/vmcore.lzo

8. /proc/sys runtime tuning for debugging

# Panic on oops (reproduce in VM)
sysctl kernel.panic_on_oops=1

# Soft lockup detection
sysctl kernel.softlockup_panic=1

# Watchdog
sysctl kernel.nmi_watchdog=1

# Slab debugging
# Boot: slub_debug=P,pagealloc

9. Debugging decision tree

Kernel issue type?
├── Panic/oops → dmesg; crash vmcore; CONFIG_DEBUG_INFO_BTF
├── Logic bug in driver → dyndbg; ftrace function_graph
├── Performance regression → perf record -g -a; trace-cmd
├── Intermittent → kprobes on suspect path
└── Module crash → kgdb; try_module_get audit

Common Problems

SymptomCauseFix
kgdb won't connectWrong tty/baudMatch kgdboc to serial adapter
ftrace empty tracetracing_off or nop tracerecho function > current_tracer; enable tracing_on
kprobe registration failsMissing CONFIG_KPROBES or inlined functionUse tracepoint or static_key
dyndbg has no effectCONFIG_DYNAMIC_DEBUG disabledRebuild kernel with option
crash can't read vmcoreWrong vmlinux debug symbolsInstall linux-image-$(uname -r)-dbg
SysRq doesn't workkernel.sysrq disabledecho 1 > /proc/sys/kernel/sysrq

Related Skills

  • skills/low-level-programming/linux-kernel-modules — module development being debugged
  • skills/kernel/device-drivers — driver-specific probe/IRQ issues
  • skills/kernel/kernel-internals — subsystem knowledge for interpreting traces
  • skills/debuggers/gdb — GDB commands shared with kgdb
  • skills/profilers/linux-perf — perf for kernel hotspots
  • skills/observability/ebpf — non-invasive kernel tracing alternative
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 318kUpdated Claude CodeCodexUnverifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 318kUpdated Claude CodeCodexUnverifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 318kUpdated Claude CodeCodexUnverifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 196kUpdated Claude CodeCodexVerifiedInstall