Skip to content

mohitmishra786/low-level-dev-skills / skills/kernel-dev/kernel-debugging-advanced

kernel-debugging-advanced

Unverified

Advanced kernel debugging skill for ftrace, trace-cmd, perf, kprobes, kgdb, and crash analysis. Use when tracing kernel functions, profiling syscalls in kernel, dynamic instrumentation, or analyzing vmcore. Activates on queries about ftrace, trace-cmd, kprobe, kgdb, kernel crash dump, or dyndbg.

Advanced Kernel Debugging

Purpose

Extend skills/kernel/kernel-debugging with production-grade tracing: ftrace, trace-cmd, kernel perf, kprobes/kretprobes, kgdb, crash dump analysis, and printk discipline. Not merged with kernel-debugging — that skill covers baseline kgdb/dyndbg; this one focuses on trace-cmd, function_graph, and live/post-mortem production workflows.

When to Use

  • Latency spikes in kernel driver without obvious bug
  • Tracing function call graph in kernel
  • Live inspection without recompiling (kprobes, bpf)
  • Post-mortem vmcore after panic

Workflow

1. ftrace function graph

# Enable function graph tracer
echo function_graph > /sys/kernel/debug/tracing/current_tracer
echo my_driver_probe > /sys/kernel/debug/tracing/set_graph_function
echo 1 > /sys/kernel/debug/tracing/tracing_on
# reproduce issue
cat /sys/kernel/debug/tracing/trace
echo 0 > /sys/kernel/debug/tracing/tracing_on

Requires CONFIG_FUNCTION_GRAPH_TRACER and debugfs mounted.

2. trace-cmd record

trace-cmd record -p function_graph -F my_probe
trace-cmd report
trace-cmd stat

Portable capture for sharing with others.

3. perf in kernel context

perf record -a -g -- sleep 10
perf report --stdio
perf probe --add my_driver:probe

See skills/profilers/linux-perf for userspace overlap.

4. kprobes (dynamic)

#include <linux/kprobes.h>

static struct kprobe kp = {
    .symbol_name = "do_sys_open",
    .pre_handler = handler,
};
register_kprobe(&kp);

Use sparingly in production; prefer static tracepoints when available.

5. dyndbg and printk discipline

echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control
echo 'file drivers/foo/*.c +p' > /sys/kernel/debug/dynamic_debug/control
pr_debug("state=%d\n", s);           /* compile-time optional */
printk_ratelimited(KERN_WARNING "hw fault\n");

6. kgdb / kdb

# kernel cmdline: kgdboc=ttyS0,115200 kgdbwait
echo g > /proc/sysrq-trigger   # break into debugger (when configured)

7. Crash dump (vmcore)

# after kdump capture
crash /usr/lib/debug/vmlinux vmcore
crash> bt
crash> dev -s

8. Agent usage

/kernel-debugging-advanced Trace my_driver_ioctl latency with ftrace function_graph

Common Problems

SymptomCauseFix
Empty traceTracer not enabledtracing_on; check current_tracer
ftrace overheadGraph all functionsFilter with set_graph_function
kprobe failInlined symbolTry nearby symbol or static key
kgdb no connectWrong kgdboc ttyMatch serial/USB gadget
crash no symbolsMissing debuginfoInstall linux-image-dbg / vmlinux

Related Skills

  • skills/kernel/kernel-debugging — baseline kgdb, dyndbg
  • skills/profilers/linux-perf — perf record/report
  • skills/observability/ebpf — BPF kprobe alternative
  • skills/kernel-dev/writing-char-drivers — ioctl trace targets
  • skills/debuggers/core-dumps — userspace core parallels
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 464kUpdated Claude CodeCodexVerifiedInstall
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 464kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 318kUpdated Claude CodeCodexUnverifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 318kUpdated Claude CodeCodexUnverifiedInstall