Choose a Linux kernel debugging method by the evidence you need, how consistently the problem reproduces, whether instrumentation changes its timing, and how much access you have to the target. Use targeted logging or tracing when possible; use performance tools to narrow a broad slowdown; use Kdump to examine a captured crash; and reserve KGDB or KDB for cases that require inspecting a stopped or controlled kernel. These approaches can be combined, and the best first step is usually the least disruptive one that can test your current hypothesis.
Start with the failure, not the debugger
Before selecting a tool, establish what triggers the failure and what access you have. A consistent failure under a known workload is different from an intermittent race; a system you can rebuild and reboot offers different options from a remote or production-like machine you cannot interrupt.
As an Amazon Associate I earn from qualifying purchases.
- Reproduction: Record the workload, triggering conditions, kernel build, and whether the failure occurs every time or intermittently.
- Timing: Ask, “Is timing a factor?” If adding logging makes the symptom disappear or change, treat the instrumentation itself as a possible influence.
- Access: Ask, “Do I need to debug with limited access?” Determine whether you can change kernel configuration, load or rebuild a module, access tracefs, retrieve logs, connect a serial interface, or reboot the target.
- Evidence: Decide whether you need selected messages, a sequence of function calls or events, performance data, a post-crash snapshot, or interactive inspection of source-level state.
These constraints matter more than a tool’s general reputation. Kernel configuration, architecture, available drivers, and the ability to stop or alter the target can rule out an otherwise suitable technique.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Match the evidence to the tool
| What you need to learn | Good starting point | Main trade-off |
|---|---|---|
| Whether existing debug messages explain a specific subsystem or code path | Dynamic debug | Only works for supported debug-print callsites compiled into the running kernel or module. |
| Which functions or events occur, and in what sequence | ftrace and event tracing | Available functions, events, and controls depend on kernel configuration and build. |
| Where an unexplained system slowdown is concentrated | Ordinary performance tools, followed by perf where useful |
Performance evidence narrows the investigation but does not automatically explain a bug. |
| What state existed when the kernel crashed | Kdump | Requires a crash dump to analyze; it is not an interactive debugger for a running system. |
| Live, source-level investigation with breakpoints and GDB | KGDB | Requires a development host, a prepared target, matching symbols, and a supported connection. |
| Console-based inspection and basic run control | KDB | Its shell-like interface is distinct from GDB’s source-level workflow and requires KGDB support. |
The options are not mutually exclusive. For example, tracing may identify a narrow code path before you enable its debug messages, while a crash dump can preserve evidence from a failure that is too disruptive to investigate live.
#1 Best Overall
Use targeted logging when messages may answer the question
Dynamic debug for existing callsites
Dynamic debug lets you enable or disable supported kernel debug-print code at runtime. When available, its control catalog is /proc/dynamic_debug/control. Queries can select callsites by source file, function, line, module, format string, or declared class; root access is required.
This is useful when you have a specific subsystem or code path in mind and want more detail without rebuilding solely to turn on every debug message. It does not insert instrumentation into arbitrary code: the relevant debug-print callsites must already exist in the kernel or module and be supported by the build. Also distinguish “the message was not generated” from “the message was generated but is not visible where expected”; kernel log-level settings can affect console visibility.
When logging changes the symptom
printk() can alter timing. If the failure is timing-sensitive, added output may mask or change it, so a clean-looking log does not necessarily show that the underlying fault is gone. Kernel debugging guidance identifies trace_printk(), which writes to the trace file, as an alternative to consider in this situation. It is still instrumentation; compare behavior with and without it rather than assuming any tracing method has zero effect.
Recommended Free Tools
Rank #2
Trace execution and events to narrow the path
Use ftrace and event tracing when the important question is what ran, when it ran, or which events surrounded the failure. The kernel tracing family includes function and event tracing as well as mechanisms such as Kprobes, Fprobes, Eprobes, boot-time tracing, histograms, and hardware or performance tracing. Pick a feature for a specific question rather than enabling broad instrumentation without a plan.
The tracefs interface exposes controls and trace buffers. Depending on the kernel and configuration, tracing workflows can include function filters, stack traces, event selection, and reading a live stream through trace_pipe. The kernel userspace tracing guide names CONFIG_DYNAMIC_FTRACE as a prerequisite for its ftrace guidance. Available functions and events vary by build, so confirm what the target actually provides and use documentation matching that kernel. KernelShark can visualize trace-cmd output, but the core tracing controls are available through tracefs.
Tracing is a useful middle ground when selected messages are too sparse but a full interactive debugger would be excessive or impractical. Start with a narrow filter or event set tied to the suspected path; expand only if the first trace leaves a meaningful gap.
Use performance analysis for a broad slowdown
If the issue is an unexplained slowdown rather than a localized crash or code-path question, begin by narrowing the affected resource. The kernel userspace debugging guidance points to ordinary system performance tools for examining load, CPU, I/O, memory, and per-process behavior, then recommends perf for deeper analysis.
Use those observations to choose the next investigation: a CPU-heavy workload, an I/O wait problem, or memory pressure call for different hypotheses. Performance analysis can identify where to look without interrupting the system, which is useful when stopping the target is not acceptable. It is evidence for narrowing the problem, not a substitute for tracing or source-level debugging when the cause remains unclear.
Capture a crash for investigation after the fact
Kdump is Linux’s kexec-based crash-dump solution. It is the appropriate route when the kernel crashes and you need to preserve and examine crash state later, rather than attach an interactive debugger to a normally running system.
Setting up the capture kernel, dump destination, and related services depends on distribution, architecture, and system configuration. Follow the current instructions for the distribution and architecture under test, and verify that the dump is actually captured and retrievable before relying on it to diagnose a future failure.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Choose KGDB or KDB when live inspection is necessary
KGDB: source-level work with GDB
KGDB connects a development machine running GDB to the target kernel under investigation. The kernel documentation states, “Two machines are required for using kgdb.” Configure GDB to use the target’s symbol-bearing vmlinux, not the compressed boot image. The target build needs debug information enabled with CONFIG_DEBUG_INFO; CONFIG_FRAME_POINTER is advised for more accurate stack backtraces, but is not required. Exact configuration names and availability can vary by kernel version and architecture.
For serial debugging, the kernel documentation describes kgdboc as the main mechanism for connecting GDB to KGDB and interacting with KDB. kgdbwait and early debugging require the KGDB I/O driver to be built into the kernel. A kgdboc plus kgdbcon setup cannot use a tty that is already serving as an active system console.
Best Value
A USB-to-serial adapter may be useful when the target’s UART and the development host require that connection, but compatibility is board-specific. Check the target board documentation for the connector, electrical requirements, and supported interface before buying or connecting an adapter.
KDB: console inspection and basic control
KDB provides a shell-like console debugger for examining system state, including memory, registers, processes, and logs, and for basic run control such as breakpoints. It is not the same interface as GDB’s source-level debugging. The kernel documentation requires KGDB support to be enabled for KDB; the two frontends can share the debug core and interoperate.
Because interactive debugging can stop or alter target execution, use it when the needed evidence justifies that disruption and the target can be prepared for it. For a remote system with limited access, first consider evidence you can collect without taking control of the kernel.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Quick Recap
A practical escalation path
- Record the conditions. Capture the kernel version and build, workload, trigger, reproduction rate, available logs, and whether timing changes the symptom.
- Use evidence already available. Check existing messages and system performance data before changing the target. If the problem is broad, narrow it by CPU, I/O, memory, or process behavior.
- Enable the narrowest useful instrumentation. Use dynamic debug for relevant compiled callsites; use ftrace or event tracing for execution sequence and event context. Keep the capture focused enough to interpret.
- Preserve failures that cannot be inspected live. If the kernel crashes and post-crash state is needed, arrange and validate Kdump using the matching distribution guidance.
- Move to interactive debugging only for a concrete question. Use KGDB when source-level GDB inspection is necessary, or KDB for console-based state inspection and basic control. Confirm the target configuration, symbols, and connection path first.
- Check version-specific documentation. Rolling kernel documentation and versioned guides may describe different configurations or capabilities; match instructions to the kernel actually under test.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
