Linux tracing records information at selected points in kernel or user-space execution so you can investigate what a system is doing. It is a family of techniques—not one tool—and the right choice depends on what you need to observe, what evidence you need, and which features your kernel provides.
Tracing, debugging, and profiling answer different questions
These terms overlap in practice, but describe different ways to examine a running system. The Linux Foundation’s 2021 introductory tutorial presents the core distinction:
As an Amazon Associate I earn from qualifying purchases.
- Debugging is commonly interactive: a debugger can stop execution at a chosen point so you can inspect state and control what happens next.
- Profiling commonly uses statistical sampling, often of performance-monitoring events, to show where activity or time is concentrated.
- Tracing records information at selected points, allowing you to analyze events or execution over time.
The categories are useful, not absolute: a tool or workflow may combine techniques. Choose based on the question. If you need to inspect state at a breakpoint, debugging fits; if you need a statistical view of hotspots, profiling may fit; if you need an event sequence or behavior at specific points, tracing is a natural option.
Linux tracing is a set of related mechanisms
The Linux kernel’s tracing guide covers several families. They observe different targets and are not interchangeable; some can also be combined.
| Family | What it can observe | Useful when |
|---|---|---|
| Ftrace | Kernel functions, events, and timing-related behavior, depending on the configured features. | You need kernel-level visibility, including latency or performance analysis. |
| Tracepoints and event tracing | Predefined instrumentation sites and the events they emit. | You want to follow meaningful kernel events and their fields over time. |
| Kernel probes | Probe-defined points in kernel execution. | You need instrumentation beyond a particular predefined event, where supported. |
| Hardware and performance tracing | Hardware- or performance-related signals, using the available mechanisms. | Your question concerns performance signals that ordinary event tracing does not capture. |
| User-space tracing | User-space events or probe points, including user events and uprobes. | You need evidence from application execution rather than only the kernel. |
| Remote tracing | Compatible ring-buffer data written outside the kernel. | Your tracing setup uses a compatible remote producer. |
This is a broad map, not a promise that every Linux system exposes every family. Kernel version, build configuration, permissions, and the particular tracing interface determine what is available.
What ftrace and tracefs provide
Ftrace is a kernel tracing framework, not merely a function tracer. Kernel documentation describes its use for understanding kernel behavior, debugging, latency analysis, performance analysis, and event tracing. Its controls and output are exposed through tracefs when the relevant support is configured.
Rank #2
- Used Book in Good Condition
The documented tracefs mount point is /sys/kernel/tracing. The documentation also notes a backward-compatible location under debugfs. Which tracers and controls appear depends on what was compiled into the kernel, so examples found online may not match your machine.
traceprovides human-readable output from the trace buffer.trace_pipeis intended for streaming: reading it consumes trace data as it is read.tracing_oncontrols writing to the ring buffer. Turning writes off does not necessarily stop all tracing overhead.
Tracepoints and probes: predefined events versus instrumentation
A kernel tracepoint is a statically placed, typed instrumentation hook with defined parameters. When a tracepoint is enabled and a probe is registered, that probe runs when execution reaches the hook. Tracepoints can support debugging, performance accounting, and understanding kernel behavior. The kernel documentation explains the tracepoint mechanism and its use.
Rank #3
For example, paired interrupt-handler entry and exit events can help you reason about handler duration. The useful process is to identify a relevant event, understand its fields, and relate matching events over time—not to treat one event in isolation as a complete explanation.
Instrumentation has a cost. Kernel documentation says a disabled tracepoint still adds a tiny branch-check time penalty and a small space cost; when enabled, its connected probe runs in the caller’s execution context. Those are documented characteristics, not a universal benchmark for every tracing method or workload.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to choose where to start
Start with the evidence you need rather than the name of a tool. Then check whether your target kernel exposes the required instrumentation and whether the collection method fits your operational constraints.
- Define the question. Are you investigating kernel function execution, a predefined event sequence, user-space activity, latency, or a hardware/performance signal?
- Choose the observation point. Consider a tracepoint for a predefined kernel event, function tracing for kernel function activity, a probe where appropriate, or a user-space event or probe for application behavior.
- Check the target system. Confirm tracefs is available and inspect the tracers and events exposed by that kernel. Configuration and permissions can affect what you see.
- Plan the output workflow. Use the human-readable
tracefile for buffered output ortrace_pipewhen you need to read a live stream. Be aware that reading the latter consumes data. - Account for overhead and context. Enable only the instrumentation relevant to the question, and interpret the resulting records with their event fields and collection conditions in mind.
The kernel’s tracing documentation is labeled “latest” and can change; it does not guarantee that a feature exists in every distribution kernel. Check the documentation and available controls for the kernel you are actually using before relying on a particular tracer or event.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
