Driver FixRecommendedSound, Wi-Fi or graphics acting up? Check drivers firstFind missing or outdated drivers fast.Check DriversHalloween DealsAmazon USHalloween week deals: useful picks before month-endAmazon US: current deals, useful picks and tech finds.Check DealsClean PCRecommendedOne scan can reveal what keeps slowing WindowsLook for cleanup and repair opportunities.Run Scan×
FREEDOM251

Freedom251 › Blog › filesystem debugging

How to Reproduce and Debug Linux Page-Cache Races with KCSAN, lockdep, and Stress Tests

By Freedom251 Team · · 11 min read

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Reproducing a Linux page-cache race requires three pieces working together: a debug kernel instrumented with KCSAN and lockdep, a workload that repeatedly overlaps the exact page-cache operations involved, and disciplined capture of every report and test condition. KCSAN can report conflicting memory accesses, but it samples watchpoints and may miss an unobserved interleaving. lockdep checks lock-order and lock-context relationships seen during execution, including paths that could deadlock without producing a final deadlock. Stress tests increase the chance of exercising the bug; a clean run is not proof that the race is absent.

The practical method is to freeze the test matrix, build the appropriate instrumentation, design actors around the suspected operation, run KCSAN and lockdep with the workload, preserve complete logs, and vary one factor at a time. Treat each report as a hypothesis to validate against the page-cache invariant and the Linux-Kernel Memory Model, not as an automatic verdict.

As an Amazon Associate I earn from qualifying purchases.

What you are trying to establish

A page-cache race is not reproduced merely because several processes perform file I/O at once. You need evidence that the same shared state can be accessed concurrently through the path implicated by the bug, and that the ordering required by the kernel’s synchronization protocol is missing or violated.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Tool or technique What it observes What a positive result means What a clean result cannot prove
KCSAN Conflicting instrumented memory accesses, sampled with watchpoints and scheduler delays At least one execution exposed accesses that may race; the report supplies stacks, access types and sizes, and task/CPU context for investigation That no race exists. An interleaving can be missed because it was not sampled or the workload never reached it
lockdep Lock-class acquisition order, lock nesting, IRQ/softirq safety, and recursive-lock rules observed while the kernel runs The observed history contains a lock relationship that can permit deadlock or lock-context misuse That all relevant paths are safe. Unexercised paths contribute no edges to the dependency graph
Stress or torture workload How often and under what scheduling, CPU, storage, and concurrency conditions the target operations overlap Higher opportunity to observe a KCSAN conflict or lockdep relationship Correctness. Duration and iteration count have no universal success threshold

Use the tools as complementary detectors. KCSAN focuses on memory-access ordering; lockdep focuses on locking discipline; the workload determines whether either detector reaches the code that matters.

Freeze the test matrix before changing code or knobs

Page-cache behavior depends on the kernel revision, filesystem implementation, architecture, and storage timing. Record the complete environment for every run so that a report can be reproduced and a disappearing report can be interpreted.

Category Record Why it matters
Kernel and toolchain Exact kernel commit, configuration, compiler and workload version Instrumentation, locking code and compiler-generated accesses change between revisions
Machine Architecture, CPU count, page size and installed memory Atomicity, cache-line layout, scheduling opportunities and memory pressure are hardware-dependent
Storage and filesystem Filesystem type, image or device, storage backend and every mount option Different filesystems reach different address-space, writeback and invalidation paths
Boot and runtime Boot parameters, preemption model, CPU affinity, process and thread counts, and memory-pressure settings These settings alter interleavings and the timing of delayed accesses
Run identity Workload seed, iteration count, start and stop times, and a unique run identifier Reproduction requires knowing exactly which execution produced each log

Keep this matrix beside the KCSAN and lockdep output. If a bug report omits one of these values, label the result as conditional rather than generalizing it to other filesystems, architectures or kernel versions.

Build a kernel with the right instrumentation

Enable KCSAN

Build with CONFIG_KCSAN=y. KCSAN is a dynamic race detector based on compile-time instrumentation and sampling watchpoints. The kernel documentation also describes strict checking with CONFIG_KCSAN_STRICT, debugfs controls under /sys/kernel/debug/kcsan, and runtime parameters including kcsan.skip_watch, kcsan.udelay_task and kcsan.udelay_interrupt.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Use the same compiler and configuration for every comparison run. A configuration change can alter both the instrumented code and the schedule, so changing KCSAN options is itself an experimental variable. Preserve the complete configuration with the report.

Enable lockdep and locking debug checks

Enable the locking-debug configuration required by the kernel version, including CONFIG_DEBUG_LOCKDEP where the debugging guide lists it as a prerequisite. lockdep constructs a dependency graph from lock acquisitions and checks inverse ordering, lock-class usage, IRQ and softirq safety, and recursive locking. It can identify a potential deadlock from observed sequences without waiting for two tasks to deadlock in production.

Lockdep’s guarantee is conditional on execution coverage. Its documented “closure” applies to simple, standalone single-task locking sequences that actually occurred during the kernel’s lifetime; an unvisited page-cache path contributes no evidence.

Use lock torture as a companion, not a substitute

When the hypothesis concerns a locking primitive or a lock-order change, enable CONFIG_LOCK_TORTURE_TEST. Its reader and writer stress parameters exercise core kernel locking primitives. They do not automatically drive a filesystem’s address-space operations, so run the torture facility alongside a workload that reaches the suspected page-cache path.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Verify the instrumented boot

Confirm from the running kernel that the intended configuration is active, that debugfs is available, and that the KCSAN control files can be inspected under /sys/kernel/debug/kcsan. Save boot parameters and the kernel configuration as artifacts; a report from a non-instrumented boot is not comparable to an instrumented run.

Design a workload that reaches the race

Start with the operation named by the bug, not with a generic file-I/O benchmark. Identify the shared page-cache object, the operation that mutates or invalidates it, and the operation that reads or faults it. Then create at least two actors whose critical windows overlap repeatedly.

Pair actors around the suspected operation

Actor pair What to overlap Qualification
Buffered reader and buffered writer Concurrent reads and writes to the same file range or pages Exact locking and cache behavior depends on the filesystem and kernel revision
Truncation or invalidation and read/write faults Removal or invalidation of pages while another task faults them in or accesses them Use the address-space path implicated by the bug; do not assume every filesystem implements it identically
Writeback and mutation Background writeback while another actor changes page state or file metadata Storage latency and dirty-page thresholds can change the overlap window
Memory mapping and invalidation Mapped access or faults concurrent with invalidation or truncation Map the same file range and verify that the workload really generates faults during the invalidation window

These are workload patterns, not claims that all filesystems take identical locks or call identical functions. Instrument or trace the target path sufficiently to confirm that each actor reaches the intended operation.

Increase overlap without changing the hypothesis

  • Repeat the smallest operation sequence that can trigger the bug instead of adding unrelated I/O.
  • Vary process and thread counts so that both low-contention and high-contention schedules are exercised.
  • Change CPU affinity across runs, including confined and spread-out placements, to alter cache and scheduler behavior.
  • Introduce controlled delays immediately before or after the suspected access, then remove them to check whether the timing window is genuine.
  • Use scheduler pressure, storage-latency variation and memory pressure only as named experimental factors; record each change.

KCSAN’s watchpoint sampling and delay settings affect the probability that a conflicting pair is observed. More iterations increase opportunity, but they do not compensate for a workload that never overlaps the relevant operations.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Run KCSAN and interpret a report

Run the original, unmodified test matrix first. Keep KCSAN’s output in a file that includes the run identifier and kernel commit. A complete report should retain both access stacks, access sizes and types, task and CPU information, and held-lock information when present.

What to look for in the report

  1. Locate the two reported accesses and map each stack to the page-cache operation in your hypothesis.
  2. Check whether the accesses can truly execute concurrently. Sequential calls in one task are not a data race merely because they touch the same address.
  3. Determine whether at least one access is a plain, unsynchronized access. Atomic operations, locks, barriers and other primitives can change the ordering argument.
  4. Identify the synchronization primitive that should order the accesses, such as a lock, reference rule or memory-ordering protocol, and verify that both paths use it consistently.
  5. Check the page-cache invariant that would be violated if the accesses were unordered: page state, mapping membership, dirty state, writeback state or another explicitly documented condition.
  6. Compare the report with held-lock information and the corresponding lockdep history. A missing held lock can be the central clue, while a lock held on both paths may indicate a false alarm or an incomplete understanding of the protocol.

A KCSAN report is evidence of an observed conflict, not a proof that the kernel is incorrect. Resolve it by reading the code and its synchronization contract, then reproduce it with the same matrix before drawing a conclusion.

Use lockdep to test the locking hypothesis

Run lockdep with the same page-cache workload whenever the suspected fix adds, removes or reorders a lock. Save the first lockdep warning in full; later splats can be consequences of the initial violation.

Classify the lockdep splat

  • Inverse ordering: two lock classes are acquired in opposite order on different paths, creating a concrete deadlock hypothesis.
  • IRQ or softirq safety: a lock is acquired in a context where an interrupt handler can acquire it in an incompatible context.
  • Recursive locking: a task acquires the same lock class again without a permitted recursive protocol.
  • Lock-class confusion: instances that should be independent are treated as one class, or instances that must share ordering are classified inconsistently.

Reconstruct the acquisition sequence shown in the splat and identify which page-cache path supplied each edge. If a warning disappears after changing the workload, first ask whether the path stopped executing; do not treat disappearance as proof that the lock relationship was corrected.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Capture logs so another engineer can reproduce the observation

Store KCSAN, lockdep and workload artifacts together. Include:

  • Kernel commit, complete configuration, compiler, architecture, CPU count, page size and memory size.
  • Filesystem type and image or device, storage backend, mount options and workload version.
  • Boot parameters, preemption model, CPU affinity, process and thread counts, and any memory or scheduler settings.
  • Workload seed, operation counts, iteration number, start and stop timestamps, and the exact test command or script.
  • The complete KCSAN report, including both stacks, access sizes and types, task and CPU data, and held-lock information.
  • The first lockdep splat and surrounding kernel log, plus any lock-torture output.
  • Changes to KCSAN watch frequency or delay parameters, with one artifact per run.

Do not trim the first warning or replace a full stack with a summary. Later reports can depend on an earlier corrupted state, and omitted context prevents reviewers from separating a primary defect from a consequence.

Change one variable at a time

After obtaining a baseline, vary a single factor and keep every other matrix entry fixed. Useful experiments include:

Single factor Question it answers How to interpret a disappearing report
KCSAN delay or watch frequency Is observation sensitive to sampling and timing? The schedule or sampling window changed; safety is not established
CPU count or affinity Does the race require particular processor placement or cache contention? The interleaving became less likely or the path was not reached
Preemption model or scheduler pressure Does the window depend on task interruption? The timing opportunity changed, not necessarily the synchronization correctness
Filesystem or mount option Is the behavior implementation- or option-specific? Scope the result to the tested filesystem and options
Storage latency Does writeback timing control the overlap? The workload window moved; preserve the latency conditions when comparing runs
Workload concurrency How many actors are required to reach the path? Lower contention may simply fail to exercise the race

First reproduce on the original configuration, then change one factor. Changing several settings together destroys the ability to identify which condition affected the observation.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

How long should you stress the workload?

There is no authoritative universal duration, success rate or iteration count for page-cache race reproduction. The required window depends on the race, kernel, hardware, filesystem and workload. Upstream documentation exposes controls and test facilities but does not define a fixed run time.

Best Value
Sale
UNIX and Linux System Administration Handbook, 4th Edition
  • New
  • Mint Condition
  • Dispatch same day for order received before 12 noon
  • Guaranteed packaging
  • No quibbles returns

Use a measured stopping rule

  1. Define the operation sequence and the minimum number of overlapping actors before starting.
  2. Run repeated, independently seeded trials under the frozen baseline and record every trial, including clean ones.
  3. Continue until you have either a reproducible report, a bounded test budget agreed for the investigation, or evidence that the workload is not reaching the target path.
  4. When a run is clean, report the exact sampling settings, duration, iteration count and coverage evidence. State that no report was observed in that window, not that the race was disproved.
  5. Repeat the baseline after each diagnostic change so that a one-off schedule does not become the new reference.

A longer run raises the chance of observing a sampled interleaving only when the workload continues to exercise the implicated operations. Thousands of iterations of the wrong path provide no evidence about the target race.

Handle intentional races with documented synchronization

Some lock-free or read-mostly protocols intentionally permit an access that would look racy to a detector. Do not silence such a report merely to obtain a clean log.

First establish the synchronization argument under the Linux-Kernel Memory Model and identify the writer, readers, ordering primitives and lifetime rules. Then use the annotations documented for KCSAN: data_race(...) for a specific intentional access, __data_racy where appropriate, or function and compilation-unit exclusions when the protocol genuinely applies to the whole scope. Pair annotations with explicit assertions, such as exclusive-writer checks, so that a future violation becomes visible.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Document why the access is safe, which primitive supplies ordering, and what invariant the assertion protects. An annotation without that explanation hides a potential defect; an assertion without a valid protocol merely turns a race into a failure later.

A repeatable investigation sequence

  1. Describe the invariant: write down which page-cache state must be ordered and which operations can change it.
  2. Freeze the matrix: capture commit, configuration, compiler, hardware, filesystem, mount options, storage, boot parameters and workload identity.
  3. Build instrumentation: enable CONFIG_KCSAN, the required locking-debug options including CONFIG_DEBUG_LOCKDEP, and CONFIG_LOCK_TORTURE_TEST when a locking primitive is part of the hypothesis.
  4. Construct actors: pair the exact read, write, fault, invalidation, truncation, writeback or mapping operations implicated by the bug.
  5. Run the baseline: collect complete KCSAN, lockdep and workload logs without changing variables.
  6. Validate reports: map stacks to real concurrent paths, inspect synchronization and check the page-cache invariant.
  7. Probe locking: reconstruct lock classes and acquisition order from the first lockdep warning.
  8. Vary one factor: change sampling, CPUs, scheduler, filesystem, storage or concurrency separately, then return to baseline.
  9. Classify clean runs correctly: record them as unobserved reports under stated conditions, never as proof of absence.
  10. Document intentional protocols: use KCSAN annotations and assertions only after the memory-ordering argument is established.

Bottom line

The strongest page-cache race investigation is an auditable experiment: a precisely described workload repeatedly reaches the suspected operations, KCSAN samples conflicting accesses, lockdep checks the locks those paths acquire, and every result is tied to an immutable kernel and hardware matrix. Stress duration has meaning only when operation coverage and sampling conditions are known. A clean log narrows what was observed; it does not certify that an unsampled interleaving cannot occur.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

filesystem debuggingKCSANLinux kernellockdeppage cacherace conditionsStress Testing

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Recommended PC Tool
Recommended PC Tool
PC Slower Than It Used to Be?Free scan - under a minute
Crashes, No Sound, or Screen Glitches?Free driver scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.