You cannot verify an invisible failure by waiting for a log or trace to appear. Instead, define what healthy service behavior looks like, inject a controlled fault that exercises the risk you care about, and measure the system before, during, and after it. If your signals do not cover the application behavior in question, the test cannot prove that behavior was handled.
What does “no trace” mean in a failure test?
A missing log or distributed trace does not prove that no failure occurred. It may mean the event was not recorded, the signal was never instrumented, or the failure affected behavior without leaving the expected telemetry.
As an Amazon Associate I earn from qualifying purchases.
So test observable outcomes rather than the absence of a trace. Useful measures can include throughput, error rate, latency percentiles, and a synthetic request that represents a customer task. These measurements act as proxies for steady state and customer experience; they do not reveal every internal event.
The experiment can establish whether measured behavior stayed within agreed limits and whether the relevant controls responded. If the question is whether work was silently dropped, for example, infrastructure health alone is not enough: application counters or end-to-end reconciliation may be needed.
Define the claim before injecting a fault
Write a falsifiable hypothesis that names the failure, the mitigation expected to respond, the impact you will tolerate, and what recovery should look like. For example: “If this dependency times out, the circuit breaker will open and the fallback will keep requests within the agreed error-rate and latency limits.”
Set the baseline and pass/fail criteria in advance. Include how quickly detection or fallback should happen and what recovery target counts as success. A resilience experiment tests both recovery mechanisms and observability: a control that works but cannot be detected may still leave operators unable to tell whether the system recovered.
Run a controlled experiment
- Choose a real risk. Use dependency maps, incident history, known weaknesses, and the resilience control you want to validate to select the failure mode.
- Confirm steady state. Check that the workload is healthy and establish baseline values for a small number of service-level outcomes. A user-facing synthetic request can help represent customer experience.
- Check instrumentation. Ensure you can observe both the faulty component and the application behavior at stake. For a queue-backed application, that might mean tracking failed sends, dropped messages, circuit-breaker state, fallback writes, and duplicate processing alongside queue-level signals.
- Set scope and safeguards. Begin outside production, target only intended resources, notify affected teams, and define stop conditions and rollback steps. Do not inject a fault into a workload already known to be unable to tolerate it.
- Inject the matching fault. Choose a fault that exercises the hypothesis, such as latency, throttling, a dependency disconnection, or denied access. These test different code paths; a non-retryable access-denied error can test whether the application stops correctly, but it does not establish that retry backoff works. For that, use a retryable failure such as throttling or a timeout.
- Observe the full window. Record baseline, fault period, and recovery. Compare service outcomes and component signals, verify alerts and mitigations, and confirm return to a known-good state. Keep the experiment data so later runs can be compared.
- Fix and repeat. If the result fails, address the resilience behavior or instrumentation and rerun the same experiment as a regression.
Choose a fault-injection approach that fits the test
Possible faults include resource termination, failover, CPU or memory stress, throttling, added latency, and packet loss. Choose based on the mechanism in your hypothesis, not on how dramatic the failure looks. Evaluate a method by whether it can reproduce the target fault, constrain scope and support rollback, expose experiment timing for correlation with monitoring, and fit the intended setting—local testing, pre-production, a canary, a game day, or an automated regression.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →AWS documents AWS Fault Injection Service and names third-party options such as Chaos Toolkit, Chaos Mesh, Litmus Chaos, and Gremlin. Google Cloud also documents a Fault Injection Testing service; its documentation labels the offering Preview and subject to Pre-GA terms, so availability and conditions may change. These are options, not prerequisites: the important requirement is a controlled fault and enough instrumentation to evaluate the result.
For AWS Fault Injection Service, experiment logging is an explicit configuration choice. When enabled, experiment logs can be correlated with monitoring data to compare fault timing with workload response.
Keep example thresholds in context
AWS’s sample experiment report illustrates a load of 85 requests per second, a 500 ms response-latency threshold, and a 4-second P99 LCP threshold. Those are example values in a sample report, not general benchmarks. Set thresholds from your own service objectives and baseline rather than copying illustrative numbers.
Rank #4
Protect the workload and make recovery verifiable
Fault injection can disrupt running infrastructure. Start in a non-production environment with redundancy, scoped targets, stop conditions, and a rollback plan. Expand to production only deliberately, with suitable controls and coordination. During the test, check not only that the expected behavior occurs, but that it occurs quickly enough; afterward, verify the workload has returned to its known-good state.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
