A background workflow can recover from a crash only if it can tell what has already happened and what still needs to happen. Persist the workflow’s identity, resumable inputs, and step history; make orchestration decisions replayable; and design external side effects so a retry cannot silently duplicate a harmful action. Checkpointing helps restore progress, but it does not by itself make an external operation happen exactly once.
What recovery has to preserve
A durable workflow needs more than a final status such as “failed” or “running.” After a process stops, the runtime or operator must be able to identify the same logical job, reconstruct the information needed to continue, and distinguish completed work from work that remains.
As an Amazon Associate I earn from qualifying purchases.
- Stable workflow identity: Give each logical operation an identifier that survives worker restarts and can be found by the caller or operator.
- Resumable input: Persist the input needed to continue, or a durable reference to it. Avoid relying on in-memory state from the interrupted process.
- Step outcomes and history: Record which steps completed and the results needed by later steps. The runtime may use this history to replay orchestration without repeating completed activities.
- Recovery controls: Define how to inspect, resume, terminate, or repair an execution, and how a terminal failure is surfaced.
Keep the workflow record useful without copying sensitive or unnecessarily large payloads into it. Retention, access controls, privacy, and storage cost are part of the design, not afterthoughts.
Separate orchestration from side effects
Orchestration decides what should happen next; activities perform work such as calling an API, writing a record, or sending a notification. Keeping those roles distinct makes it possible for a durable runtime to replay orchestration while reusing recorded activity results.
#1 Best Overall
Azure Durable Functions documents this replay model: it saves execution history and enqueues work, allowing an orchestrator to be unloaded. When more work arrives, the orchestrator replays from the beginning and consults its history to reuse results from completed activities. Microsoft’s guidance is explicit: “For the replay pattern to work correctly and reliably, orchestrator function code must be deterministic.”
In practice, replayed orchestration should not make unrecorded decisions from wall-clock time, random values, or mutable external state. Use the runtime’s durable operations for supported interactions, and put I/O and other side effects in activities whose outcomes can be recorded. Otherwise a replay might choose a different branch from the one taken before the interruption.
Why retries can repeat a side effect
Consider a workflow that charges a payment method and then records “charge complete.” The payment service may accept the charge, but the worker can crash before saving the completion. On retry, the workflow sees no recorded success and may submit the charge again. A checkpoint cannot make the external call and the workflow’s own state update one atomic transaction.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
For that reason, treat retries as potentially repeating an operation unless the specific runtime and operation establish a stronger guarantee. AWS Durable Execution SDK documentation warns: “Replay and retry can each run the same operation more than once.” A guarantee scoped to one attempt or retry does not automatically mean a workflow-wide effect can happen only once.
Make repeatable operations safe
- Assign a stable business-operation identifier and pass it as an idempotency key when the receiving API supports one. Reuse the same key for retries of the same logical action; do not mint a new key for each attempt.
- Prefer naturally repeatable writes where appropriate, such as an upsert keyed by the workflow’s business identifier, rather than an unqualified create.
- Persist enough information to reconcile an uncertain result with the receiving system before issuing another non-idempotent request.
Choose a deliberate path for non-repeatable work
If an action cannot safely be repeated and the recipient offers no idempotency mechanism, decide how uncertainty will be handled. Options include disabling automatic retry for that action, checking the external system before retrying, routing the execution to an operator, or applying a compensating action when the business process allows it. Compensation is a new business operation, not a rollback that erases the original external effect.
Define retry scope and failure handling
Retries should be designed around the operation, not applied uniformly to every failure. A temporary network interruption may justify a bounded retry; invalid input or a rejected business decision usually needs a different response. Document which errors are retryable, how many attempts are allowed, and what happens when the limit is reached.
- Transient failure: Retry only when the action is safe to repeat or can be reconciled.
- Permanent failure: Stop automatic attempts and expose a terminal state with enough context to diagnose it.
- Uncertain outcome: Treat a timeout after a request as potentially successful at the recipient. Reconcile or query by stable operation identity before retrying.
- Partial completion: Identify which prior effects remain in place and whether the workflow can continue, compensate, or requires human repair.
Test the interruption boundary between a successful external effect and recording its result. Also test crashes before the effect, after its result is recorded, and while a retry is being scheduled. These cases expose duplicate-effect risks that a test of ordinary task failure can miss.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Compare workflow guarantees by mode, not product name alone
Execution guarantees, history, and recovery controls vary by workflow mode. “Exactly once” should be read as a statement about the documented scope of that mode—not as proof that every external side effect across a multi-service business process can never be duplicated.
| Option or mode | Documented execution behavior | History and recovery implications |
|---|---|---|
| AWS Step Functions Standard | The AWS guide describes exactly-once workflow execution. | State is persisted between transitions. The guide reports that Standard execution history data is removed after 90 days; verify the current service documentation and applicable retention settings for your use case. |
| AWS Step Functions Asynchronous Express | The guide describes at-least-once execution. | Step Functions itself does not capture Express execution history, so plan explicit logging and correlation if that history is needed for diagnosis or recovery. |
| AWS Step Functions Synchronous Express | The guide describes at-most-once execution. | Step Functions itself does not capture Express execution history; check what other operational records your design needs. |
| Azure Durable Functions | Orchestration uses recorded history and deterministic replay. | History lets the orchestrator replay and reuse completed activity results. Storage-provider behavior and the selected hosting model affect operational fit. |
These descriptions are mode-specific platform guarantees, not a substitute for checking what happens at the boundary with an external service. Confirm the current documentation for the selected mode, including retry behavior, how history can be inspected, retention, and the controls available to resume or terminate executions.
Rank #4
- Book - powershell for sysadmins: workflow automation made easy
- Language: english
- Binding: paperback
Account for storage and runtime lifecycle
Azure documents that its Azure Storage provider coordinates storage tables and queues using eventual-consistency patterns because those services do not provide transactional consistency together. Microsoft identifies Durable Task Scheduler as an alternative managed backend. Choose a backend with its consistency and operational behavior in mind; do not assume every provider implements state coordination identically.
Runtime lifecycle also matters. Microsoft Learn says support for the Durable Functions in-process model ends on November 10, 2026, and recommends the isolated worker model. For a new deployment or a migration plan being made now, account for that date rather than treating the in-process model as a long-term default.
Quick Recap
A practical design checklist
- Assign identity: Create a durable workflow ID and a stable business-operation ID for each external effect that may need deduplication.
- Save what recovery needs: Persist workflow input or a durable reference, completed-step outcomes, and the state needed to select the next action.
- Make orchestration replay-safe: Keep decisions deterministic and route time, randomness, and external I/O through framework-supported durable operations or activities.
- Protect side effects: Use receiver-supported idempotency keys or repeatable writes. Define reconciliation or operator handling for effects that cannot safely be retried.
- Set retry policy: Bound retries, classify failures, and make the terminal-failure path explicit for each kind of operation.
- Exercise crash points: Interrupt execution before and after effects, before recording outcomes, and during retry scheduling; verify no harmful duplicate occurs.
- Plan operations: Decide how staff will find executions, inspect events, correlate logs, resume or stop work, and handle history retention and sensitive data.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
