A payment marked “failed” in your database is not necessarily a payment that failed. Often it is a payment whose outcome your system never learned. The usual culprits are a timeout, a retry that looks like a new request, a bank decision that arrives after the first response, or a notification delivered twice. In each case the local record is a guess presented as a fact.
This article walks through those mechanisms as hypotheses to check against your own logs. It doesn’t describe a specific incident, processor or schema. It then covers the design habits that stop a database from inventing outcomes.
As an Amazon Associate I earn from qualifying purchases.
The core mistake: treating “no answer” as “declined”
When your service calls a payment provider and the call times out, three things may have happened. The request never arrived, it arrived and is still processing, or it arrived and finished, but the response was lost. Your code can only see “no response.” If an exception handler turns that into status = 'failed', you’ve recorded an unknown as a confirmed refusal.
Stripe’s developer material lists network timeouts, server crashes, database locks, downstream API errors and user interruptions as ordinary failure conditions in distributed payment systems, not rare edge cases. Its answer is the idempotency key: repeating a request with the same key returns the original outcome rather than creating a second operation.
#1 Best Overall
Five ways a local record ends up wrong
1. Timeout recorded as failure
The provider accepted the charge, but your client gave up first. The database says failed, the provider says succeeded. If the customer then retries, you may also collect a second payment.
2. Initial response treated as final
PayPal notes that a payment may not fail in the initial response. A bank can authorize at first and decline later. PayPal’s guide recommends webhooks to track these later outcomes. A system that stops listening after the first response will show a status the provider no longer holds. This works in both directions: it can show success for a payment that later failed.
3. Duplicate or replayed notifications
Notifications are retried too. ePay recommends an idempotent handler: store the payment or transaction ID, check whether you’ve already processed it, and don’t repeat the state change on a duplicate. Without that, a replay can apply the same ledger effect twice. It can also overwrite a newer status with an older one if events arrive out of order.
Free tools Windows power users keep installed
One-click scans. No signup required.
4. A retry confused with a new attempt
Plaid advises checking the status of the prior attempt before retrying, so an already successful payment isn’t duplicated. A genuinely new attempt uses a new idempotency key. If your code generates a fresh key on every retry, the provider has no way to know it’s the same intent. If it reuses one key for a deliberately new attempt, you’ll get the old result back.
5. Internal status mixed with provider status
One column often has to hold several things: what the provider said, what your retry scheduler is doing, and what the UI shows. When they share a field, an internal “retry pending” can overwrite a provider-confirmed result, or the reverse.
Not all failures deserve the same response
PayPal lists declined or expired payment methods, insufficient funds, risk restrictions and business validation errors. GOV.UK Pay separately documents rejected payment methods, expiry, cancellation and provider errors. These fall into different buckets:
| Category | Example | Sensible response |
|---|---|---|
| Unknown outcome | Timeout, dropped connection | Query the provider or wait for its event. Don’t charge again yet. |
| Transient | Provider-side error, temporary connectivity issue | Bounded retry, reusing the same idempotency key. |
| User-correctable | Expired card, insufficient funds | Prompt the customer to fix it or use another method. |
| Final refusal | Risk restriction, rejected method | Stop automatic retries and record the reason. |
The buckets are an organizing aid, not a provider taxonomy. Map each provider’s own codes into them and check that provider’s documentation. Salesforce documents the same idea at product level: retry rules based on error category, intervals, maximum attempts and payment gateway.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsRetry schedules are provider-specific
PayPal’s subscription documentation, last updated September 14, 2026, describes a configurable flow that retries every five days, up to twice per billing cycle. After the second failure, the amount is added to the next cycle’s balance. Treat that as a dated PayPal example, not an industry norm. GOV.UK Pay offers a similar caution: its API reference (accessed October 5, 2026) says a payment expires if the payer doesn’t complete it within 90 minutes. That figure describes its own flow, not a general timeout standard.
Rank #3
Designing records that can’t invent outcomes
Model attempts, not just orders
Keep one row per payment attempt. Useful fields include the provider transaction or payment ID, your request identity (the idempotency key), timestamps, amount and currency, and both a normalized status and the raw provider status. The sources don’t prescribe a universal schema, so adapt this to your system.
Add an explicit unknown state
Make “outcome unknown” a first-class status, separate from declined. A timeout moves an attempt there. Only a provider response, a webhook, or a provider lookup moves it out. Until then, don’t start a new attempt unless you’re certain it’s safe.
-- illustrative sketch, not tied to any engine or provider
payment_attempts(
id, order_id, idempotency_key,
provider_txn_id, provider_status_raw,
status, -- pending | unknown | succeeded | failed
created_at, last_confirmed_at
)
processed_events(event_id PRIMARY KEY, received_at)Make event handling safe to repeat
- Verify the notification as your provider requires.
- In one transaction, insert the event or transaction ID into a table with a unique constraint.
- If the insert conflicts, you’ve seen it before: acknowledge and stop.
- Otherwise apply the state change, guarded so it can’t move a record backward from a newer provider-confirmed state.
- Commit, then acknowledge.
Reuse keys for the same intent
Persist the idempotency key with the attempt before sending the request, so a crash and restart sends the same key. Generate a new key only when you intentionally create a distinct attempt, following that provider’s semantics.
Reconcile against the provider
Webhooks can be missed, so periodically compare local attempts against provider state and event history. Start with attempts stuck in pending or unknown, and with failures that have a provider ID. The sources recommend webhooks and prior-attempt checks but give no universal reconciliation cadence, so choose one based on your volume and how long customers can tolerate a wrong status.
Keep provider-confirmed outcomes separate from internal retry state and UI display state. The display layer should derive from the first two, never write back into them.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.A diagnostic checklist for suspected phantom failures
- Do the “failed” rows share a timeout or exception message instead of a provider decline code?
- Do any have a provider transaction ID that the provider shows as successful?
- Are there customers with a failed attempt followed shortly by a success, or two successes?
- Does your retry code generate a new key per call?
- Is the webhook endpoint returning errors that cause redelivery, and does the handler deduplicate?
- Can an older event overwrite a newer status?
If several answers are yes, the database isn’t lying out of malice. It’s recording what it was able to see, and the fix is to represent what it couldn’t see.
Limits of this analysis
These are failure mechanisms drawn from provider documentation by Stripe, PayPal, Plaid, ePay, GOV.UK Pay and Salesforce, not findings from a particular production system. No industry failure or recovery rate is claimed. Confirm status definitions, retry behavior and idempotency semantics in your own provider’s current documentation.
Recommended Free Tools
For broader background on reliability and consistency in data systems, Designing Data-Intensive Applications by Martin Kleppmann (a second edition with Chris Riccomini has been reported for 2026) is a relevant read. Check edition details before buying.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
