The lost response problem
An API can finish an operation even when the client never receives its response. Retrying a payment, invoice or provisioning request with a new identity may create a second operation. An idempotency key lets the server recognize that the retry represents the same intent. It is neither authentication nor a guarantee that every downstream service is idempotent.
Use one key for one logical operation. Keep the key unchanged across timeouts and network retries. A genuinely new purchase needs a new key even when its amount matches the previous purchase.
Design the durable record
An illustrative record contains tenant_id, operation_type, key, request_hash, status, result_reference, created_at and expires_at. A unique constraint on the tenant, operation and key arbitrates concurrent attempts. The request hash prevents accidentally reusing a key for a different payload.
| Existing state | Response policy |
|---|---|
| No record | Atomically claim the operation |
| In progress, same request | Return an explicit pending or retry response |
| Complete, same request | Return the stored outcome |
| Same key, different request | Reject the mismatch |
| Uncertain external outcome | Reconcile before repeating the side effect |
Stripe's documentation describes its own key behavior, parameter comparisons and retention policy. Treat those as provider-specific rules; do not assume another API follows the same contract.
The response is lost after the operation commits
A customer taps Pay, the server completes its work, and the mobile connection drops before the response arrives. The client sees an error, but the server sees a completed operation. Retrying with a fresh identifier makes the retry look like a second purchase. Reusing an idempotency key tells the server that the client is asking about the same logical attempt. The key belongs to the operation, not to a particular HTTP connection or retry counter.
For a concrete order API, create the key when the user begins the confirmed purchase and keep it across timeout retries. Store a normalized request fingerprint alongside the key, operation state and eventual response. If the same key arrives with a different basket or amount, reject the mismatch rather than silently returning an unrelated result. If processing is still in progress, expose a documented pending state or a status endpoint. A cached success response alone does not handle concurrent first requests; the server must also claim the key atomically before performing the protected effect.
Store the record beside the effect it protects
For an operation confined to one database, insert the idempotency record and apply the business mutation in the same transaction. A unique constraint on the scoped key makes concurrent attempts meet at one authority. The scope should include the tenant or account and endpoint semantics; a short key accidentally reused by another customer must not reveal their response. Keep the stored response small and avoid logging secrets simply because they passed through an idempotent request.
For an external payment, the local database cannot atomically commit the provider's state. Forward a stable provider-supported idempotency identity, persist the provider reference, and reconcile ambiguous outcomes before issuing a new operation. For event consumers, use a stable event or business identifier and record deduplication with the durable mutation. Expiration is a product decision: if records disappear before clients stop retrying, an old key can become a new effect. Publish a retention window and make retry behavior fit it. Provider-specific rules, including what errors are retained, must be checked against that provider's documentation rather than copied from another API.
Keep state changes together
If your effect is entirely inside one database, create the deduplication record and business row in one transaction. A crash should not leave a completed key without the corresponding business result. PostgreSQL's constraint documentation explains the uniqueness primitives used to arbitrate conflicting writes.
Remote effects need a recovery path. If a provider accepts your request and your worker crashes before recording success, a local pending row cannot tell you whether the effect happened. Pass a stable provider key where supported, store its reference, and reconcile its status. An outbox can make local event publication durable; it does not make an arbitrary remote effect transactional.
Retention is part of correctness
Choose the storage window from the longest plausible retry or redelivery period. Expiring a record means a later duplicate may look new. A permanent business constraint can still prevent duplicate invoices after the temporary key expires. Scope keys to the authenticated tenant so one customer cannot replay another customer's stored response. Avoid logging full payloads if they contain private data.
Exercise the failure paths
Send simultaneous identical requests. Reuse a key with a changed amount. Crash after the database commit. Crash after the provider response. Retry after expiry. For each case, inspect both durable records and external effects. Success means the intended business invariant holds, not just that the endpoint returns 200.
Use AiFolder JSON formatting for a sanitized request example and compare distributed locking when workers also need coordination.
Test ambiguity rather than only repeated successes
A meaningful verification plan sends two simultaneous requests with the same key, not just two sequential requests after the first has finished. It then changes the request payload while retaining the key and checks that the server refuses the mismatch. Interrupt the process immediately before and after the durable mutation. The expected outcome is one committed effect or a recoverable pending operation, with enough state to decide which occurred.
Watch for idempotency records left indefinitely in processing. A timeout alone does not prove that the effect failed; a recovery worker should consult the authoritative destination before clearing or replaying the operation. Track duplicate-hit rate, payload conflicts, pending-record age and reconciliation outcomes independently. Frequent duplicate hits can be normal on unreliable networks, while old pending records usually need attention. The system should preserve the original response contract as well as the business invariant: returning a different identifier or contradictory status for the same completed operation forces clients to invent their own unreliable reconciliation logic.
