How a payment gateway guarantees that a request sent twice charges a customer once, even when the network refuses to say what happened.
A payment request times out. The customer's card may have been charged, or may not. The merchant's server has no way to tell from the timeout alone — and somebody has to decide what happens next.
This is the defining problem of payment engineering. Networks fail in the least helpful way possible: not by clearly refusing, but by going quiet after the request has already been sent. Retrying might fix nothing and charge twice.
The industry's answer is the idempotency key — a client-generated token that lets the server recognise a retry as the same logical operation and return the original outcome instead of performing a second charge.
Think of it like this: it is like posting a letter and losing your internet connection before the confirmation arrives. You do not know if it was delivered. Writing a reference number on the envelope means the post office can tell you whether they already have that exact letter, rather than sending a second one.
Money movement imposes constraints that ordinary application code does not face:
A duplicate charge cannot simply be deleted. It requires a refund, a reconciliation entry and often a support conversation.
A timeout carries no information about whether the operation succeeded, only that the answer did not arrive.
A payment traverses merchant, gateway, processor, network and bank, each of which can fail independently.
Many payment methods complete minutes or hours later, so the final state is not known when the request returns.
Every state transition must be auditable long after the fact, which rules out mutating records in place.
The architecture is built around one idea: a payment is not an action, it is a record with a state that changes over time. Every request either creates that record or reports its current state.
Sits in front of everything. Looks up the supplied key and either replays the stored response or allows exactly one execution to proceed.
In shortA doorman with a guest list — if your reference is already on it, you get the same answer as last time.
A durable record created before any money moves, holding the amount, currency, method and current state. It is the anchor for everything that follows.
In shortA file is opened for the payment before anything is attempted, so there is always something to look up.
The payment advances through explicitly defined states with only legal transitions permitted, so it cannot jump from created to refunded without passing through captured.
In shortThe payment walks a fixed path, and shortcuts are simply not allowed.
Talks to the card network or bank, translating between the gateway's model and each provider's protocol, and normalising their varied failure responses.
In shortThe translator that speaks to each bank in its own dialect.
Notifies the merchant of state changes with retries and signatures, since many outcomes arrive long after the original request closed.
In shortIt calls the merchant back later to say what finally happened.
Compares internal records against the processor's settlement files daily, catching anything the live path missed or recorded incorrectly.
In shortAn end-of-day audit that checks the books against the bank's own statement.
Before sending, the merchant's system creates a unique idempotency key for this logical payment and stores it alongside the order.
In shortA reference number is written down before the request is even sent.
On arrival, the gateway attempts to record the key. If it already exists, the stored response is returned and no charge occurs.
In shortThe gateway checks whether it has seen this reference before.
For a new key, a payment record is persisted in a pending state before any external call is made, so the attempt survives a crash.
In shortThe paperwork is filed first, so nothing is lost if the system dies mid-request.
The processor is asked to authorise the amount, reserving funds on the customer's card without yet moving them.
In shortThe bank sets the money aside but does not hand it over yet.
On capture, funds are claimed and the resulting state is written durably before the response is returned to the merchant.
In shortThe money is actually taken, and the outcome is recorded before anyone is told.
A webhook informs the merchant of the final state, and the payment is later matched against the processor's settlement file.
In shortThe merchant is told, and the next day the books are checked.
Every serious payment bug lives in the gap between an operation happening and the caller learning that it happened.
Two parties communicating over an unreliable channel can never both be certain the other received the last message. It is provably unsolvable, which is why payments manage uncertainty rather than eliminating it.
In shortYou can never be completely sure the other side got your message, so the system is built to cope rather than to be certain.
The charge succeeded but the response was lost. A naive retry charges the customer twice for one purchase.
In shortIt worked, but nobody heard back, so it gets done again.
The request never reached the processor. Here a retry is not only safe but necessary, and the system cannot distinguish this case from the previous one without a key.
In shortIt never happened, so it should be retried — but from the outside it looks identical to the case where it did.
The charge succeeded at the processor but the gateway crashed before recording it, leaving internal state disagreeing with reality until reconciliation.
In shortThe bank thinks it happened and the gateway does not, until the books are checked.
A tempting shortcut is to check whether a similar payment exists before creating one. Under concurrency this is the same read-then-write race that causes double charges in the first place:
In short: idempotency is not achieved by checking for duplicates. It is achieved by making the database refuse the second one. The uniqueness constraint on the key is the actual mechanism; everything else is bookkeeping around it.
Because outcomes arrive late, the notification path needs the same rigour as the request path — and something must still verify both.
Webhooks are retried with backoff until acknowledged, which means merchants will sometimes receive the same event twice and must handle it idempotently themselves.
In shortThe callback may arrive more than once, so the merchant also has to cope with duplicates.
Retries mean a later event can arrive before an earlier one, so consumers must apply events by version or timestamp rather than by arrival order.
In shortMessages can land in the wrong order, so each carries its place in the sequence.
Payloads are signed so the merchant can confirm the notification genuinely came from the gateway and was not forged by anyone who guessed the endpoint.
In shortEach callback is stamped so you know it is really from the payment provider.
The processor's daily file is the ultimate authority. Any disagreement with internal records is flagged and investigated rather than assumed benign.
In shortThe bank's own statement is the final word, and any mismatch gets chased.
Payment systems optimise for correctness first, and accept architectural constraints that a higher-throughput system might not:
| Layer | Common Choices |
|---|---|
| Idempotency Store | Relational table with a unique constraint, or Redis with durable backing |
| Payment Records | Strongly consistent relational storage with append-only transitions |
| State Machine | Explicit transition tables with database-enforced legal states |
| Processor Integration | Per-provider adapters with normalised error taxonomies |
| Async Processing | Durable queues for captures, refunds and webhook dispatch |
| Webhooks | Signed payloads, exponential backoff, dead-letter handling |
| Reconciliation | Batch matching against processor settlement files |
Idempotent payments are the canonical answer to a problem every distributed system eventually faces: how to retry safely when you cannot tell whether the first attempt worked. The pattern — a caller-supplied key, a uniqueness constraint, a stored response and a reconciliation pass — transfers directly to any operation that must not happen twice.
A quick, no-nonsense translation of the technical terms used above.
The property that performing an operation repeatedly has the same effect as performing it once.
A unique token supplied by the caller so the server can recognise a retry as the same logical request.
Reserving funds on a card without yet transferring them.
Actually claiming previously authorised funds.
The proof that two parties on an unreliable channel can never both be certain a message was received.
A callback from the gateway to the merchant announcing that something changed.
A guarantee that a message will arrive, possibly more than once, requiring the receiver to deduplicate.
Comparing internal records against the processor's settlement file to find and resolve discrepancies.