Design a Payment System

A payment system moves money on behalf of a merchant platform: it collects a payment from a buyer, records it, and later pays out to sellers. This design covers the pay-in flow (buyer to platform) and pay-out flow (platform to seller) for an e-commerce site that integrates a PSP (payment service provider, such as Stripe, Adyen, or PayPal) instead of talking to card networks directly. The core difficulty is correctness under failure: a payment must happen exactly once in effect, and every state must be auditable.

Requirements

Functional

• Pay-in: a buyer pays for an order using a card or another method; the platform receives the funds.
• Pay-out: the platform sends money to sellers after settlement.
• Refunds, full and partial.
• Payment status queries and history.
• Support for multiple PSPs and payment methods.

Non-functional

• Exactly-once effect: no double charge, no lost payment. Retries must be safe.
• Strong consistency and durability for all money records; an auditable trail of every state change.
• Reliability across unreliable dependencies: PSPs time out, return errors, or send duplicate and out-of-order callbacks.
• Security: card data must never be stored by the platform unless it is PCI DSS compliant. The design uses PSP-hosted fields or tokenization so card numbers do not enter platform servers.
• Latency is secondary to correctness; a payment taking a few seconds is acceptable.

Estimation

Assumptions: 1 million payment transactions per day.

• Average QPS: 1,000,000 / 86,400 is about 12 per second. Assume a 10x peak: about 120 per second. This is a low rate; the difficulty is correctness, not throughput.
• Records per payment: one payment order, one or more payment events, and 2 to 4 ledger entries. At about 1 KB total, 1,000,000 x 1 KB = 1 GB per day, about 365 GB per year. Financial records are retained for years, so plan archival to cheaper storage after an active window.
• Callback traffic: each payment produces a PSP webhook or two, roughly doubling inbound requests.

A single well-configured relational primary handles this rate. Sharding is reserved for growth beyond this scale.

What the numbers imply

• About 120 payments/s at peak, with 3 ledger rows each (about 400 inserts/s) and about 240 webhook calls/s, fits one primary. That keeps the unique idempotency constraint, the status update, and the ledger entries in one ACID transaction. Sharding would break that simplicity, so defer it.
• The cost of correctness is commit latency: synchronous replication to another zone adds a few milliseconds per commit, which is acceptable at this rate.
• 365 GB per year with years of retention means partitioning tables by month, keeping about 3 months hot, and archiving older partitions, while remaining queryable for audits and disputes.
• A 10x flash-sale peak is about 1,200 payments/s, still comfortable for one primary. The hazard is not throughput but contention on hot rows such as a single merchant balance (see failure modes).

API design

POST /v1/payments                { checkout_id, order_id, amount, currency, buyer_id, idempotency_key } -> { payment_id, status }
GET  /v1/payments/{payment_id}   -> { status, amount, events[] }
POST /v1/payments/{payment_id}/refunds  { amount, idempotency_key } -> { refund_id, status }
POST /v1/webhooks/psp/{psp}      (called by the PSP; verified by signature)

The idempotency_key is generated by the client for each logical payment attempt. The payment service stores it with a unique constraint. If the same key arrives again, it returns the stored result instead of starting a new payment (see idempotency and pagination). Amounts are integers in the smallest currency unit (cents) with an explicit currency code; floating point is never used for money.

Data model

Money data requires ACID transactions, so a relational database is used.

• payment_order: payment_order_id, checkout_id, buyer_id, seller_id, amount, currency, status, psp, psp_reference, idempotency_key (unique), created_at, updated_at. Status values: NOT_STARTED, EXECUTING, SUCCESS, FAILED, REFUNDED.
• payment_event / status history: append-only record of every status change with timestamp and cause. Never updated in place.
• ledger_entry: entry_id, transaction_id, account_id, direction (DEBIT or CREDIT), amount, currency, created_at. Append-only.
• account: buyer, seller, platform revenue, PSP clearing.

The ledger uses double-entry bookkeeping: every money movement is recorded as at least two entries, a debit and a credit of equal amount, in one transaction. The sum of all debits equals the sum of all credits at all times, which gives a built-in consistency check. For a $100 sale with a $10 platform fee:

Ledger entries are never modified; a correction is a new reversing entry. This immutability is what makes audits and reconciliation possible.

High-level design

flowchart LR
    B["Buyer"] --> CO["Checkout / order service"]
    CO --> PS["Payment service"]
    PS --> DB[("Payment DB: orders, events")]
    PS --> EX["Payment executor"]
    EX --> PSP["PSP (external)"]
    B -->|"card details (hosted fields)"| PSP
    PSP -->|"webhook"| WH["Webhook handler"]
    WH --> PS
    PS --> LG["Ledger service"]
    LG --> LDB[("Ledger DB")]
    PS --> Q[["Event bus"]]
    Q --> WA["Wallet / seller balance"]
    Q --> NT["Notification"]
    PSP --> REC["Reconciliation job"]
    LDB --> REC
    REC --> AL["Alerts and manual review"]

• Payment service. Receives payment events from checkout, creates the payment order, and coordinates the executor and ledger.
• Payment executor. Calls the PSP. Isolated so PSP-specific code, timeouts, and retries are in one place.
• Hosted payment page or fields. The buyer's card details go directly from the browser to the PSP, which returns a token. The platform never sees the card number, reducing PCI scope.
• Webhook handler. Verifies the PSP signature and applies status updates idempotently.
• Ledger service. Records double-entry transactions after a successful payment.
• Reconciliation. Compares internal records with PSP settlement files.

Deep dive

1. Payment flow with a hosted page

sequenceDiagram
    participant B as Buyer
    participant CO as Checkout
    participant PS as Payment service
    participant PSP as PSP
    B->>CO: place order
    CO->>PS: create payment (idempotency key)
    PS->>PS: insert payment_order NOT_STARTED
    PS->>PSP: register payment (returns token and redirect URL)
    PS-->>B: payment page URL
    B->>PSP: enter card details on PSP page
    PSP-->>B: redirect to return URL
    PSP->>PS: webhook: payment succeeded (signed)
    PS->>PS: status SUCCESS, write ledger entries (one transaction)
    PS-->>CO: payment completed event

The return redirect from the PSP only informs the browser; the webhook (or a status poll to the PSP) is the authoritative signal. The service does not mark a payment successful based on the browser redirect.

2. Exactly-once effect: idempotency, retries, and unknown outcomes

Failure points: the request to the PSP times out, the PSP charges but the response is lost, the webhook arrives twice or never, or the service crashes between the PSP call and the database write. Rules:

• Persist intent before acting. Insert the payment order in NOT_STARTED (with the unique idempotency key) and commit before calling the PSP. After a crash, recovery finds orders stuck in EXECUTING and resolves them.
• Send the idempotency key to the PSP. Major PSPs accept a per-request idempotency key so that a retried charge is not executed twice. Use the same key on every retry of the same payment.
• Unknown outcome is a state, not an error. On timeout, the status stays EXECUTING. A recovery job queries the PSP by reference to find the true state. It does not create a new charge.
• Retry with backoff. Retries follow exponential backoff with jitter, only for operations that are idempotent at the PSP.
• Idempotent webhook handling. Each webhook carries an event id. The handler records processed ids (unique constraint) and ignores repeats. Status transitions are validated against the state machine, so an out-of-order update that would move a payment from SUCCESS back to EXECUTING is rejected.

import java.util.HashMap;
import java.util.Map;

public class Main {
    record Payment(int id, long amount) {}

    record Result(Payment payment, boolean created) {}

    static class PaymentService {
        private final Map<String, Payment> byKey = new HashMap<>();
        private int nextId = 0;

        // In a database this is an insert guarded by a unique constraint on the key.
        synchronized Result pay(String key, long amount) {
            var existing = byKey.get(key);
            if (existing != null) {
                return new Result(existing, false); // replay: return the stored result
            }
            var payment = new Payment(++nextId, amount);
            byKey.put(key, payment);
            return new Result(payment, true);
        }
    }

    public static void main(String[] args) {
        var service = new PaymentService();
        var first = service.pay("key-1", 1000);
        var retry = service.pay("key-1", 1000); // client retry
        System.out.println(first.payment() + " created=" + first.created());
        System.out.println(retry.payment() + " created=" + retry.created());
    }
}

Payment[id=1, amount=1000] created=true
Payment[id=1, amount=1000] created=false

A production version stores the key with the request hash and rejects a reuse of the key with different parameters. The synchronized method stands in for the unique constraint: two concurrent requests with the same key cannot both create a payment.

State machine:

stateDiagram-v2
    [*] --> NOT_STARTED
    NOT_STARTED --> EXECUTING: sent to PSP
    EXECUTING --> SUCCESS: confirmed
    EXECUTING --> FAILED: declined or error confirmed
    SUCCESS --> REFUNDED: refund
    FAILED --> [*]
    REFUNDED --> [*]

3. Ledger consistency and the database transaction

The status change to SUCCESS and the ledger entries are written in one local database transaction, so they are both visible or neither. Other services (order fulfillment, notifications) learn about the payment by events. Publishing the event directly after the commit risks losing it on a crash; the transactional outbox writes the event to an outbox table in the same transaction and a relay publishes it at least once. Consumers deduplicate (see idempotent consumers).

If the payment service and the ledger are separate services with separate databases, a distributed transaction is avoided by a saga: the payment service records the result, emits an event, and the ledger service applies its entries idempotently keyed by the payment id.

4. Reconciliation

Even with careful design, internal state and the PSP's records can diverge (a lost webhook, a bug, a manual PSP-side action). Reconciliation compares them periodically:

1. Each day the PSP provides a settlement file listing every transaction.
2. A job matches each row to an internal payment by PSP reference and compares amount, currency, and status.
3. Mismatches are classified: missing internally, missing at the PSP, amount differs, status differs.
4. Automatable cases (for example, a success at the PSP with an internal EXECUTING) are corrected by applying the missing transition. Others go to a manual review queue.

Reconciliation is the last line of defense that detects errors the runtime protections missed.

5. Consistency between the payment service and the PSP under timeouts

A long timeout to the PSP blocks a thread; a short one increases unknown outcomes. Use a moderate timeout, mark EXECUTING, and reconcile asynchronously by polling the PSP status with backoff until a terminal state is reached or a manual review threshold is exceeded. Never respond FAILED to the buyer for an unknown outcome, because a charge may have succeeded.

6. Thresholds and consistency choices

• Consistency. Payment and ledger writes go to the primary with unique constraints and conditional updates (UPDATE ... SET status='SUCCESS' WHERE id=? AND status='EXECUTING'), which makes concurrent webhook and poller updates safe without explicit locks. Status reads right after a payment go to the primary or use read-your-writes; replicas may lag by seconds and could show EXECUTING after success.
• Timeouts and polling. Use a PSP call timeout of about 10 seconds (card authorizations normally finish in 1 to 3 seconds). Recovery polling backs off at 1, 5, 15, and 60 minutes, then escalates to manual review after about 24 hours in EXECUTING.
• Ledger invariant. Each ledger transaction is inserted in one database transaction and checked (SUM(debit) = SUM(credit)) before commit. A nightly job re-verifies the global balance.
• Alternative to a saga. Two-phase commit across payment and ledger databases blocks on a crashed coordinator. A single database or an outbox-plus-saga has no such blocking, so choose one of those.

Bottlenecks and scaling

• Throughput is not the constraint at 120 requests per second. Scale by shard key (payment order id or merchant id) only when needed (see sharding).
• PSP availability: support multiple PSPs with routing and failover. Failover for new payments is safe; for in-flight payments, resolve the original PSP state first to avoid double charging.
• Database durability: synchronous replication and backups for ledger data; point-in-time recovery (see multi-region and disaster recovery).
• Webhook floods: acknowledge quickly, enqueue, and process asynchronously; verify signatures first.
• Hot merchants: rate limit and partition by merchant to isolate.
• Security: encrypt sensitive fields at rest, restrict access, tokenize card data, and keep an audit log (see encryption and TLS).

Failure modes

• PSP timing out or down. A circuit breaker opens when, for example, more than 50% of calls fail over 30 seconds. New payments route to a second PSP; in-flight payments stay EXECUTING and are resolved against the original PSP. Buyers see "processing", never "failed".
• Crash between the PSP call and the DB write. The EXECUTING row was committed before the call, so the recovery job finds it and queries the PSP by reference with the same idempotency key.
• Primary failure. Failover to the synchronous replica takes tens of seconds. The API returns retryable 503 responses, the client retries with the same key, and no money moves without a committed intent. Asynchronous replication would risk losing committed payments on failover, which is not acceptable for the ledger.
• At 10x scale. Peak is about 1,200 payments/s and 4,000 ledger inserts/s. Partition by month, and shard by merchant or hash of the idempotency key if needed (the key must route to the same shard on retry). The real bottleneck is a hot account row (platform revenue updated by every sale). Do not update a running balance in the payment transaction; insert entries only and derive balances from entries plus periodic aggregation.

Trade-offs and follow-up questions

• Synchronous versus asynchronous confirmation: synchronous is simpler for users; asynchronous with webhooks is required for methods that settle later (bank transfers, wallets).
• Authorization and capture: card payments often first authorize (reserve funds) and later capture (collect). Authorization holds expire, so capture must occur within the PSP's window.
• Refunds and chargebacks: refunds are new ledger transactions referencing the original; chargebacks are initiated by the card issuer and arrive as PSP events that require reversing entries.
• Multi-currency: store each ledger entry with its currency and record exchange rates used; convert only at defined points.
• Why not store status only, without events? Without an append-only history the system cannot answer what happened and cannot reconcile reliably.
• How to make pay-outs safe? The same principles: idempotency key per payout, persisted intent, unknown-outcome state, reconciliation with the bank file.
• The user double-clicks Pay? Both requests carry the same idempotency key, and the unique constraint makes the second return the first result.
• The PSP charged but our DB says FAILED? That would only happen if we treated an unknown outcome as failure. The reconciliation job detects a PSP success against an internal failure and either applies the missing transition or refunds the buyer, with a ledger entry either way.
• Concurrent partial refunds? Lock the payment order row and check that the sum of refunds does not exceed the captured amount before inserting each refund.