Payment Gateway

Assume a mature global platform processing 100M payment attempts per day across cards, wallets, and bank methods. Average traffic is about 1,160 payment attempts per second, and peak checkout traffic is 10x average, or about 11,600 attempts per second.

Assume 85 percent of attempts are authorized, 80 percent of authorizations are captured, 5 percent of captured payments are refunded, and every payment attempt generates several durable writes: payment state, provider attempt, idempotency record, outbox event, and ledger entries. Keep seven years of transaction history for audit and finance.

• Attempts: 100M attempts per day divided by 86,400 seconds is about 1,157 attempts per second, rounded to 1,160.
• Peak: a 10x peak multiplier gives about 11,600 payment attempts per second.
• Authorizations: 85 percent of 100M daily attempts gives 85M authorization successes per day.
• Captures: 80 percent of 85M authorizations gives 68M captures per day.
• Refunds: 5 percent of 68M captured payments gives about 3.4M refunds per day.
• Ledger entries: every capture writes at least debit and credit entries, every refund writes at least debit and credit entries, and the system also writes authorization holds, fee entries, reversals, reserves, and clearing entries. A practical planning number is around 285M immutable ledger entries per day.
• Ledger write TPS: 285M ledger entries per day divided by 86,400 seconds is about 3,300 ledger entries per second on average. With a 10x peak and extra fee or reserve lines, design for roughly 33,000 to 45,000 ledger writes per second.
• Storage: if payment intent, provider attempt, webhook, audit, and ledger data average about 2KB per payment attempt after indexing metadata is excluded, 100M attempts per day creates about 200GB raw per day. Seven years is about 511TB for the full history. With tiering, compaction, and cold object storage, keep roughly 140 to 180TB in queryable raw operational stores and archive the rest in cheaper immutable storage. Replication and indexes can multiply this by 3x or more.
• Webhooks: a payment can emit created, authorized, captured, settled, failed, and refunded events. If the average is 2.5 to 4 merchant deliveries per attempt including retries, expect 250M to 400M webhook delivery attempts per day.

• idempotency_keys.merchant_id, idempotency_key is unique and must be checked before mutating state.
• payment_intents.merchant_id, created_at supports merchant dashboards and customer support lookups.
• payment_attempts.provider, provider_request_id is unique so provider retries cannot create duplicate attempts.
• payment_attempts.provider_reference supports webhook correlation from external providers.
• ledger_entries.ledger_transaction_id supports balance checks that every ledger transaction has equal debits and credits.
• provider_events.provider, provider_event_id is unique for webhook replay protection.

A payment intent has many payment attempts because authorization, capture, refund, and retry operations are separate. Each successful money movement references one balanced ledger transaction with two or more immutable entries. Provider events may arrive before or after synchronous responses, so they correlate through provider_reference and can update the same payment attempt idempotently.

Pure NoSQL storage is risky for the ledger because financial correctness needs transactions, uniqueness constraints, and balance invariants. A common production split is distributed SQL for idempotency, state, and ledger records; a log or queue for outbox events; object storage for raw provider payloads and settlement files; and an OLAP store for analytics and merchant reporting.

Payment systems cannot rely on clients or networks to send a request exactly once. A buyer may tap twice, a mobile app may retry after a timeout, an API gateway may retry a 502, or a worker may crash after calling the provider but before storing the response. Without idempotency, any of those cases can double-charge the customer.

The gateway requires an idempotency key for every mutating operation. The key is scoped to merchant, operation, and request fingerprint. The first request inserts a key record and creates the resource inside the same transaction. Identical retries return the stored resource or response. A retry with the same key but a different amount, currency, or payment method fails with a conflict.

Exactly-once in payments is an effect, not a single network guarantee. Internally, the ledger enforces unique source IDs so a capture or refund can post once. Externally, provider adapters send stable provider request IDs and use PSP idempotency APIs when available. When the provider result is ambiguous, the gateway marks the attempt pending and reconciles through status polling, webhooks, or settlement files instead of issuing a blind second charge.

A clear state machine prevents illegal transitions. The core path is created -> authorized -> captured -> settled. Terminal or side states include failed, canceled, refunded, partially_refunded, chargeback_open, and chargeback_lost. Authorization reserves funds, capture starts money movement, settlement confirms funds reached the acquiring or platform account, and refunds reverse part or all of a captured payment.

No real gateway can hold a distributed transaction across merchant order service, fraud service, card network, acquiring bank, ledger, and webhook delivery. Use a saga. Each step persists local state and an outbox event before calling the next dependency. If a later step fails, the system performs compensating actions: void an authorization, refund a capture, cancel fulfillment, or open a manual review.

For operations that need local atomicity, use a two-phase local pattern: prepare the state transition and outbox record in one transaction, then have a worker perform the external call, then commit the final state and ledger effects after confirmed success. This avoids losing work when a process crashes between database commit and network call.

The operational payment state answers what the checkout is doing now; the ledger answers where the money is. A ledger transaction contains two or more entries and must balance by currency. For a capture, the system may debit a customer clearing account and credit merchant receivable, platform fee revenue, tax liability, and reserve accounts. For a refund, it appends reversing entries rather than editing the original capture.

Immutability matters because finance, auditors, and regulators need a history that cannot be silently rewritten. If a provider later reports a fee adjustment, chargeback, or settlement mismatch, the gateway posts an adjustment ledger transaction with a reason and source reference. Reports are derived from ledger entries, not from mutable payment status rows.

The ledger posting service should enforce uniqueness on source_type and source_id, currency balance checks, positive amounts, monotonic sequence numbers per account, and append-only permissions. Reads can be served from replicas or OLAP stores, but writes should go through one audited posting path.

External payment rails are heterogeneous and imperfect. One PSP may support separate authorize and capture; another may only support sale. Some return synchronous declines, some rely heavily on webhooks, and some produce settlement files hours later. Network timeouts do not mean the payment failed; they mean the gateway does not yet know.

Provider adapters isolate this complexity. They map internal operations to provider APIs, maintain provider-specific idempotency keys, translate error codes into normalized decline or retry categories, and expose capabilities such as partial capture, void, refund, 3-D Secure, network token support, and settlement timing.

Routing should consider merchant configuration, currency, country, card brand, provider health, cost, approval rate, and risk. Failover is safe before authorization. After an authorization has been created at one provider, capture and refund usually need to return to the same provider reference, so routing decisions become sticky.

Provider webhooks are asynchronous, duplicated, delayed, and sometimes out of order. The gateway should verify signatures, persist the raw payload, deduplicate by provider event ID, and process state changes idempotently. A capture webhook received before an authorization webhook should either be buffered or applied through a transition rule that can tolerate missing earlier events.

Retries need bounded exponential backoff with jitter. Retry only safe operations using provider idempotency keys. Distinguish retryable failures such as timeouts, 429, and 503 from hard declines such as stolen card or insufficient funds. After the retry budget is exhausted, the message goes to a dead-letter queue with merchant, payment, provider, request ID, payload hash, and last error.

Reconciliation is the safety net. Jobs compare internal attempts and ledger entries with provider status APIs, webhook logs, chargeback feeds, and settlement reports. The job should identify missing captures, duplicate provider references, unsettled payments, fee mismatches, and ledger transactions that do not match settlement totals.

The safest core gateway is one that never sees raw card data. Use hosted payment fields, mobile SDK tokenization, network tokens, or a dedicated vault service so the main payment API works with opaque tokens. Mask PAN to last four digits, never store CVV, encrypt sensitive metadata, and block sensitive fields from logs and analytics.

Fraud is part of the payment decision, not an afterthought. Score before authorization using velocity, device, IP, merchant, BIN, customer age, historical disputes, amount anomalies, and card testing patterns. High-risk traffic can be challenged, blocked, routed through stronger authentication, or throttled through rate limits.

PCI and fraud design also improve reliability. By keeping tokenization and risk evaluation explicit, the orchestrator can make deterministic decisions, explain declines, and replay safe operations without exposing sensitive data or re-running non-idempotent card collection flows.

Alternative one is a thin PSP wrapper. It is fast to launch and relies on the PSP for idempotency, vaulting, ledger-like reports, and webhooks. It is suitable for a small merchant but weak for marketplaces, multi-provider routing, and custom finance controls.

Alternative two is a wallet or stored-value system where users preload balances and purchases move internal money first. This can reduce card authorization latency for repeat users, but it adds regulatory, custody, and ledger complexity.

Alternative three is direct acquiring integration without a third-party PSP. It can reduce cost and improve control at massive scale, but it requires deeper compliance, certification, dispute handling, and bank relationships.

Do not build a full payment gateway if the product only needs occasional low-volume checkout in one country. A hosted PSP checkout can provide compliance, fraud tools, and payment method coverage with far less engineering and operational risk. Build the gateway layer when you need multi-merchant orchestration, high volume, custom ledgering, routing control, or deep product integration.