Three weeks after launching a checkout flow for a client, we got a support ticket that made my stomach drop: a customer had been charged twice for the same order. Same amount, same order ID, ninety seconds apart. Nothing was wrong with our payment gateway integration in the way we first suspected. The bug was simpler and worse: our mobile client had a retry-on-timeout policy, the first request had actually succeeded server-side, the response just never made it back to the phone before the timeout fired, and the retry created a second, entirely separate charge. That bug is why every payment endpoint we build now requires an idempotency key, no exceptions.

What idempotency actually means here

An idempotent operation produces the same result no matter how many times you perform it. `PUT /users/5 {name: "Alex"}` is naturally idempotent — running it once or five times leaves the user in the same state. `POST /charges {amount: 5000}` is not, by default, because every POST creates a new resource. The fix isn't to avoid POST for creation, it's to let the client supply a unique key that identifies the *intent*, not the request, and have the server treat repeated requests with that same key as "already handled."

The implementation we settled on

The client generates a UUID once, when the user first clicks "pay," and sends it as an Idempotency-Key header on every attempt, including retries. The server checks that key against a table before doing anything else.

async function createCharge(req, res) {
  const idempotencyKey = req.headers['idempotency-key'];
  if (!idempotencyKey) {
    return res.status(400).json({ error: 'Idempotency-Key header required' });
  }

  const existing = await db.idempotencyKeys.findOne({ key: idempotencyKey });
  if (existing) {
    // Same key, same request: return the original result, don't re-run the charge
    return res.status(existing.statusCode).json(existing.responseBody);
  }

  const charge = await paymentGateway.charge(req.body);

  await db.idempotencyKeys.insertOne({
    key: idempotencyKey,
    statusCode: 201,
    responseBody: charge,
    createdAt: new Date(),
  });

  return res.status(201).json(charge);
}

The key detail people miss: this needs to be atomic. If two requests with the same key arrive within milliseconds of each other — which happens more than you'd think, thanks to overeager retry logic on flaky mobile networks — you need a unique constraint on the key column in your database, not just an application-level check-then-insert. We learned this the hard way in staging, where a load test sent concurrent duplicate requests and briefly created two charges before we added the unique index. Let the database enforce the invariant your application logic is trying to enforce; don't trust a race-free world that doesn't exist.

How long do you keep the keys

This is a real tradeoff, not a solved problem. Keep idempotency records forever and your table grows without bound. Expire them too aggressively and a legitimate slow retry, arriving after the window closes, creates a duplicate charge, which is exactly the bug you built this system to prevent. We settled on 24 hours for payment endpoints specifically, because that comfortably covers any realistic client retry window, including a user who closes their laptop mid-checkout and retries the next morning from the same cart session — which, yes, actually happened, and the 24-hour window caught it correctly.

What this buys you beyond payments

Once you build idempotency into one endpoint properly, it's tempting to add it everywhere, and honestly for anything that mutates state and might get retried by a client, it's worth it. Order creation, inventory reservation, email-sending triggers — anywhere a duplicate side effect is worse than a slightly more complex API contract. We now treat idempotency keys as a default requirement on any endpoint that isn't naturally idempotent, the same way we default to requiring auth headers. It's not extra work once it's a habit; it's extra work only the first time you build it without one and have to explain to a client why their customer got double-charged.

The uncomfortable truth

Most API design guides treat idempotency as an edge case worth a paragraph. In payment systems it's not an edge case, it's the whole game. Networks are unreliable, clients retry, and "just don't retry" is not advice you can give to a mobile app running on someone's spotty coffee shop wifi. Build the idempotency layer before you need it, because the first time you need it will be a customer's money, and by then it's already too late to design it calmly.