API Idempotency, Retries and Timeouts: Making Failure Predictable

The incident that makes this concrete

A payment endpoint returned 503 for about forty seconds. The mobile client retried three times, the API gateway retried three times, and the service's own HTTP client retried three times. That is not three retries, it is 27 requests per user action. Forty seconds of instability became several minutes of self-inflicted load, and a handful of customers were charged twice. Nothing in that chain was individually unreasonable. The composition was.

Timeouts, retries and idempotency are usually discussed separately. They only work when designed together, and the reason is arithmetic: retries multiply across layers, and without idempotency every multiplication is a chance to duplicate a side effect.

Timeouts: allocate a budget, do not guess

The common mistake is choosing a timeout per service in isolation. Start from the opposite end: decide the slowest acceptable response for the user, then divide it downward. If a request must finish within 2 seconds, a workable split is 200 ms connect, 800 ms read at the first hop, and 1 second total including retries.

Every upstream timeout must be larger than the sum of the downstream timeouts it can trigger. If it is not, the upstream gives up and retries while the downstream is still working, and now two copies of the same work are in flight.

Retry only failures that can succeed

Backoff with jitter

Backoff without jitter makes every client retry at the same instant, which reproduces the original spike. Full jitter is the simplest scheme that works:

const BASE = 200, CAP = 10_000;

function delay(attempt, random = Math.random) {
  const ceiling = Math.min(CAP, BASE * 2 ** attempt);
  return random() * ceiling; // full jitter: uniform in [0, ceiling)
}

// attempt 0..5 with BASE=200, CAP=10000
// max ceilings: 200, 400, 800, 1600, 3200, 6400

Full jitter spreads attempts across the whole window and is what AWS recommends for most workloads. Equal jitter, which keeps half the delay fixed, is slightly gentler when you need a guaranteed floor.

Cap amplification with a retry budget

Per-request caps are not enough when the overall error rate is high. A retry budget caps retries as a fraction of total requests:

// token bucket: each success adds a token, each retry spends one
class RetryBudget {
  constructor(ratio = 0.1, max = 100) {
    this.ratio = ratio;
    this.max = max;
    this.tokens = 0;
  }
  onSuccess() {
    this.tokens = Math.min(this.max, this.tokens + this.ratio);
  }
  trySpend() {
    if (this.tokens < 1) return false;
    this.tokens -= 1;
    return true;
  }
}

When the bucket is empty, retries stop. During a real outage this is the difference between a bad minute and a bad hour.

Do not retry at every layer

Gateway, service mesh, application client and mobile SDK can all retry independently. For a single user action those multipliers compound: 3 × 3 × 3 = 27. Pick one layer to own retries for a given dependency and disable the others, or make every layer consult the same budget.

Idempotency is what makes retries safe

An operation is idempotent when executing it many times has the same effect as executing it once. Reads are naturally idempotent. Writes are not, unless you design them to be.

The standard pattern is a client-generated idempotency key sent with the request:

1. The client generates a key per business operation, typically a UUID, and reuses it for every retry of that operation.
2. The server inserts the key into a table with a unique constraint before doing the work.
3. If the insert succeeds, the server performs the operation and stores the response.
4. If the insert conflicts, the server returns the stored response, or waits if the first request is still in progress.
5. Keys expire, for example after 24 hours, so the table does not grow forever.

CREATE TABLE idempotency_keys (
  key          text PRIMARY KEY,
  request_hash text        NOT NULL,
  status       text        NOT NULL, -- in_progress | completed
  response     jsonb,
  created_at   timestamptz NOT NULL DEFAULT now(),
  expires_at   timestamptz NOT NULL
);

The race nobody tests

Two retries can arrive concurrently with the same key. If both read "not found" before either inserts, both execute. The unique constraint on the primary key is what actually prevents the duplicate; the read-before-write check is only an optimisation. After the insert conflicts, the second request must wait for status = completed or return a 409, never proceed.

Also compare the request hash. If the same key arrives with a different payload, that is a client bug, and returning the first response would hide it.

Exactly-once is a marketing term

Distributed systems give you at-least-once delivery with retries, or at-most-once without them. Exactly-once is achieved only by combining at-least-once delivery with idempotent processing, which is what the key table does. If someone promises exactly-once without an idempotency mechanism, ask what happens when the acknowledgement is lost.

Observability

Log a request ID on every attempt, and emit four metrics: retry rate, retry success rate, extra latency introduced by retries, and duplicate-key hits. A persistently high retry rate is not a retry-strategy problem, it is a signal that the dependency is unhealthy.

Method semantics: why PUT is easier than POST

HTTP already encodes part of this. GET, HEAD and PUT are defined as idempotent; POST is not. A PUT to a known resource replaces it, so replaying it is harmless. A POST creates a subordinate resource, so replaying it creates two. That is why payment and order endpoints, which are naturally POST, need an explicit key, while a well-designed update endpoint often does not.

Choosing the key

Generate the key on the client, not the server. A server-generated key changes on every retry and defeats the purpose. Scope it to the business operation rather than the HTTP request, and keep it stable across attempts. A UUIDv4 is fine; if you want ordering for debugging, UUIDv7 or ULID gives you time-sortable keys without coordination.

Async consumers need the same treatment

Retries do not stop at HTTP. Message brokers deliver at least once, so a consumer that crashes after processing but before acknowledging will see the same message again. The fix is the same idea at a different layer: record the message ID in a processed table inside the same transaction as the side effect, or make the handler naturally idempotent by writing an absolute state rather than incrementing a counter.

Watch for the ordering trap as well. Two messages for the same entity can arrive out of order after a retry. Include a version or timestamp and reject stale writes, rather than letting an older message overwrite a newer one.

When the work spans services

An idempotency key protects a single service. A business operation that touches three services needs a saga: each step has a compensating action, and the orchestrator records progress so a retry resumes rather than repeats. The key insight is that the orchestrator's state must be durable before the next step starts; otherwise a crash between steps leaves the system in a state no retry can reason about.

Do not solve this with a distributed lock

Reaching for a distributed lock to prevent duplicate execution is a common detour. Locks fail open under partitions, expire while the work is still running, and add a dependency that can take the whole path down. The unique constraint on the idempotency key is simpler and more reliable, because it fails closed.

Checklist

1. Timeouts allocated top-down, each upstream larger than the sum below it.
2. Retries only on retryable errors, with full jitter and a hard cap.
3. A retry budget that stops amplification during an outage.
4. Retries owned by exactly one layer per dependency.
5. Idempotency keys on every non-idempotent write, enforced by a unique constraint.
6. Request hash checked so a reused key with a different payload is rejected.
7. Metrics for retry rate and duplicate keys, not just error rate.

Try it: UUID generator, create request IDs and idempotency keys.