Skip to content
build-3000
backend

Idempotency and Retries

Operations safe to repeat: idempotency keys, dedup windows and retry budgets that respect both sides.

Before this: Event-Driven Architecture

Objective

Make unsafe operations repeatable by design: client-supplied idempotency keys with stored responses, dedupe windows, concurrent-submit handling - plus retry policies with budgets so clients back off politely instead of stampeding.

Vocabulary

The words to use when directing your agent.

idempotent operation
One whose repetition after the first call leaves no additional effect.
idempotency key
A caller-provided unique marker letting the server recognize and absorb repeats.
dedupe window
How long the server remembers processed keys before forgetting them.
retry budget
A cap on total retry traffic so clients cannot amplify incidents.
at-least-once
Delivery guarantee promising attempts happen - possibly more than once.

Understand it

The key lifecycle

Client generates a unique key per logical operation (uuid works) and sends it on the request. Server checks: seen this key recently? Return the STORED original response without re-executing. Otherwise execute, store key-plus-response inside the same transaction as the effect, set expiry. The subtle part is concurrency: two identical requests racing must resolve to one execution - via unique constraints or locks - never two.

Retry etiquette

Clients owe the server restraint: bounded attempts, exponential backoff with jitter, honor Retry-After headers, and a total budget so a misbehaving loop cannot multiply load indefinitely. Servers owe callers clarity: 409-style signals for conflicts, idempotency support on every unsafe endpoint. Politeness on both sides is what keeps partial failure from becoming total failure.

Core concepts

Retries turn blips into successes - or bugs into disasters

A timeout followed by a successful retry rescues ordinary network weather. But if the FIRST attempt actually landed server-side, the blind retry now double-charges, double-books or double-sends. Idempotency keys convert dangerous retries into safe ones: same key, same stored result - the operation addresses itself.

Exactly-once is really at-least-once plus dedupe

No practical system guarantees exactly-once execution by magic; they guarantee at-least-once ATTEMPTS and give you the tools to make effects once-only: keys, stored responses, dedupe windows. Understanding this reframes every 'guaranteed exactly-once' claim you will ever read.

Examples

Double-click, single charge

User double-clicks Pay; both requests carry the same idempotency key. First executes and stores its response under the key. Second finds the stored response and returns it verbatim - one charge, two identical replies, calm user. Without the key: two charges, one furious support ticket.

The crash between accept and apply

Server accepts a transfer, crashes before committing. Client retried with the SAME key: the half-finished attempt left nothing stored, so the retry executes cleanly. Different key would have risked a second transfer - key discipline is the whole ballgame.

Common mistakes

Common mistake

Generating a fresh key per RETRY instead of per OPERATION.

Fix

Key lives across the operation's retries - regenerate only for genuinely new operations. Per-retry keys are theater.

Common mistake

Remembering keys forever, or worse, for seconds only.

Fix

Choose a dedupe window honestly from retry horizons - long enough to cover realistic retries, short enough to bound storage. Document it.

Working with your AI agent

Specify the contract before code: which endpoints accept keys, header name, window length, concurrent-race strategy, and the exact stored-response replay format. Ask your agent to write the race-condition test FIRST - two parallel same-key requests - because concurrency is precisely where hand-rolled dedupe falls apart.

Build it

The module’s real project. Your agent builds; you direct and inspect.

Project: Replay Shield

Armor the Ledger's unsafe endpoints with real idempotency: key-based request dedupe storing original responses, race-proof against parallel submits, TTL'd windows - and a documented client retry policy your own scripts obey.

Requirements

  • Unsafe endpoints (create/update/delete/payment-flavored) accept an idempotency-key header
  • First request executes and stores key plus response; replays return the STORED response with a replay marker
  • Concurrent identical-key requests resolve to exactly one execution - race-tested
  • Dedupe entries expire after a documented window; expired keys re-execute honestly
  • Client retry policy documented: max attempts, backoff curve, Retry-After obedience, total budget
  • Your own test scripts demonstrate compliance with that policy

Context for your agent

Implement idempotency for my Expense Ledger's unsafe endpoints. Contract first: header name, window TTL, storage schema (key, request fingerprint, response, status, expiry), and the concurrency strategy for two same-key requests racing - propose options. Stored responses replay verbatim with an idempotency-replayed marker header. Write the parallel-submission race test BEFORE implementation, watch it fail, then build until green including it. Finally draft the client retry policy and a script that obeys it.

Verify it - twice

Check every item yourself first. Then hand your agent the prompt and compare verdicts.

Self-verified0/11
Functional
Architecture
Security
Testing
Production

Verification prompt - paste into your AI agent

Attack my Replay Shield as a hostile client who profits from double-execution. Attempt each and grade PASS or FAIL against my DoD: (1) fire fifty parallel requests sharing one key - count actual executions; anything above one is blood, (2) reuse ANOTHER USER's key verbatim - did you see their stored response or influence their execution?, (3) wait out the dedupe window and replay - does it behave exactly as documented?, (4) hammer retries WITHOUT backoff - does my retry-budget documentation describe behavior my server actually enforces? Then identify which endpoint still lacks shield coverage and what one request there could double-execute today.

Run this against “Idempotency and Retries” after your own checklist pass. Where the agent and you disagree, investigate before moving on.

Review questions

Answer out loud. If you cannot, revisit before moving on.

  1. Explain why exactly-once is marketed but at-least-once-plus-dedupe is engineered.

  2. Walk through your key lifecycle: generation, storage, replay, expiry.

  3. What does your client owe your server during an outage - and vice versa?

You can now

  • Design unsafe operations to be repeat-safe via keys and stored responses
  • Handle concurrent duplicate submissions race-free
  • Set retry etiquette that survives contact with real outages

Up next

Eventual Consistency