Safe repeated requests (Idempotency)

The problem: does “I got no reply” mean “it was not done”?

A customer taps the pay button in the shop app. The app sends the request to the payment service. The payment service takes the money from the bank. But the reply gets lost on the way back. For example, the customer’s mobile internet drops for a moment.

The app does not know what happened. It only knows it got no reply. Three cases are possible:

1. The request never reached the server.
2. The request arrived, but the server crashed in the middle of the work.
3. The work was fully done, but the reply was lost.

From the app’s side, these three cases look exactly the same. So the app tries again (Retry). If the server cannot tell that this is the same request as before, the money is taken twice.

What Idempotency means

An operation is Idempotent if doing it two times, three times or a hundred times gives the same result as doing it once.

Why does this matter for a Senior developer? Because in a distributed system, repeats are normal, not an exception:

1. The user clicks twice. Or reloads the page.
2. The client or Polly tries again. After every timeout error.
3. Kafka and RabbitMQ deliver the message again. Their delivery is “at least once”.
4. The Outbox pattern sends the message again. If it crashes after sending and before marking the message as sent.

So the question is not “will repeats happen?”. The question is “what happens when they do?”.

HTTP methods

The HTTP standard says which methods must be Idempotent:

This is only an agreement. If your code creates a new row for a PUT, it is not Idempotent. The main problem is usually POST. To make POST safe, we need an idempotency key.

Idempotency Key

The idea is simple:

1. The client creates a unique ID. For example a Guid, when the user opens the payment page. One ID for one user “intent”.
2. It sends the same ID in every try. Usually in a header named Idempotency-Key.
3. The server checks the key before doing the work. If the key is new, it does the work and saves the reply next to the key. If it has seen the key before, it returns the saved reply.

Live example

Run it once without a key and once with a key. Watch the counter for money taken from the account:

The main trap: two requests at the same moment

The simple idea that comes to mind is this: “First check if the key exists. If not, do the work. At the end, save the key and the reply.” This method has a Race Condition:

1. The first request checks: the key is not there.
2. Two milliseconds later, the second request checks: the key is still not there. The first request has not saved anything yet.
3. Both of them make the payment.

The right way: first save the key, then do the work. The keys table has a Unique Constraint on the customer ID and the key. The database makes sure only one request can create that row. This is true even if the two requests are on two different Pods.

Code

The keys table

public enum KeyStatus { InProgress, Completed }

public sealed class IdempotencyRecord
{
    public required Guid CustomerId { get; init; }   // key part 1
    public required string Key { get; init; }        // key part 2
    public required string RequestHash { get; init; }
    public Guid PaymentId { get; init; } = Guid.NewGuid();
    public KeyStatus Status { get; set; } = KeyStatus.InProgress;
    public int? ResponseStatus { get; set; }
    public string? ResponseBody { get; set; }
    public DateTimeOffset CreatedAt { get; init; } = DateTimeOffset.UtcNow;
}

// In OnModelCreating: one row per customer + key.
modelBuilder.Entity<IdempotencyRecord>().HasKey(r => new { r.CustomerId, r.Key });

The payment endpoint

app.MapPost("/payments", async (
    [FromHeader(Name = "Idempotency-Key")] string? key,
    PayRequest request, ShopDbContext db, IBankGateway bank, CancellationToken ct) =>
{
    if (string.IsNullOrWhiteSpace(key))
        return Results.BadRequest("Idempotency-Key header is required.");

    var hash = Convert.ToHexString(SHA256.HashData(JsonSerializer.SerializeToUtf8Bytes(request)));
    var record = new IdempotencyRecord { CustomerId = request.CustomerId, Key = key, RequestHash = hash };

    // 1. Claim the key first. The database lets only one request win.
    db.IdempotencyRecords.Add(record);
    try
    {
        await db.SaveChangesAsync(ct);
    }
    catch (DbUpdateException ex) when (IsDuplicateKey(ex))
    {
        db.ChangeTracker.Clear();
        var old = await db.IdempotencyRecords.SingleAsync(
            r => r.CustomerId == request.CustomerId && r.Key == key, ct);

        if (old.RequestHash != hash)
            return Results.UnprocessableEntity("Same key was used for a different request.");
        if (old.Status == KeyStatus.InProgress)
            return Results.Conflict("This payment is still in progress.");

        // 2. Same request again: return the saved response.
        return Results.Content(old.ResponseBody, "application/json", statusCode: old.ResponseStatus);
    }

    // 3. Only one request reaches this line. PaymentId is the same in every retry to the bank.
    var receipt = await bank.ChargeAsync(record.PaymentId, request.Amount, ct);

    record.Status = KeyStatus.Completed;
    record.ResponseStatus = StatusCodes.Status201Created;
    record.ResponseBody = JsonSerializer.Serialize(receipt);
    await db.SaveChangesAsync(ct);

    return Results.Created($"/payments/{record.PaymentId}", receipt);
});

// PostgreSQL with Npgsql. Other databases use other error codes.
static bool IsDuplicateKey(DbUpdateException ex) =>
    ex.InnerException is PostgresException { SqlState: PostgresErrorCodes.UniqueViolation };

A few points about this code:

1. A hash of the request body. If the same key comes with a different amount, the client has a bug. We do not do the work, and we return a 422 error.
2. A fixed payment ID. We send the same ID to the bank. Most payment gateways do not create a new payment for a repeated ID. So even between us and the bank, a repeat is safe.
3. Code 409 for “in progress”. The client waits a little and asks again with the same key.

Message consumer

The same idea works for messages. Each message has an ID. The receiver saves the ID together with its work in one transaction. If the message arrives again, saving the ID fails and the work is not repeated. This method is called Inbox, and it is covered fully in the Outbox and Inbox lesson.

A simpler way is to write the work so that it is Idempotent by nature:

// Not idempotent: a duplicate message adds stock twice.
product.Stock += message.Quantity;

// Idempotent: a duplicate message changes nothing.
if (order.Status != OrderStatus.Paid)
    order.MarkPaid(message.PaymentId);

Important rules

1. Design every operation with a side effect for repeats. Payment, creating an order, sending a text message, reducing stock.
2. The client creates the key, once for each intent. Not for each try.
3. First save the key, then do the work. Use a unique key in the database, not a check in memory.
4. Save the reply and return the same one. A repeated request must get the same reply that the first request got.
5. Tie the key to the customer ID. If not, one customer’s key can return another customer’s reply.
6. Do not treat “I do not know” as “no”. After a timeout error, first ask for the status, then try again.
7. Delete keys after some time. For example after 24 hours. This time must be longer than the longest time a client may keep retrying.

Common mistakes

When do you need an idempotency key?