Olawale Lawal
Writingtransmissions / 025

How to Propagate Idempotency Keys to Downstream Services

13 min readDistributed SystemsC#system designpayment

To propagate an idempotency key downstream, hash the client’s key together with the caller, the provider and the operation, then send the result as the provider’s key. Every retry produces the same value, so the provider can spot the duplicate, and each downstream call still gets a key of its own.

My payment rails post says idempotency has to survive the whole chain, and its example does that with one line:

var downstreamKey = GenerateDownstreamKey(request.IdempotencyKey);

That line skips the hard part, and its signature is already too small: one argument can’t tell a charge from a refund. This post is what belongs inside that method, what Stripe, Adyen and PayPal each expect from the result, and where a good key still won’t save you.

The examples use IdempotencyGuard, my ASP.NET Core library, but the approach carries over to any stack. For the edge of the problem, including how claims and fencing work, see how I built IdempotencyGuard.

Why Your Own Idempotency Layer Isn’t Enough

An idempotency key at your API’s edge protects your service. The money moves somewhere else. Walk through a lost response:

  1. A client sends POST /payments with a key.
  2. Your handler calls the payment provider, and the provider charges the card.
  3. The response never reaches you, because of a timeout or a crashed process.
  4. Your service has no stored response for the key, so the claim is released or expires.
  5. The client retries, and your handler runs again.
  6. The handler calls the provider a second time.

Your middleware did nothing wrong at any step. It couldn’t know the charge succeeded, because that knowledge was in the response that never arrived. Only the provider can catch this duplicate, and only if both calls carry the same key.

Two Approaches That Don’t Work

Generating a Random Key per Call

// Wrong: every retry sends a new key
var result = await provider.Charge(request.Amount, Guid.NewGuid().ToString());

It looks responsible, and provider docs do recommend UUIDs. But the provider’s duplicate check compares keys, and these never match, so step 6 is a brand new charge. A random key is fine if you generate it once and persist it before the call. Generated inside the handler, it protects nothing.

Forwarding the Client’s Key

// Wrong: the client's key goes straight to the provider
var result = await provider.Charge(request.Amount, request.Headers["Idempotency-Key"]);

This one survives a retry, which is why it’s so common. It breaks in four other ways:

  • One request, several calls. A checkout might authorize a card and then capture the payment. One forwarded key can’t cover both, and PayPal’s docs say an ID has to be unique per API call type.
  • Callers share your provider’s key space. Your provider sees one account: yours. If two of your customers both send order-1, the second payment looks like a duplicate of the first. Depending on the provider and the parameters, that customer gets the first customer’s result or an error.
  • Length limits. The client picks the key’s length, and your provider has a limit. Adyen’s is 64 characters.
  • Sensitive data. Stripe’s docs warn against putting things like email addresses in keys, and you don’t control what your clients put in theirs.

Derive the Key With a Hash

Hash the client’s key together with the provider and the operation. This is IdempotencyGuard’s default generator:

public string Generate(string originalKey, string provider, string operation)
{
    var input = $"{originalKey}:{provider}:{operation}";
    var hash = SHA256.HashData(Encoding.UTF8.GetBytes(input));
    return Convert.ToHexString(hash).ToLowerInvariant();
}

And you call it from a handler:

app.MapPost("/payments", async (
    PaymentRequest request,
    IIdempotencyContext idempotency,
    IPaymentProvider provider) =>
{
    var stripeKey = idempotency.DownstreamKey("stripe", "charge");
    var result = await provider.Charge(request.Amount, stripeKey);

    return Results.Created($"/payments/{result.Id}", result);
});

This is what the published 2.0.0 package produces, shortened to the first 16 characters:

order-123, stripe, charge         : 3bd85b2792ca27c0... (64 chars)
order-123, stripe, charge (retry) : 3bd85b2792ca27c0... (64 chars)
order-123, stripe, refund         : 338b7af324c26306... (64 chars)
order-123, adyen, charge          : 3e4f95c20aa63ffa... (64 chars)
order-124, stripe, charge         : 68c1875b352428ec... (64 chars)

A retry gets the same key, and everything else gets a different one. There’s no state behind it, so the key survives a crash, a takeover by another instance and a deployment. Storing a generated key can’t say the same without a durable write before every call.

Scope the Input by Caller

The input has to identify the caller as well as the request, or the shared key space problem comes back. IdempotencyGuard derives downstream keys from the prefix, the caller’s scope and the client’s key, but scoping is opt-in, so turn it on:

options.KeyScope = context => context.User.FindFirstValue("sub");

Two things to watch:

  • Turning on scoping changes the keys. A retry that spans the change sends the provider a new key, so the provider won’t catch that duplicate. Do it at a quiet time, once in-flight retries have drained.
  • End any prefix with a delimiter. Without one, tenant-1 with the key 2abc and tenant-12 with the key abc hash the same input.

Keep Delimiters Out of the Names

Joining with a delimiter has its own trap. These two calls produce the same key:

k:stripe, charge, x   : a9c4977059b5e210... (64 chars)
k, stripe, charge:x   : a9c4977059b5e210... (64 chars)

The client chooses the key and can put a : in it, so a : in your names could let one operation’s key match another’s. With no : in the provider or operation, the last two segments of the input can only be read one way.

When the Hash Needs a Secret

A plain hash is reproducible by anyone who knows the inputs, and order numbers and sub claims aren’t secrets. Scoping doesn’t change that.

It matters mostly on Adyen, which stores keys per company account and recommends random UUIDs, specifically so that two API credentials on one account can’t read each other’s responses. If you run several credentials on one account and they don’t all belong to people you’d show every response to, swap the generator for an HMAC:

public class HmacDownstreamKeyGenerator(byte[] secret) : IDownstreamKeyGenerator
{
    public string Generate(string originalKey, string provider, string operation)
    {
        var input = $"{originalKey}:{provider}:{operation}";
        var hash = HMACSHA256.HashData(secret, Encoding.UTF8.GetBytes(input));
        return Convert.ToHexString(hash).ToLowerInvariant();
    }
}

// Register before AddIdempotencyGuard, which only adds its default if none exists
builder.Services.AddSingleton<IDownstreamKeyGenerator>(
    new HmacDownstreamKeyGenerator(secretBytes));

Rotating that secret changes every downstream key, so plan it the way you’d plan turning on scoping. A single credential on a single account doesn’t need any of this.

Name Operations by What They Do

An operation name has to stay the same across retries and differ for each side effect. A handler with several calls gets several names:

var authKey = idempotency.DownstreamKey("adyen", "authorize");
var captureKey = idempotency.DownstreamKey("adyen", "capture");

If you call a provider in a loop, build the name from something stable about the item:

foreach (var item in request.Items)
{
    var key = idempotency.DownstreamKey("ledger", $"transfer-{item.LineId}");
    await ledger.Transfer(item, key);
}

Pick that ID with some care. A product ID gives two lines for the same product one key, so the second transfer never happens. And if the ID comes from the request body, check it for : first, for the reason above.

Never include an attempt counter, a timestamp or a generated ID. Each one brings back the random key problem, and it’s easy to miss because the code still looks deterministic.

Send the Same Parameters Every Time

A stable key isn’t enough on its own. Stripe compares a request’s parameters with the original’s, and returns an error if they differ. So this handler fails on retry, even with the right key:

// Fails on retry at Stripe: the metadata differs each time
var charge = new ChargeRequest
{
    Amount = request.Amount,
    Metadata = { ["processed_at"] = DateTime.UtcNow.ToString("O") }
};

Build provider requests only from the client’s request and data that won’t change between runs.

What Each Provider Expects

These details come from each provider’s docs and published API specs as of November 2026, so check them before relying on them:

Stripe (API v1) Adyen PayPal
Header Idempotency-Key idempotency-key PayPal-Request-Id
Max length 255 64 108 on Orders v2
Key lifetime At least 24 hours, then may be removed 7 to 14 days 6 hours on Orders v2
Replay gives Saved first response, including 500s Result of the processed request Latest status of the request

PayPal needs the most care, because it varies by API. Orders v2 stores keys for 6 hours by default, and your account manager can raise that to 72. Its spec allows 108 characters and the Payments v2 spec allows 10,000, while its general idempotency guide recommends UUIDs because they meet a “38 single-byte character limit”. A 64-character hash passes both specs and breaks that guide, so test your keys against the sandbox for each endpoint you call.

The default hash fits Stripe, Adyen and Orders v2. If you hit a shorter limit, swap the generator, registered the same way as the HMAC one above:

public class ProviderAwareKeyGenerator : IDownstreamKeyGenerator
{
    private readonly DefaultDownstreamKeyGenerator _inner = new();

    public string Generate(string originalKey, string provider, string operation)
    {
        var key = _inner.Generate(originalKey, provider, operation);

        // 32 hex characters is still 128 bits, and fits under a 38-character limit
        return provider == "legacy-gateway" ? key[..32] : key;
    }
}

Use the Same Value as a Payment Reference

A derived key also makes a good payment reference. Every retry sends the same value, you don’t need to save it before the call, and after a timeout or a crash you can recompute it from the client’s key and ask the provider what happened.

var reference = idempotency.DownstreamKey("gateway", "payment-reference");

try
{
    var result = await gateway.Initialize(request.Amount, reference);
    return ToResponse(result);
}
catch (DuplicateReferenceException)
{
    // An earlier attempt reached the provider, so find out what happened to it
    var existing = await gateway.GetByReference(reference);
    return ToResponse(existing);
}

The wrapper is yours to write, and the exception stands in for however your provider reports a duplicate.

Some providers also reject a reference they’ve already seen, and that check doesn’t share the idempotency key’s window. Don’t assume yours does. Two that document something:

  • PayPal returns DUPLICATE_INVOICE_ID on Orders v2 when a repeated invoice_id meets an account setting that requires them to be unique. The field takes up to 127 characters.
  • Flutterwave says tx_ref “must be unique for every transaction”, but its docs don’t say what happens when you reuse one.

Stripe’s metadata is handy for reconciling, but I found nothing saying Stripe enforces uniqueness on it. Neither PayPal nor Flutterwave says how long its check lasts, so test it in the sandbox before you rely on it.

A reference behaves differently from an idempotency key in four ways:

  • A duplicate isn’t a replay. You’ll usually get an error where an idempotency key would return the original response, so your handler has to look the payment up, as above.
  • A failed payment may use up the reference. If your provider treats a reference as spent once a payment against it is declined, a genuine new attempt needs a new client key.
  • Collisions become visible. Without caller scoping, two customers sending order-1 share a reference, and the second customer’s payment can be refused.
  • It’s only as stable as the client’s key. A client that sends a new key on every retry gets a new reference every time, so the provider sees no duplicate. If you can’t rely on your clients’ keys, build the reference from the order or checkout ID.

Errors You’ll See

No idempotency context is available for this request

DownstreamKey throws this InvalidOperationException when the guard didn’t run for the request, so there’s no key to derive from. There are two usual causes. One is a request with no key on an endpoint that doesn’t require one: an unmarked endpoint under the default UnmarkedEndpoints, or one marked with Required = false. The other is an endpoint you forgot to mark under UnmarkedEndpoints = Ignore, where the guard skips it even when a key is sent.

Don’t catch the exception and fall back to a random key. Mark the endpoint, which makes the key mandatory:

app.MapPost("/payments", CreatePayment).RequireIdempotency();

Stripe: idempotency_key_in_use

If a claim expires and another instance takes over, two handlers can call the provider at once with the same key. That’s the design working, and this 409 is what it looks like at Stripe. Stripe doesn’t save a result for the conflicting request, so wait and retry with the same key and the same parameters. A new key can start a second charge.

Adyen: 704 “request already processed or in progress”

Adyen’s version comes back as a 409 or a 422, when a duplicate arrives before the first request has finished. It also marks retryable failures with a transient-error: true header, and its docs are firm that you shouldn’t retry when that header is missing or false.

PayPal’s idempotency guide gives no error code for this. It says PayPal processes the first request and “might fail” the second.

None of these three mean the payment failed. The first call may well succeed.

Stripe: the request didn’t match the original

You reused a key with different parameters, so look for a value in the request that changes per run. If you really do need to change the parameters, Stripe’s advice is to first establish that the original didn’t succeed and isn’t still running, and only then use a new key.

Where Downstream Keys Still Fail

The Windows Don’t Line Up

Your store and your provider each forget keys on their own schedule. IdempotencyGuard’s default response TTL is 24 hours, and Stripe may prune keys once they’re 24 hours old. A retry after both have expired is a new charge, with nothing left to stop it.

A TTL longer than your provider’s window is fine on its own, because a stored response replays without calling the provider. The danger is a request that never stored a response, after a crash or a released claim. The provider is its only protection, and with PayPal Orders v2 that protection ends after 6 hours by default.

Raising your TTL won’t help, since it only controls how long stored responses last. Your clients’ retry window has to fit inside the shortest window of any provider the endpoint calls. After that, check with the provider instead of retrying with an old key, and a payment reference gives you a way to do that.

Regions are another gap. Adyen’s docs say keys aren’t checked across regional endpoints, so a fallback to another region can repeat a payment.

A Replayed Error Isn’t a Fresh Attempt

Stripe’s API v1 saves the first response for a key, including a 500, and retries return it. So retrying with the same key won’t always get you past a failure. Decide what your handler does when one comes back: fetch the payment’s state, or surface it. Sending a new key is the one thing to avoid, unless you know the original didn’t charge.

A Timeout Means Unknown

If the provider call times out, you don’t know what happened, and your record of the payment shouldn’t say “failed”. Keep it in a distinct state, and retry with the same key or check with the provider. That’s the payment status unknown state from the payment rails post.

A Key Can’t Vouch for the Answer

A key stops the provider running your operation twice. It says nothing about whether the answer it gave you stays true. On TryGrip, one provider returned “success” for transactions that failed hours later through a batch reversal, and a perfect key would have replayed that “success” faithfully. Reconciliation has to catch that, however good your keys are.

Some Downstreams Don’t Support Keys

Stripe, Adyen and PayPal publish all of this, and plenty of services publish nothing. The protection then has to come from somewhere else:

  • Your own database: put the derived key in a column with a unique constraint, so a second insert fails.
  • Message queues: use the derived key as the message ID, and dedupe in the consumer.
  • Services with a client reference field: send the derived key as the reference, then look it up before retrying.
  • Services with none of these: you can’t make the call safe, so reconcile afterwards, and keep your TTL longer than your slowest call.

How to Test Downstream Keys

Use a fake provider that records every key it receives, and run the scenarios that cause real duplicates:

  • Retry after a lost response: make the fake charge the card and then fail, retry the request, and assert the second call used the same key.
  • Takeover: hold a handler open past its claim TTL, let a second one run, and assert both sent the same key.
  • Two callers, same key: send the same client key as two users, and assert the provider saw two different keys.
  • Two operations: assert each downstream call in one handler got a different key.

A unit test on the generator alone won’t catch a handler that adds a timestamp to the operation name.

Checklist

Question If not
Is the key derived, not random per call? Every retry is a new charge
Does the input include the caller? Two customers’ payments collide
Does each operation get its own key? A second call replays the first
Do retries send identical parameters? The provider rejects the retry
Does the key fit every provider’s limit? Calls fail outright
Is your clients’ retry window inside the provider’s? Late retries charge again
Does a duplicate reference error trigger a lookup? You report a failure for a payment that went through
Does a timeout record “unknown” rather than “failed”? You charge again, or refund a charge you can’t see

Frequently Asked Questions

Should I pass the client’s idempotency key to Stripe?

No. Derive a new one from the client’s key, the caller, the provider and the operation. Each operation then gets its own key, and two customers can’t collide.

Why not store a generated key instead of deriving one?

You can: generate a UUID, save it before the call, and reuse it on retry. It suits you if you already persist payment intent first, and it avoids the reproducible-input problem without a secret. But it needs a durable write before every call, and deriving doesn’t.

Can I use the same key for authorize and capture?

No. They’re separate operations, and each needs its own key. PayPal’s docs say explicitly that the ID must be unique per API call type.

Do I still need an idempotent API if my provider supports keys?

Yes. The provider protects only the charge. Your own database writes, emails and other side effects still run twice, and your client gets no consistent response without an idempotent layer of your own.