"stateless" consistency: Constantly check if client is still connected?

Question

So the use case is this thin line between being stateful and not being stateful -- payment apis. So when I worked for a payment gateway, our connection to the processor was over TCP, so it was easy to verify that the client or server got the entire message, but when you have to provide a REST API which is supposed to be stateless, it's harder to know. A lot of scenarios can lead to duplicate transactions such as:

1. Mobile app sends a payment request
2. Server processes the message
3. Mobile app loses its connection
4. Server returns a response but the client never gets it so it doesn't know if it was successful or not.
5. Mobile app sends the same payment request again

On one hand, we could place a cache in between that basically locks the same transaction from being performed again (client has to provide a unique operation/transaction id that we use), but I feel like that comes with other complexities like invalidation. I wonder if at least this scenario could be covered using wire protocol in .net?

So I thought to try something like this:

public async Task<IActionResult> Do(CancellationToken abort)
{
    // simulate processing
    await Task.Delay(5_000);

    // see if client is still connected
    if (abort.IsCancellationRequested)
    {
        // if its not, clean up or rollback etc.
        Console.WriteLine("do a rollback");
    }

    return Ok();
}

The problem with this is that, not only can the client still lose connection while writing the response, but even the check itself could be wrong. For example, if the client loses connection, then it never sent the disconnect command, we'd still think they were connected until the server keep-alive fails and it times out and by the time it does, the client may have already started a retry.

I'm wondering if there is a way to have my service rapidly send keep-alives (ex. .5-1 second intervals) so that we can fail early and roll back. And the followup question is: is there anyway to check after return Ok() that the client received the full response? Maybe with middleware that can dig out an id and throw if (to trigger a rollback) the response wasn't fully read?

Answer 1

when I worked for a payment gateway, our connection to the processor was over TCP, so it was easy to verify that the client or server got the entire message

Very similar problems exist at the TCP level. You can have the client send an ack message, but what happens if connectivity is lost immediately after the server receives it? The server would not be able to send a TCP ACK, so as far as the client knows, the server never got the ack message and it should re-send the transaction. The window may be smaller, but this problem never goes away completely; that's the nature of distributed computing.

On one hand, we could place a cache in between that basically locks the same transaction from being performed again (client has to provide a unique operation/transaction id that we use), but I feel like that comes with other complexities like invalidation.

The standard solution is to make requests idempotent if possible. This can be done with a cache; usually some long lifetime like 7 or 30 days is easy to implement and leaves very little room for missed transactions. My favorite implementation for this kind of "de-duplication" cache is CosmosDb because it's highly reliable, fast, and supports expiration. Bonus if you add a timestamp to the transaction and have the client refuse to send ones (or the server refuse to accept ones) that are too old.

Then your requests are idempotent, and the client can call them all day if it wants.

I wonder if at least this scenario could be covered using wire protocol in .net?

Not without a lot of difficulty to achieve something of very questionable benefit.

if the client loses connection, then it never sent the disconnect command, we'd still think they were connected until the server keep-alive fails and it times out and by the time it does, the client may have already started a retry.

Yup.

I'm wondering if there is a way to have my service rapidly send keep-alives (ex. .5-1 second intervals) so that we can fail early and roll back.

Probably not. It might be possible to get at the underlying socket (though I doubt it), and then you could do some hacky stuff to turn on per-socket TCP/IP keepalives. But even if this were possible, it wouldn't get you much. TCP/IP keepalives can be dropped by any intervening network (they're an optional part of the TCP/IP specification). And even if that were possible and it actually worked, then you'd just have a smaller window for the exact same problem to occur - the problem hasn't actually been solved .

And the followup question is: is there anyway to check after return Ok() that the client received the full response? Maybe with middleware that can dig out an id and throw if (to trigger a rollback) the response wasn't fully read?

Nope. The HTTP protocol gives you one response per request; that's it. Technically, the socket would either know that the client received the entire response or that the client may or may not have received the entire response. That information isn't exposed to your app by any framework I'm aware of, mainly because it's not really useful information.

TL;DR: Design for idempotency. Use the unique id you are given, and count yourself blessed that you even have that. Not all systems do.