ZeroMQ

Idempotent Services

Idempotency is not something you take a pill for. What it means is that it’s safe to repeat an operation. Checking the clock is idempotent. Lending one’s credit card to one’s children is not. While many client-to-server use cases are idempotent, some are not. Examples of idempotent use cases include:

• Stateless task distribution—i.e., a pipeline where the servers are stateless workers that compute a reply based purely on the state provided by a request. In such a case, it’s safe (though inefficient) to execute the same request many times.

• A name service that translates logical addresses into endpoints to bind or connect to. In such a case it’s safe to make the same lookup request many times.

And here are examples of non-idempotent use cases:

• A logging service. One does not want the same log information recorded more than once.

• Any service that has an impact on downstream nodes (e.g., sends information on to other nodes). If that service gets the same request more than once, downstream nodes will get duplicate information.

• Any service that modifies shared data in some non-idempotent way; e.g., a service that debits a bank account is definitely not idempotent.

When our server applications are not idempotent, we have to think more carefully about when exactly they might crash. If an application dies when it’s idle, or while it’s processing a request, that’s usually fine. We can use database transactions to make sure a debit and a credit are always done together, if at all. If the server dies while sending its reply, however, that’s a problem, because as far as it’s concerned, it has done its work.

If the network dies just as the reply is making its way back to the client, the same problem arises. The client will think the server died and will resend the request, and the server will do the same work twice, which is not what we want.

To handle non-idempotent operations, we use the fairly standard solution of detecting and rejecting duplicate requests. This means:

• The client must stamp every request with a unique client identifier and a unique message number.

• The server, before sending back a reply, stores it using the combination of client ID and message number as a key.

• The server, when getting a request from a given client, first checks whether it has a reply for that client ID and message number. If so, it does not process the request, but just resends the reply.