ZeroMQ

Binary Logging Protocol

Now that we have the logging framework working properly, let’s look at the protocol itself. Sending strings around the network is simple, but when it comes to WiFi we really cannot afford to waste bandwidth. We have the tools to work with efficient binary protocols, so let’s design one for logging.

This is going to be a PUB-SUB protocol, and in ØMQ v3.x we do publisher-side filtering. This means we can do multilevel logging (errors, warnings, information), if we put the logging level at the start of the message. So, our message starts with a protocol signature (two bytes), a logging level (one byte), and an event type (one byte).

In the first version we send UUID strings to identify each node. As text, these are 32 characters each. We can send binary UUIDs, but it’s still verbose and wasteful. In the log files we don’t care about the node identifiers. All we need is some way to correlate events. So what’s the shortest identifier we can use that’s going to be unique enough for logging? I say “unique enough” because while we really want zero chance of duplicate UUIDs in the live code, log files are not so critical.

The simplest plausible answer is to hash the IP address and port into a 2-byte value. We’ll get some collisions, but they’ll be rare. How rare? As a quick sanity check I wrote a small program that generates a bunch of addresses and hashes them into 16-bit values, looking for collisions. To be sure, I generated 10,000 addresses across ...