UDP Broadcasting And Easily Finding Network Services

Local area networks (LANs) that use technologies like Ethernet and Wi-Fi are incredibly useful for letting devices talk with each other. Yet a core problem here is knowing which devices are where on the network, as anyone who has ever tried to add a network printer or network share to their system can probably attest to. Unless you happen to know the IP address of the LAN device, the port, and protocol, the target device may as well be located on the Moon without further help, such as automatic network discovery in lieu of waddling over to the device and reading the label listing its IP address.

Over the decades quite a few ways have been developed to enable such network discovery, with many of them using UDP broadcast as the first step. By broadcasting a global message on the entire LAN, any device that has an actively listening UDP socket on that particular port can parse said message and decide whether it’s feeling sociable enough to reply.

The topic of UDP broadcasting is however not as straightforward as it may sound if you’re just getting started, including the existence of many opinions on the ‘right way’. There is also a massive divide between a sprawling service discovery protocol like mDNS and a light-weight one like that one that I had to implement a few years ago for an open source project.

Network Broadcasting

The obvious advantage of a broadcast message is that a client device that seeks its protocol soul mate on the LAN doesn’t need to ping all possible IP address and subnets. Instead,  a broadcast message is designed so that all connected networking devices know that it should be forwarded to all other known devices. Thus with a single message from the client, in theory, only a single message will then neatly land at every single other connected system.

Of course, this ignores happy joy fun things such as convoluted network configurations, such as those involving overlapping Wi-Fi repeaters and subsequent routing, but in general we can assume that this is how it works. Various edge cases and fascinating complications of these will be considered in a later section.

Much of this service auto-discovery is tossed under the header of ‘zero-configuration networking‘, or zeroconf for people who don’t like typing. The best part about zeroconf is probably that there are so many standards here, ranging from DNS-SD to mDNS, UPnP, SLP and others. Perhaps unsurprisingly, one of the major issues here is that platform support here is spotty, with mDNS – despite being one of the most universal – not having much support outside of MacOS/OS X with Bonjour and Linux/BSD with Avahi.

Thus while trying to add the auto-discovery of NymphCast receivers and media servers by NymphCast clients, I found myself asking the daunting question of whether I was at risk of being about to embark on reinventing the proverbial wheel. After all, nobody wants to become the subject of an xkcd comic.

UDP Discovery Basics

As it turns out, I ought not to have been too worried, as despite looking everywhere I could find nothing along the lines of the NyanSD network service discovery (NSD) protocol that I ended up implementing and integrating into NymphCast. What I wanted after all was the most no-frills NSD possible that could be easily integrated, while working the same across just about any desktop, server and embedded platform imaginable.

All that’s needed for this is a way to create an appropriate UDP socket, and a way to either broadcast a query and receive the response, or to listen for incoming UDP packets. Here you can figure out the platform-native method for each target platform, or not reinvent the wheel and use an existing networking library for C++ like Poco. This is what I used for NyanSD, along with my ByteBauble utility to handle endianness conversions.

For the UDP server — the listening side — the procedure is fairly standard, with a regular UDP listening socket. As UDP is a connectionless protocol, there is not a lot of preamble here, just a UDP socket instance (here Poco::Net::DatagramSocket), which is bound to the target port and regularly polls for any fresh UDP packets to process. This can all be seen in the single source file for NyanSD which covers both the client and server side code.

Where things get spicy is with the client that sends the broadcast query and waits for any replies. If we were to just shove the query data into the socket along with the request to toss it over to a regular IP address, not a lot would happen. To make it into a broadcast request we need a few things:

1. Let the network subsystem know that we want to do broadcast things.
2. Create the special broadcast address for the target network interface.

With Poco the first point is easily handled by simply calling setBroadcast(true) on the UDP socket instance. For BSD sockets this sets the appropriate flag on the socket, which is essentially repeated across all OS implementations due to how prevalent the BSD socket library is.

The second point can be summarized for IPv4 as a curt ‘make it end with .255’. For example 192.168.0.255 when the client network interface’s IP address is 192.168.0.42. If there are multiple interfaces on the client system, you can go through the list one by one to broadcast on each of them before filtering out potential duplicate returns.

As for how to do broadcasting with IPv6: you don’t, as this protocol relies on multicast and special multicast receiver groups, which is another kettle of fish and of not much relevance for LANs.

Complications

If you look at the NyanSD API, it may give the impression that the query process is incredibly straightforward, with the sendQuery() function neatly returning a stack of remote systems that responded to our query. While these are definitely all the responses, it’s important to remember that NyanSD queries every single network interface. This means that the responses are likely to contain duplicates, which may even come from the loopback address when a service runs locally.

The filtering of this is captured in the NymphCast client library (libnymphcast) where the findServers() function in the main source file calls the isDuplicate() and isDuplicateName() functions, as well as the removeLoopback() function that nukes any responses that match a remote service found via a non-loopback interface. This last filtering is essential for NymphCast when e.g. using playback groups that would otherwise get confused by a stray loopback address.

Although one may think that such in-depth filtering is unnecessary if all you have is a single Wi-Fi or Ethernet interface in your system, one of the curveballs that I encountered during real-life testing was apparently related to Wi-Fi repeaters. For some reason it seems that the way that the repeaters did their broadcasting led to erroneous duplication of packets and thus multiple returns from a single system.

Depending on your exact use case and network configuration you may encounter any such issues and perhaps an exciting new one.

NyanSD Findings

Over the years that NyanSD has been used in the NymphCast project, it has proven to be one of the most reliable and probably nearly zero-fuss components. I have so far used it on Windows, various Linux distributions, FreeBSD, Haiku, Android, and the ESP32 via FreeRTOS and ESP-IDF. What this experience has proven to me most of all is that service discovery doesn’t have to be complicated.

The basic UDP protocol is simple and reliable enough that, barring a very sick LAN, there shouldn’t be any issues here. Assuming you get your filtering sorted of the responses, it’s probably the last part of a project to worry about.

One thing that I’m also very happy with in NyanSD is that there’s no set port in the protocol, like how mDNS always uses port 5353. What this means is that I can have NyanSD listen with a UDP socket on the same port as the NymphCast server’s TCP socket, which also means that different services with their own port can be targeted directly rather than every NyanSD-enabled service on the network getting blasted by every NyanSD query.

I did also do some work on a NyanSD daemon as a more central services database, but so far I have had no real need for it in a practical deployment. I guess that such a thing could be very useful if the port of a service is not set in stone, but generally that’s the one aspect of network services that tends to be boringly predictable.