Overlay networks & the routing mesh

Cross-node networking and load balancing.

Two containers on the same machine talk over a bridge network without any trouble. Move them onto separate machines and the conversation goes dead quiet. An overlay network is how you get it back. It works like a private phone line strung between two office buildings: the cable runs under public streets, but everyone on the line talks as if they share one room. Docker builds that line with VXLAN (Virtual Extensible LAN, a way of hiding one network inside another). It wraps each container's packets inside ordinary UDP packets (User Datagram Protocol, the plain no-handshake way machines fire data at each other), ships them to the right node, and unwraps them on arrival. A container on node-1 then reaches one on node-2 by name, as if both were plugged into the same switch.

You create an overlay from a manager node. Attach a service to it and Docker stretches that network out to every node running one of the service's tasks. Nodes with nothing scheduled on it stay out, which keeps the mesh small. One overlay can carry many services, and a single service can sit on several overlays at once. That is how you keep a frontend network separate from a backend one. Add --attachable when you want the option of hooking a standalone debug container onto the same network by hand later.

$ docker network create -d overlay --attachable appnet
$ docker service create --name api --network appnet registry.internal/api:1.0
$ docker service create --name web --network appnet -p 80:80 registry.internal/web:1.0

qk8f2p7q1w8v3n5m6b4c2d1e0f
lm3n4o5p6q7r8s9t0u1v2w3x4y
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged
z1y2x3w4v5u6t7s8r9q0p1o2n3
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged

After that, api and web find each other by name. web calls http://api:8080 and Docker's built-in DNS (Domain Name System, the address book that turns names into IP addresses) answers with a live api task, wherever in the cluster that task happens to be running. You never write down an IP.

The routing mesh

Publish a service port the usual way and something surprising happens. In a big company, any receptionist can transfer your call to the right desk, even a desk in another building, and you never need to know who sits where. Swarm's routing mesh behaves the same way. It opens the published port on every node in the cluster, including nodes with no replica of that service, and a request landing on any of them gets passed along to a healthy task somewhere in the swarm. So you can point a plain external load balancer at all your node IPs and stop tracking which node holds which replica. Kill a replica, scale up, let a crashed task reschedule somewhere else, and the mesh keeps routing.

Here is the mesh doing that. web.1 lives on node-1, yet a request sent to node-3 still comes back served.

$ docker service ps web --format 'table {{.Name}}\t{{.Node}}\t{{.CurrentState}}'
$ curl -s http://node-3:80

NAME    NODE     CURRENT STATE
web.1   node-1   Running 6 minutes ago
 
<h1>web app 1.0</h1>
handled by web.1 (node-1)

When you need the real client IP

That convenience has a price. To forward your request, the ingress network rewrites the source address, a trick called NAT (network address translation, the same thing your home router does to every device in the house), so your app sees an internal swarm address instead of the real caller. Most days that costs you nothing. It hurts the moment you rate-limit by IP, or your access logs are meant to show who actually called. Publish in host mode instead. The port binds straight onto the node running the task, and the client IP arrives untouched. You give up the any-node convenience, so put a load balancer in front that aims only at the nodes actually running those tasks.

$ curl -s http://node-1:80/ip
$ docker service update --publish-rm 80 \
    --publish-add mode=host,published=80,target=80 web
$ curl -s http://node-1:80/ip

client ip: 10.0.0.2
web
overall progress: 1 out of 1 tasks
verify: Service converged
client ip: 203.0.113.47

One name and one VIP, or every task IP

A company switchboard number stays the same for years while the staff behind it come and go. Inside the swarm, every service gets that kind of number: a stable VIP (virtual IP, one address standing in for a whole group of real ones). Resolve the service name and DNS hands back that single VIP, and the swarm spreads your connections across the healthy tasks sitting behind it. Callers use one name and never chase task addresses. The VIP holds steady while tasks die and get rescheduled, so clients never feel the churn. Some clients do want the raw task addresses, like certain stateful protocols or a balancer you wrote yourself. Set the endpoint mode to dnsrr (DNS round-robin, where the name answers with every address in turn) and it resolves to each task address instead of a VIP. On a VIP service you can still list the members by looking up the special tasks.<name> record.

$ docker service create --name api --endpoint-mode vip --network appnet api:1.0
$ docker service create --name cache --endpoint-mode dnsrr --network appnet redis:7
 
$ docker exec -it debug nslookup api
$ docker exec -it debug nslookup tasks.api
$ docker exec -it debug nslookup cache

Name:      api
Address 1: 10.0.1.5 api
 
Name:      tasks.api
Address 1: 10.0.1.6
Address 2: 10.0.1.7
Address 3: 10.0.1.8
 
Name:      cache
Address 1: 10.0.1.20
Address 2: 10.0.1.21

Overlay and routing mesh together

Overlay networks let tasks on different nodes talk over one virtual network. The routing mesh publishes a service port on every node and balances traffic to healthy tasks, so a node with no replica on it still accepts the connection and forwards it. Anyone expecting a published port to bind only where a task runs gets caught out by that.

The knobs that come up in production are encrypted overlays, packet size trouble on cloud VPCs (virtual private clouds, your own fenced-off network at the provider), and mesh publishing versus host mode. VXLAN wraps every packet in extra headers, so an overlay riding a link with a tight MTU (maximum transmission unit, the largest packet a link will carry) can pass small requests happily and stall on big ones. Mesh publishing buys operational ease. Host mode buys a traffic path you can point at on a diagram. For services that only talk to each other inside the cluster, skip publishing altogether and let them find each other by overlay DNS name.

When something breaks, work outward from the concrete addresses: the task IPs, the service VIP, then docker network inspect on the overlay itself. If the mesh has opened a port wider than you meant it to, narrow the publish mode and tighten the host firewall.

Overlay changes are easy to make and easy to forget, because nothing on the node's own interfaces advertises what you did. Before you flip a service between ingress and host publishing, write down the current publish mode, the endpoint mode, the node you ran the command on, and the exact command that puts it back. Save a healthy nslookup of both the service name and tasks.<name> from a debug container while things still work. That pair of answers is what tells you later whether DNS broke or the VXLAN data plane did, and guessing between those two at 2am is how short outages turn into long ones.

Try this

Run these on a lab engine (Docker 24+ is fine). Read the sample output first, so you know what a healthy result looks like before you lean on these commands in production.

$ docker network create --driver overlay --attachable front
$ docker service create --name web --network front --replicas 2 -p 8080:80 nginx:1.27-alpine
$ docker service ls
NAME   REPLICAS   IMAGE
web    2/2        nginx:1.27-alpine
# hit :8080 on any node — routing mesh forwards to a task
# STATUS: READY — overlay attached; mesh publishing 8080

Takeaway

Publishing a port in swarm opens it on every node in the cluster, so make that choice on purpose rather than out of habit. Encrypt the overlay when traffic between nodes crosses ground you do not control, and for anything private, leave the port unpublished and let services reach each other by name.

Related