You run three managers and one manager's host loses power at 2am. What happens to the swarm?

Scheduling carries on. Two out of three is still a majority. The two survivors are a majority of three, so they can elect a Leader and keep committing decisions. Running tasks never notice, and new ones still get placed. Replace the dead manager soon, because until you do, one more failure freezes everything.

`docker swarm init` prints a join command with a token in it, something like `SWMTKN-1-2p9x...8kd`. How should you handle that string?

As a shared secret: anyone holding it can add a machine to your cluster, so protect it and rotate it with `docker swarm join-token --rotate worker` if it leaks. The token works like a password for joining the swarm, so rotate it the moment you suspect someone has seen it who should not have.

You have a three-manager swarm and need to patch one manager's host. A colleague says: 'Drain it. That takes it out of the Raft quorum, so we're safe to work on it.' Are they right?

No: draining only stops it running application tasks. It stays a voting member of the quorum, and demote is what actually removes it. Drain and demote are different jobs. A drained manager still votes, so treating it as gone can leave you below a majority the moment a second manager drops.

Courses Docker in depthSwarm mode & architecture

Swarm mode & architecture

Managers, workers, and the raft log.

Intermediate14 min · lesson 16 of 30

Compose is fine right up until your app outgrows one machine. Everything it starts lands on the same host, so when that host reboots at 3am, the whole stack goes down with it. Swarm is Docker's answer to that. It takes a pile of separate Docker hosts and makes them behave like one big machine, driven with the same docker command you already type every day. The closest everyday version is a taxi dispatcher: you don't phone one specific driver, you say 'someone pick this up,' and the dispatcher works out who is free. Same shift here. You stop asking which server should run this container and start saying 'keep three copies of this service alive somewhere in the cluster.' The cluster chooses the hosts and quietly replaces anything that dies.

Swarm mode ships inside the Docker Engine, so there is nothing extra to install. You switch it on and it is there. Next to Kubernetes it is far smaller and much easier to hold in your head, and it reuses ideas you have already met: images, containers, overlay networks (one virtual network stretched across every machine so containers on different hosts can talk as if they shared a cable). The catch is fewer features and fewer knobs. For a handful of machines running services you own, that is usually a fair swap.

manager — start the swarm

$ docker swarm init --advertise-addr 10.0.1.10
Swarm initialized: current node (l3f9x8q1abc) is now a manager.
 
To add a worker to this swarm, run the following command:
 
    docker swarm join --token SWMTKN-1-2p9x...8kd 10.0.1.10:2377
 
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

--advertise-addr is the address every other node will use to reach this manager, so pick one they can genuinely route to. The join token is a shared secret. Anyone holding it can attach a machine to your cluster, so treat it the way you treat a production database password, and rotate it with docker swarm join-token --rotate worker the moment you think it has escaped. Swarm also wants a few ports open between nodes: 2377 for cluster management, 7946 for the gossip traffic (a constant background chatter where nodes tell each other who is still alive), and 4789 for overlay network traffic. Miss one of those and nodes fail to join for reasons the error message will not spell out for you.

worker — join the cluster

docker swarm join --token SWMTKN-1-2p9x...8kd 10.0.1.10:2377
This node joined a swarm as a worker.

manager — who's in the cluster

$ docker node ls
ID              HOSTNAME   STATUS   AVAILABILITY   MANAGER STATUS   ENGINE VERSION
l3f9x8q1abc *   mgr-1      Ready    Active         Leader           27.3.1
7g2kma44wp1     mgr-2      Ready    Active         Reachable        27.3.1
9d4pzc07nk2     mgr-3      Ready    Active         Reachable        27.3.1
r8w1vt63hd9     node-1     Ready    Active                          27.3.1
b6x3nq82fl4     node-2     Ready    Active                          27.3.1

The * marks the node you are typing on. Three rows here are managers: one Leader, two Reachable followers. The two blank cells under MANAGER STATUS are plain workers. docker node ls only works from a manager, because only managers hold the state needed to answer the question. Run it on a worker and Docker tells you to go find a manager. That column doubles as your first health check: a manager showing anything other than Leader or Reachable is a vote you no longer have.

Managers and workers

Every node in a swarm is one of two things. Workers are the muscle. They pull images, run the containers a manager hands them, and report back on how those containers are doing. Managers are the brains. They hold the cluster's desired state, decide which node runs what, and answer the docker commands you point at the cluster.

The awkward part is that managers cannot each keep a private copy of the truth and hope the copies match. They agree with each other using Raft, a consensus algorithm (a set of rules a group of computers follows so they all end up holding the same answer). It works like a committee with one rule: a decision counts only once more than half the members have voted for it. One manager is elected Leader and does the deciding, and the others follow along, ready to take the job if the Leader falls over. Because the rule is 'more than half,' you want an odd number of managers, normally three or five. A fourth manager buys you no extra safety. It only adds one more machine that can break.

How the two roles split the work

Managers (odd number: 3 or 5)

Hold desired state

kept in the replicated Raft log

Elect a Leader

one decides, the rest stand by

Schedule tasks

pick which node runs what

Answer your commands

only managers hold cluster state

Workers (scale these out)

Run tasks

pull images, start containers

Report status

health and state back to managers

Hold no state

safe to add and remove freely

The count starts to matter the second something breaks. With three managers you can lose one and carry on, because two is still a majority of three. With five you can lose two. That surviving majority has a name: the quorum. Fall below it and the swarm goes read-only. Containers already running keep running and keep serving traffic, but nothing new gets scheduled and you cannot change anything until enough managers come back. Quorum is what keeps the cluster able to make decisions. It is not what keeps your app up.

Taking a node down for maintenance

Say node-1 needs a kernel patch (an update to the core of its operating system) and a reboot, and you would rather its containers did not vanish for the ten minutes it is offline. Drain it first. Draining tells the swarm to move that node's tasks onto other nodes and to stop placing new ones there. The node stays a member of the cluster the whole time. It goes quiet, nothing more. Patch it, reboot it, set it back to Active, and work starts flowing onto it again.

manager — drain, patch, restore, swap roles

$ docker node update --availability drain node-1
node-1
 
$ docker node ps node-1
ID            NAME     IMAGE         NODE     DESIRED STATE   CURRENT STATE
u7f2k9d1a3q   web.2    nginx:1.27    node-1   Shutdown        Shutdown 6 seconds ago
 
$ docker node update --availability active node-1
node-1
 
$ docker node promote node-2
Node node-2 promoted to a manager in the swarm.
 
$ docker node demote mgr-3
Manager mgr-3 demoted in the swarm.

That Shutdown line is the whole point. Draining had already asked node-1's copy of web to stop and started a replacement elsewhere, so your replica count never actually dipped. If you are pulling a manager out for good rather than for a quick reboot, demote it to a worker first and then remove it, and promote another worker so the manager count stays odd. Promote and demote change a node's job title and nothing else. They do not rebuild the machine or disturb the containers already running on it.

A drained manager still votes

Setting a manager to drain stops it running application tasks, but it stays a full voting member of the Raft quorum. Plenty of people miss this. They drain one manager, treat it as out of the cluster, take a second one down for maintenance, and now two of three managers are gone and the whole swarm is read-only. If you are doing maintenance on managers, demote first, or count the managers still up and confirm they are more than half of the total. And never let yourself land on exactly two managers: with two, a majority needs both votes, so a single failure freezes every scheduling decision you have.

What bites people in production

A one-manager swarm is a lab, not a highly available cluster. That single machine holds the only copy of the Raft log, so losing it loses the cluster. Keep the manager join token away from workers, too. A worker only ever needs the worker token, and the manager token is a different animal: anyone who reads it can attach a machine to your cluster as a manager, with a full copy of your cluster state and the right to schedule anything anywhere.

Compared with Kubernetes, Swarm is smaller and sits much closer to Compose. That is a real trade: a thinner ecosystem in exchange for far less machinery to keep alive. If your team already lives in Docker and needs multi-node scheduling, secrets and overlay networks without standing up etcd (the separate database Kubernetes keeps its state in) and a kubelet (the agent Kubernetes runs on every node), Swarm still earns its place. Know where your control plane lives and back up /var/lib/docker/swarm on the managers. That directory is the cluster.

Three habits worth building. Drain a node before you touch it, so the swarm moves work off deliberately instead of finding out the hard way. Keep manager clocks in sync, because Raft leader elections and the cluster's own certificates both care what time it is. And never hand an untrusted workload a mount of the Docker management socket: a container that can talk to the daemon on a manager can do everything to the cluster that you can do. For placement, label your nodes and constrain services to those labels instead of hard-coding hostnames into stacks. Hostnames change on you. Labels you can move.

Before you touch a real cluster, save the node list exactly as it looked while everything was healthy: which node was Leader, which were Reachable, how many managers there were, and which host you were typing on. That one record is what tells you an hour later whether the unreachable manager is the one you were working on or a fresh problem. Most of the steps here undo cleanly, so write the reverse into the ticket before you run the forward version: setting availability back to Active undoes a drain, and promoting a node undoes a demote. If a colleague cannot replay your maintenance from the ticket alone, the runbook is not done yet.

Try this

Run this on a throwaway machine (any Docker 24 or newer engine is fine). A swarm of one node is perfectly legal and it is the cheapest way to learn the commands. Read the output carefully so you know what healthy looks like before you go near a cluster other people depend on.

terminal

$ docker swarm init --advertise-addr 192.168.1.10
Swarm initialized: current node (abc123) is now a manager.
$ docker node ls
ID       HOSTNAME   STATUS   AVAILABILITY   MANAGER STATUS
abc123   mgr1       Ready    Active         Leader
# STATUS: READY — single-manager lab swarm; add workers with join-token

Takeaway

Two numbers decide whether your swarm survives a bad night: how many managers you run, and how many of them are up right now. Three or five, never two, and read the MANAGER STATUS column before you take any of them down. Guard the join token like the password it effectively is. Everything else in this course, services, stacks, overlay networks and secrets, is stacked on top of that Raft log, which is why /var/lib/docker/swarm on a manager is the one directory you cannot afford to lose.

Network troubleshootingServices, replicas & scaling

What Docker is, and whyDocker for beginners Namespaces from first principlesAdvanced container security Docker CLI cheat sheetCheat sheet Docker vs PodmanComparison Docker interview questionsInterview prep