>_SecOpsLog_
CoursesDocker in depthNetwork troubleshooting

Network troubleshooting

A method for "it can't connect".

Intermediate10 min · lesson 15 of 30

When one container can't reach another, the error you get back is close to useless. Connection refused. Could not resolve host. Timeout. Three short strings that cover a dozen unrelated causes, which is why this kind of bug eats whole afternoons. The way out isn't a hunch. It's a fixed checklist you walk from the outside in, stopping the moment a check answers no.

Work outside-in, stop at the first no

Tracing a dead phone line, you check whether the handset has power before you blame the exchange. Same order here. Is the container running at all? Is it attached to the network you believe it's on? Does the name turn into an IP (Internet Protocol) address? Is the target port open? Is the app behind that port actually listening? Each of those is one command. The first one that answers no is your bug. You rarely need all five, because anything sitting below a broken layer looks broken too, and chasing those ghosts is how people lose a morning.

Outside-in: stop at the first no
11 Running?
docker ps
22 Right network?
docker inspect .NetworkSettings.Networks
33 Name resolves?
nslookup db
44 Port open?
nc -zv db 5432
55 App listening?
ss -tlnp, bound to 0.0.0.0?

Break it on purpose first

Here's the failure you'll meet most often. You create a network, start a database on it, then start the web container and forget the --network flag. Docker doesn't complain. It quietly drops web onto the default bridge, and now web has no way to find db. Both containers look healthy in docker ps. The startup logs say nothing at all. That silence is the whole problem.

reproduce.sh
docker network create appnet
docker run -d --name db  --network appnet -e POSTGRES_PASSWORD=secret postgres:16
docker run -d --name web nginx:alpine
output
7d3a9f1c2b6e08a4d5c1f0e9b8a7d6c5e4f3a2b1c0d9e8f7a6b5c4d3e2f1a0b9
3f9a1c8e2b70d5a4c6e1f0b9a8d7c6e5f4a3b2c1d0e9f8a7b6c5d4e3f2a1b0c9
a1e4d6c0f582b3c7d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2

Diagnose from inside the container

docker exec puts you inside the container's own view of the network, and that's the view that matters. Start with name resolution. Every user-defined Docker network comes with a small phone book: a DNS (Domain Name System) server parked at 127.0.0.11 that turns container names into addresses. The default bridge ships no phone book at all. So if a name refuses to resolve, you have already cut the search space in half.

web: does the name resolve?
docker exec web nslookup db
output
Server:		192.168.65.7
Address:	192.168.65.7:53
 
** server can't find db: NXDOMAIN

Two tells in four lines. The name doesn't resolve (NXDOMAIN is DNS shorthand for "no such domain"), and the Server line reads 192.168.65.7 rather than 127.0.0.11. The second one is the real evidence. web is asking some outside resolver instead of Docker's embedded one, and that only happens when the container isn't on a user-defined network. Check reachability anyway with nc (netcat), a small tool that tries to open a TCP (Transmission Control Protocol) connection to a port and reports what happened.

web: is the port reachable?
docker exec web nc -zv db 5432
output
nc: bad address 'db'

nc never gets as far as the port. It can't turn db into an address, so it dies at the lookup stage. Both checks lean the same way: this is a name problem, and a name problem is nearly always a wrong-network problem. (You may be tempted to reach for ping here. Don't. Plenty of images ship without it, ICMP (Internet Control Message Protocol, the protocol ping speaks) is often blocked outright, and what you care about is whether a TCP port accepts a connection, not whether a host is polite enough to answer pings.)

Find the cause

Ask both containers which networks they're on. One inspect call covering both, so the two answers sit next to each other and the mismatch jumps out:

which networks are they on?
docker inspect -f '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}' web db
output
bridge 
appnet

There it is. web on bridge, db on appnet. Two separate networks with no path between them, and the default bridge wouldn't resolve container names even if there were. You can ask the same question from the network's side, which reads far better once you have twenty containers and want to know who actually joined:

who is attached to appnet?
docker network inspect appnet -f '{{range .Containers}}{{.Name}} {{end}}'
output
db

Fix it and verify

Attach web to appnet. Nothing needs to be destroyed and rebuilt; docker network connect adds a second network interface to a container that's already running. Then run the exact two commands that failed a minute ago. Proving the fix beats assuming it.

connect, then re-test
docker network connect appnet web
docker exec web nslookup db
docker exec web nc -zv db 5432
output
Server:		127.0.0.11
Address:	127.0.0.11:53
 
Name:	db
Address: 172.19.0.2
 
db (172.19.0.2:5432) open

The Server line now reads 127.0.0.11, the name resolves to 172.19.0.2, and the port comes back open. That last line is what you were after. Rerunning the failed command and watching it pass is the difference between fixing a problem and hoping you did. The durable cure is to stop wiring networks by hand: put both services in one Compose file, where they land on a shared network without you having to remember a flag.

When the image has no tools

Images built from scratch or distroless carry no shell utilities, so there's no nslookup or nc for docker exec to run inside them. Borrow a toolbox instead. nicolaka/netshoot is a throwaway image stuffed with network tools, and because you attach it to the network you're debugging, it sees exactly what your app sees. That single move pulls apart the three failure modes people keep confusing: the name won't resolve, the port is closed, or the app was never listening.

borrow a toolbox on the network
docker run --rm -it --network appnet nicolaka/netshoot nslookup db
output
Server:		127.0.0.11
Address:	127.0.0.11#53
 
Name:	db
Address: 172.19.0.2
A mapped port with nobody home
The mirror image of a DNS failure catches people just as often. You publish a port with -p 8080:80, docker port agrees the mapping exists, and curl localhost:8080 still says connection refused. Look at what the app inside is bound to: docker exec web ss -tlnp. If the answer is 127.0.0.1:80, the app is talking only to itself. Treat 127.0.0.1 as an intercom that reaches inside the container and nowhere else, while 0.0.0.0 (all network interfaces) is the front door people can actually knock on. No port mapping can rescue a process bound to loopback. The fix lives in the app's own config: tell it to listen on 0.0.0.0.

Why the order matters on a bad day

Under pressure, connection refused, NXDOMAIN and timeout all feel like the same emergency. They aren't. Running, attached, resolving, port open, app listening: that order, every time, and stop at the first no. Reaching for a packet capture before those five checks is how a ten-minute problem turns into a two-hour one.

Keep a toolbox container in your notes so you can attach one to the patient's network in seconds. Compare what docker inspect says about attachments against what your Compose file intended, because the two drift. For traffic arriving from outside, look at host firewall rules and at which address the published port is bound to. For multi-host setups, the usual suspects are the overlay network's MTU (Maximum Transmission Unit, the largest packet a link will carry) and a service that never joined the network. Restarting both sides feels productive and often buries the real fault, such as an overlapping subnet that will be back next week.

Write the five questions into the runbook with the literal commands beside them. Someone on their first on-call shift should be able to paste the output into the ticket and have everyone agree on which layer failed.

When you run this against a live incident, record the host you ran the commands from, the network name, the container addresses you saw, and the one change you made. docker network connect is easy to undo with docker network disconnect, but only if the ticket says which container you attached to which network. A three-line paper trail in the ticket beats anyone's memory of a chat thread at 3am.

Try this

Run these against a lab engine (Docker 24 or newer is fine). Read the sample output first, so you know what a healthy answer looks like before you lean on these commands during an incident.

terminal
$ docker ps --filter name=web --format '{{.Names}} {{.Status}}'
web Up 3 minutes
$ docker inspect web --format '{{json .NetworkSettings.Networks}}'
{"appnet":{"IPAddress":"172.18.0.3",…}}
$ docker run --rm --network appnet alpine:3.20 nslookup web
Name:      web
Address 1: 172.18.0.3 web.appnet
$ docker run --rm --network appnet alpine:3.20 wget -qO- http://web/
<!DOCTYPE html>…
# STATUS: PASS — running, attached, DNS ok, HTTP 200

Takeaway

Five checks, outside in, and you stop at the first no. If one line stays with you from this lesson, make it the Server address: 127.0.0.11 means Docker's embedded resolver answered, and anything else means the container isn't on the network you thought it was.

Quick check
01docker exec web nslookup db comes back NXDOMAIN, and the Server line reads 192.168.65.7 instead of 127.0.0.11. What's the likeliest explanation?
Correct — 127.0.0.11 only answers on user-defined networks. A Server address from somewhere else tells you web isn't on one, so container-name lookup can't work. Attach web to appnet and run the check again.
Incorrect — No. Read the Server line. If web were on a user-defined network it would say 127.0.0.11 whatever state db is in, so an outside resolver address points at web's attachment, not db's health. (A stopped db would drop out of embedded DNS too, so its state can't produce this exact signature.)
Incorrect — No. Publishing with -p is about reaching a container from the host. Two containers on a shared network talk to each other without any published port at all.
Incorrect — No. A bind-address fault only surfaces once the name resolves and you get as far as the port. Here the lookup fails first, so you never reach Postgres to find out how it's bound.
02The checklist runs five checks from the outside in (running, right network, name resolves, port open, app listening) and tells you to stop at the first no. Why stop there rather than finish all five?
Incorrect — The reason is the order of diagnosis, not permissions.
Correct — A lower layer can't work while a higher one is down, so the first no points at the cause and the remaining checks would only send you off course.
Incorrect — No such cap exists. Stopping early is about not being misled by knock-on failures.
Incorrect — The list is outside-in on purpose. The point is to stop early, not to invert it.
03You publish a port with -p 8080:80, docker port web shows 0.0.0.0:8080->80/tcp, and curl localhost:8080 still returns connection refused. docker exec web ss -tlnp reports the app listening on 127.0.0.1:80. What's broken, and what fixes it?
Correct — 127.0.0.1 inside the container reaches nothing beyond that container, so the app has to bind 0.0.0.0 (all interfaces) before a -p mapping has anywhere to deliver traffic.
Incorrect — docker port shows the mapping was created, so nothing is fighting over 8080. The bind address inside the container is the fault.
Incorrect — This is host to container over a published port, and the mapping is present, so attachment isn't the issue. The 127.0.0.1 bind is.
Incorrect — Nothing here involves name resolution. ss already showed the app bound to loopback, which is the whole story.
Container linking (legacy) vs…Swarm mode & architecture

Related

What Docker is, and whyDocker for beginnersNamespaces from first principlesAdvanced container securityDocker CLI cheat sheetCheat sheetDocker vs PodmanComparisonDocker interview questionsInterview prep