Network debugging tools
tcpdump, ss, mtr, nmap, iperf.
A network that feels slow, or that fails once in every twenty tries, is one of the hardest things to debug. The evidence isn't on your screen. It's out on the wire, in the gap between two machines, and by the time your application prints 'connection timed out' the useful details are already gone. Think of a package that never showed up. Did it ever leave your building? Did it get lost somewhere on the highway? Or did it arrive at a door that was locked? You can't fix the delivery until you know which of those happened. Five tools answer those questions, and each looks at a different part of the trip. ss checks who's connected on this machine. tcpdump watches the actual packets leave and come back. mtr walks the road one hop at a time. nmap knocks on the far end's doors. iperf measures how fast the road really is. The skill is reaching for the one that matches your question instead of guessing.
ss: take inventory of the sockets
Before you touch the wire, look at what your own machine already knows. ss (short for socket statistics, the modern replacement for the older netstat command) is the guest list at the door. It prints every socket, which is the operating system's record of one end of a network conversation: 'this program is talking to that address on that port.' The flags you'll reach for most are -t for TCP (Transmission Control Protocol, the reliable connection type most services use), -u for UDP (User Datagram Protocol, the fire-and-forget type), -l for listening sockets, -n to print raw numbers instead of resolving names (much faster), and -p to show which process owns each socket. That last one needs root.
Read that list the way a bouncer reads a guest list: every open door is something an attacker can try. sshd on port 22 (the SSH service, short for Secure Shell, which is how you log into the box remotely) and nginx on port 80 (a web server) belong there. The line that should stop you is redis-server listening on 0.0.0.0:6379. The address 0.0.0.0 means 'every network interface, reachable from anywhere,' and Redis with no password on a public interface is one of the most-scanned, most-exploited misconfigurations on the internet. This one command is how you catch a service that was supposed to bind to localhost and didn't. Running ss -tulpn on a box you own and being able to explain every line is a real hardening check. One more detail worth knowing on a LISTEN socket: Recv-Q is the number of finished connections waiting for the application to pick them up, and Send-Q is the size of that waiting room, the accept backlog. A Recv-Q sitting right at the Send-Q limit means the application isn't calling accept() fast enough, so completed connections pile up and the kernel (the core of the operating system that manages hardware and processes) starts dropping new ones. That is what an overloaded server looks like from the inside. A SYN flood is a different animal. An attacker sends the opening packet of the TCP handshake (the SYN, short for synchronize) over and over and never finishes it, so those half-open connections never complete and never reach this accept queue. They pile up in a separate half-open queue, which you'd see as a heap of sockets in the SYN-RECV state, not as a full Recv-Q here.
Two states are worth memorizing because each points straight at a cause. SYN-SENT means your machine fired off the first packet of the handshake (the SYN) and heard nothing back. The far end is unreachable, or a firewall is dropping you silently. A pile of sockets stuck in SYN-SENT is your side failing to reach the peer, not the peer rejecting you, because a rejection comes back fast as a reset. CLOSE-WAIT is the opposite tell. It means the other end hung up, but your own application never called close() to release the socket. A few are normal. Hundreds that keep climbing are a file-descriptor leak (a file descriptor is the small numbered handle the kernel gives a program for each open socket or file), and when the process runs out of them it stops accepting new connections and falls over. A count in the hundreds like this one is a bug in your app, not a problem with the network.
tcpdump: watch the actual packets
When ss says a socket is stuck but can't tell you why, go to the wire. tcpdump is a wiretap on a network interface: it copies packets as they pass and prints them, filtered so you don't drown in traffic. It answers the most basic questions in a network fight, the ones every other tool is only guessing at. Is my request actually leaving this box? Is anything coming back? Is the reply a real answer or a reset (a RST packet, the network's way of saying 'go away, nothing is here')? The filter language is where the power is. The command below reads as 'on interface eth0, don't resolve names or ports (-nn, which keeps it fast and honest), and show me only traffic to or from host 10.0.2.1 on port 443.'
Three packets, and the story is already clear. Each line is a SYN (Flags [S]) leaving your host, and nothing ever comes back. Look at two details. The sequence number is identical every time, so this is the same SYN being retransmitted, not three fresh attempts. And the gaps grow: one second, then two. That doubling is TCP's exponential backoff, the kernel waiting longer and longer for a reply that never arrives. That pattern, a SYN going out with no SYN-ACK (the acknowledging reply that would complete the handshake) coming back, means a firewall is dropping your packets or nothing is listening on the far side. If the far end were actively refusing you, a RST would come back instantly instead. When you need to keep the evidence, add -w capture.pcap to write a capture file, then read it later with tcpdump -r capture.pcap, or open it in Wireshark for a full protocol decode.
mtr: find where on the path it dies
tcpdump proves a packet left and got no answer. It can't tell you where on the road it died. mtr (short for My Traceroute) can. It folds two old tools, ping and traceroute, into one live table. It sends packets with a deliberately short lifespan (the TTL, or Time To Live, a counter that every router along the way decrements by one) so that each router in turn is forced to send back an ICMP (Internet Control Message Protocol, the small control-and-error messages that routers exchange) 'time exceeded' reply, which reveals its address. Then it repeats that over and over and tracks how many replies it lost at each hop. The result is a map of the path with a loss percentage at every step. The command below runs ten cycles in report mode (-r), in wide format (-w) so long hostnames don't get cut off.
This is the report people misread most, so read it slowly. Hop 4 shows 100% loss, which looks alarming. It isn't. If hop 4 were really dropping everything, hop 5 and the destination couldn't answer at all, and they answer fine. What's happening is that the router at hop 4 deprioritizes the 'time exceeded' replies mtr measures with, while still forwarding real traffic normally. That is ordinary ICMP rate limiting, not a fault. Loss that shows up at one middle hop and clears at the next is a measurement artifact. The loss that matters is the kind that starts at a hop and continues all the way to the destination. Here hop 5 (the destination, 10.0.2.1) shows a real 40% loss, with latency jumping to about 80 milliseconds. That is where your connection is actually dying, and now you have something concrete to hand the network team instead of 'the internet is broken.'
nmap: which doors are actually open
mtr found where packets die along the path. nmap answers a different question: on the machine at the far end, which doors are actually open? nmap (short for Network Mapper) knocks on ports and reports back what it finds. Attackers run it first, before anything else, to map what you've exposed. You should run it against your own hosts for the same reason, from outside the box, so you see what the world sees. The scan below skips the initial ping check (-Pn, handy when a host is set not to answer pings), uses a SYN scan (-sS, which needs root and is fast because it never finishes the handshake), asks for service and version detection (-sV), and checks all 65535 ports (-p-).
nmap's three port states are the whole point, and they line up exactly with what tcpdump showed earlier. 'open' means something is listening and answered. 'closed' means the port is reachable but nothing is listening, so the host sent back a reset. 'filtered' means nmap got no answer at all, which is the fingerprint of a firewall silently dropping the packet, the same silent drop you watched happen packet by packet with tcpdump. Notice that port 5432, PostgreSQL, is filtered here: a firewall is keeping the database off the outside, which is what you want. The defensive use is direct. Scan your own host from an untrusted network and compare the open ports against what you meant to expose. Any surprise 'open' line is either a misconfiguration or something that shouldn't be there. The version strings are a second gift, and a double-edged one: they tell you, and any attacker, exactly which software versions to check against lists of known vulnerabilities.
iperf: is it the network or the app
The last question is the one everyone argues about: is the network slow, or is the application slow? iperf3 settles it by measuring the raw speed of the link with nothing else in the way. Think of it as renting a moving truck and driving it flat-out down the road to find out how fast the road really allows, instead of guessing from how long your last grocery delivery took. You run a server on one host and a client on the other, and it pushes data as hard as it can for a fixed time. Start iperf3 -s on the receiving host, then point the client at it with a ten-second test.
Two things to read. The Bitrate column sits at 9.42 gigabits per second, which is essentially line rate for a 10-gigabit link, so the path between these two hosts is healthy. The Retr column (short for retransmits) is zero, meaning TCP never had to resend a packet. A climbing Retr count is the signature of loss or congestion on the path, and it lines up with what mtr would show you. Now the payoff for operators: if iperf reports full speed but your application still crawls, the network is not your problem and you can stop digging there. The bug is above the network, in the app, the database, or a slow dependency it waits on. For real-time traffic like voice or video (VoIP, short for Voice over Internet Protocol), add -u to test with UDP instead, which reports jitter and packet loss rather than throughput, the numbers that actually decide whether a call sounds fine or breaks up.
Try this
Work through “iperf: is it the network or the app” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: a packet capture is a bag of other people's secrets. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.