Signals: kill, TERM vs KILL

Ask nicely before you force it.

A signal is a tap on the shoulder. You walk over to someone who is deep in their work, tap them, and depending on how you tap they read it differently: "wrap up when you get a chance," "stop right now," or "freeze where you are." On a Linux machine, the tapping is done by the kernel (the core part of the operating system that talks directly to the hardware). You tell the kernel who to tap and which of these small, numbered messages to deliver. The program on the receiving end usually decides what to do about it. Usually, but not always, and that exception is the whole point of this lesson.

The command you use is called kill, which oversells what it does. Most signals are not fatal. kill really means "send a signal to a process," and you aim it using that program's PID (process ID, the unique number the kernel hands to every running program). The two signals you will reach for almost every day are SIGTERM and SIGKILL. SIG is short for signal, so read those as signal-terminate and signal-kill.

Two Messages You'll Send Constantly

SIGTERM (signal 15) is the closing-time announcement. It is the bartender flicking the lights and saying "we're closing, please finish your drink and head out." The process hears it and gets to shut down on its own terms: finish the request it is handling, flush data sitting in memory out to disk, release any locks it holds, then exit. SIGKILL (signal 9) is cutting the power to the whole building. The process gets no announcement and no chance to react. The kernel removes it on the spot. That difference, a graceful exit versus an instant stop, is why the order you send them in matters so much.

You can watch the difference. Here is a tiny script that catches SIGTERM and does a little cleanup before it leaves.

#!/usr/bin/env bash
cleanup() {
  echo "caught SIGTERM: flushing work, releasing lock"
  rm -f /tmp/worker.lock
  exit 0
}
trap cleanup TERM              # run cleanup() when SIGTERM arrives
 
echo "worker started (PID $$), holding lock"
touch /tmp/worker.lock
while true; do sleep 1; done    # pretend to do real work

Run it in the background, then send the polite signal (the default one) with a plain kill and no number at all.

The process heard the request, ran its cleanup, deleted its lock file, and exited cleanly (the shell reports Done). Now do the same thing but reach for the hammer.

No "caught SIGTERM" line this time, and the lock file is still sitting there. SIGKILL never gave the cleanup code a chance to run, so the process left its mess behind. Multiply that lock file by a real database's half-written files and you can see the shape of the problem. This also has a security edge: a well-behaved program cleans up on SIGTERM, but hostile code can catch that same signal and use the moment to delete its files and cover its tracks. A graceful stop is a request to code you may not trust.

What 'Catching' a Signal Means

Most signals are like a doorbell. The program has wired up a handler (a small piece of its own code that runs when the bell rings) and can answer however it likes: tidy up and leave, ignore the bell completely, or do something else entirely. That freedom is deliberate and useful. It is also why a plain kill is a request, not a command. A buggy program can wedge and never answer the door. A malicious one can catch SIGTERM on purpose, either to refuse to die or to wipe evidence the instant you ask it to stop.

Two signals have no doorbell. SIGKILL (9) and SIGSTOP (19) are handled entirely by the kernel and never reach the program's own code. There is nothing to catch, nothing to ignore, nothing to trap. That is exactly why SIGKILL always wins, and why, in a hurry, it feels so tempting. The catch is that "always wins" also means "no cleanup, ever."

When a process dies from a signal, that fact travels back to whatever started it. Your shell reports it as an exit status of 128 plus the signal number, and container runtimes like Docker do the same. Press Ctrl-C, which sends SIGINT (signal 2, signal-interrupt), and you can read it straight off.

130 is 128 plus 2. A process cut down by SIGKILL (9) exits 137, and one that quits on SIGTERM (15) exits 143. So when you spot status=137 in a log, that is the machine telling you something reached for the big hammer, whether that was you, systemd, or an out-of-memory killer stepping in.

Naming Names Instead of Numbers

Typing PIDs by hand gets old, and it is dangerously easy to fat-finger one and kill the wrong program. You can work by name instead. pgrep finds processes, pkill signals them by name, and killall does the same by exact program name. You can also ask the shell to translate between a signal's name and its number.

pkill -u deploy limits the blast radius to processes owned by the deploy user, and -f matches against the whole command line, not only the program name. Those two flags are the difference between stopping one runaway worker and taking down every node process on the box. SIGHUP (signal 1, signal-hangup) is a leftover from the dial-up era, when it meant "the phone line dropped." Daemons (programs that run quietly in the background) repurposed it to mean "reread your configuration." Send SIGHUP to nginx or sshd and it reloads its config file in place without dropping the connections it is already serving. It is the difference between swapping a recipe card and closing the whole kitchen to reprint the menu.

Let Systemd Do the Escalation

You rarely want to run this ladder by hand for a real service. systemd (the manager that starts, stops, and supervises services on most modern Linux systems) already does the graceful-then-forceful dance for you. When you stop a service it sends SIGTERM, waits, and only if the process is still breathing after a timeout does it send SIGKILL. You can read the exact settings for any unit.

Read that as: ask with SIGTERM, wait a minute and a half, then force with SIGKILL. When that timeout actually fires, the journal (systemd's log) spells it out.

Look at the timestamps: 09:14:59 to 09:16:29 is exactly ninety seconds. The service ignored the polite request, its time ran out, and systemd escalated to SIGKILL on its own. If your app needs longer to drain connections, raise TimeoutStopSec in the unit; if a wedged app is stalling your deploys, lower it. Docker runs the same play with a shorter fuse: docker stop sends SIGTERM and waits ten seconds before SIGKILL.

Freeze First, Kill Later

Here is where a DevSecOps habit earns its keep. You find a process you do not trust, say a mystery python script phoning home from /tmp. Your instinct is to kill it. Hold off. A plain kill sends SIGTERM, which hostile code can catch and use to shred its own files on the way out. Instead, freeze it. SIGSTOP (signal 19) is the uncatchable pause. The process stops mid-instruction and cannot run a single line of its own code to react.

The T in the STAT column means stopped. The process is frozen in place, its memory intact, unable to touch the disk or the network. Now you can capture that memory for analysis, note its open files and network connections, and only then end it. SIGCONT (signal 18) would resume it if you needed to watch it run; SIGKILL ends it for good once you have collected what you need.

Two more things a defender leans on. First, you can only signal a process you own, unless you are root. Try to touch someone else's and the kernel says no.

That one line, Operation not permitted, is a permission boundary doing its job: an ordinary user cannot kill root's audit daemon or another user's session. The flip side is the danger. An attacker who reaches root can silence your logging and monitoring (auditd, the Linux audit daemon, or your EDR (endpoint detection and response) agent) with a single signal, which is why those processes are worth watching for unexpected death. Second, there is a signal that sends nothing: signal 0. It delivers no message but still runs the kernel's permission-and-existence check, so scripts (and quiet attackers) use it to ask "is this PID alive, and may I touch it?" without disturbing anything.

Try this

Work through “Freeze First, Kill Later” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: never SIGKILL a database to save time. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related