Debugging & power tools
gdb, ltrace, core dumps, jq, ripgrep, fzf.
A log line tells you what a program chose to announce. It says nothing about the parts it never mentions: the lock it is blocked on, the pointer that was zero, the library call that failed while nobody was checking. So when a service hangs, crashes, or misbehaves and the logs go quiet, you drop below the application and read the process itself. The tools here do that. Some freeze a running program and read its memory. Some record every call it makes. Some cut through the flood of output afterward. There is a defender's reason to know them cold: the same tools that find your deadlock are the ones an attacker uses to read secrets out of a process, and the settings that stop the attacker are the ones that will block your debugger too.
Reading a live process with gdb
gdb (the GNU Debugger, a program that inspects and controls another running program) is like a mechanic who can freeze a running engine in place, then walk around and study every spinning part exactly where it stopped. It attaches to a target by its PID (process ID, the number the kernel gives every running program) and reads that program's memory: its variables, its call stack, and every thread inside it.
When a service hangs, the question that pays off is simple. What is every thread waiting for? A thread (one independent line of execution inside a process) that is stuck is usually blocked on a lock that another thread is holding and not releasing. gdb can print the backtrace (the chain of function calls that led to the current spot) of every thread at once, so one command shows you the whole traffic jam.
Read the addresses, not the function names alone. Thread 3 (LWP 815, where LWP means light-weight process, the kernel's name for a thread) is asleep in __lll_lock_wait on mutex 0x561f0d2a4f60, called from flush_metrics. Thread 1 is asleep the same way, but on a different mutex, 0x561f0d2a5120, called from collect_metrics. Two threads, two locks, and neither one will wake. Line the two code paths up in metrics.c and the trap shows itself: collect_metrics took 0x561f0d2a4f60 first and is now stuck waiting for 0x561f0d2a5120, while flush_metrics took 0x561f0d2a5120 first and is stuck waiting for 0x561f0d2a4f60. Each thread is sitting on the exact key the other one needs. That is a deadlock caused by a lock-order inversion, and gdb walked you to both source lines without a single log message and without rebuilding anything. This is your first move on any frozen daemon (a program that runs quietly in the background as a service).
Attaching stops the target while you look (the danger note below spells that out). If you want the stacks without holding a busy service still, two lighter moves help. gcore 812 writes a snapshot of the process's memory to a file and lets the process keep running, so you study the copy offline. And eu-stack -p 812 (from the elfutils package) grabs a backtrace with a far shorter pause than a full gdb session, which is gentler on something that is still serving traffic.
Core dumps: the crash caught in amber
A core dump is a flight recorder for a crash. The instant a program dies from a fault, say it reads through a null pointer, the kernel (the core of the operating system that controls the hardware) can write the entire contents of that program's memory to a file, the core dump, before it lets the process go. You open the wreck later and see exactly where and why it went down. No racing to attach in time, no reproducing the bug. The crash is frozen for you.
Core dumps are often switched off, because they can be large and can hold secrets. ulimit -c unlimited turns them on for your shell. On a systemd machine, though, the kernel usually does not write a file at all; it hands the crash to a helper program instead. That policy lives in a kernel setting called core_pattern, and reading it tells you what will happen on the next crash.
The leading pipe symbol is the tell. Instead of a filename, the kernel is told to run a program and stream the core into it. Here that program is systemd-coredump, which stores the dump, records who crashed and why, and makes it retrievable with coredumpctl. The %P %u %g %s codes pass it the crashing PID, user, group, and signal number. (On a stock Ubuntu box you may see apport in that slot instead; install the systemd-coredump package to get the coredumpctl workflow below.)
buf=0x0 is the whole story. Something called parse_header with a null pointer (an address of zero, which points at nothing), and line 214 tried to read a field through it. SIGSEGV (signal 11, a segmentation fault, the kernel killing a process for touching memory it is not allowed to) followed. The backtrace shows the path that got there: a request came in, handle_request ran, parse_header choked. For a defender this reads two ways at once. A repeatable crash on attacker-controlled input like an HTTP header is often the front half of a memory-corruption exploit, and the core dump is your evidence for exactly which input reached which line.
Two levels of eavesdropping: strace and ltrace
A program talks to the outside world through two doors. One door opens onto the kernel: every time the program wants to read a file, open a network socket (an endpoint for sending and receiving data over the network), or ask the operating system for memory, it makes a system call (a request across the boundary into the kernel) through that door. strace, covered in the tracing lesson, sits at that door and logs every crossing. The second, inner door opens onto the program's libraries, the shared code it was built against, like libc (the standard C library that provides the basics: string handling, memory allocation, and so on). ltrace sits at that inner door and logs the calls into those libraries.
That library view is often closer to what the program means than the raw system calls beneath it. You see the malloc, the strlen, the library's own functions by name, which sit closer to intent than the read and write calls they eventually become. The -c flag adds up where the calls and the time went, which turns a vague "it's slow" into an algorithm you can name.
Over a million strlen calls dominate a run that made only nine thousand memcpy calls. That lopsided count is the fingerprint of code that measures the same string over and over, classically a strlen sitting inside a loop over a buffer that keeps growing, so the cost climbs with the square of the size. You found the hot spot without opening the source and without attaching a profiler (a tool that samples a running program to show where it spends its time).
Sifting the output: jq, ripgrep, fd, fzf
Every tool above produces a flood of text or JSON. The last group is about surviving that flood. These are newer command-line programs that do the jobs grep and find do, faster and with better defaults, plus one that adds interactive search. None ship by default, so on a fresh host: apt install jq ripgrep fd-find fzf. Put them on every box you operate.
jq is a query language for JSON (JavaScript Object Notation, the curly-brace text format that logs and APIs speak in), the way awk is a query language for columns of text. Modern logs are JSON, and journalctl -o json hands you the systemd journal as one JSON object per line. jq filters it, reshapes it, and pulls out only the fields you care about.
Priority in the journal runs 0 (emergency) to 7 (debug), so <= 3 keeps errors, critical, alerts, and emergencies and drops the noise. One line filters a full boot down to the four things that actually went wrong: a failed SSH handshake, a filesystem error, a service that died, and the out-of-memory killer shooting a process. That is a triage view no grep over plain text gives you cleanly, because the priority is a real field, not a word buried in a message.
ripgrep (rg) is grep rebuilt for speed. It searches directories on its own with no -r, skips binary files and anything in .gitignore by default, and uses every CPU core. On a big log tree the difference is minutes versus a fraction of a second, and --stats shows you the damage.
fd is find with defaults that match how you actually search. fd -e conf . /etc/nginx walks that tree for files ending in .conf and skips hidden and ignored paths automatically. One naming gotcha to know cold: on Debian and Ubuntu the binary is installed as fdfind, because the name fd was already taken by another package, so people alias it back to fd. fzf is a fuzzy finder: pipe any list into it and you get an interactive, type-to-narrow selector. Bound to Ctrl-R in your shell (the fzf package sets this up), it replaces the blind reverse-history search with one you can see and refine, and history | fzf does the same on demand.
Before you reach for gdb, strace, or ltrace on a shared host, check one setting. It decides whether the kernel will even let you attach.
Try this
Work through “Sifting the output: jq, ripgrep, fd, fzf” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: core dumps hold live secrets. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.