Debugging & power tools

gdb, ltrace, core dumps, jq, ripgrep, fzf.

When a program crashes, hangs, or misbehaves in a way logs do not explain, you go below the application into debugging and tracing tools. The classics remain essential: gdb inspects a live process or a crash, ltrace shows library calls, and core dumps capture the moment of a crash for offline analysis. Alongside them, a set of modern command-line "power tools" has become famous for making everyday data work far faster.

gdb and core dumps

gdb (the GNU debugger) attaches to a running process or loads a core dump — the memory image the kernel writes when a program crashes — and lets you inspect the stack, variables, and threads at the moment of failure. For a hung process, gdb -p <pid> then thread apply all bt shows every thread’s stack, instantly revealing a deadlock or where it is stuck. For a crash, enabling core dumps (they are often disabled by default) means the next crash leaves you a full postmortem to open in gdb, rather than just a log line.

# a hung process: dump every thread’s stack to find the deadlock
$ sudo gdb -p 812 -batch -ex "thread apply all bt" 2>/dev/null | head
Thread 3 ... in futex_wait ()   <- stuck waiting on a lock
# enable + find core dumps for crash postmortems
$ ulimit -c unlimited
$ coredumpctl list                 # systemd captures cores here
$ coredumpctl gdb 812              # open the crash in gdb

ltrace and strace

Where strace (from the tracing lesson) shows system calls, ltrace shows library calls — the functions a program calls in shared libraries like libc. That is a different, often more application-meaningful view: you see the malloc, the string operations, the library API calls, not just the raw syscalls underneath. Between strace (kernel boundary) and ltrace (library boundary), you can watch what a program does at two levels without any source code or recompilation.

$ ltrace -c ./myapp                 # -c: summarize which LIBRARY calls dominate
% time     seconds  calls      function
 68.2      0.412     1049283   strlen
 20.1      0.121       88211   malloc
#   → the program is dominated by strlen/malloc — an algorithmic clue, no source needed

Modern power tools

A handful of newer CLI tools have become famous for speed and ergonomics and are worth installing everywhere you work. jq is a query language for JSON — indispensable now that logs and APIs are JSON, letting you filter and reshape it the way awk does text. ripgrep (rg) is a dramatically faster grep that respects .gitignore. fd is a friendlier, faster find. fzf is a fuzzy finder that supercharges history and file selection. They do the same jobs as the classics, much faster and more pleasantly.

$ journalctl -o json | jq -r 'select(.PRIORITY<="3") | .MESSAGE'   # filter JSON logs
$ rg -n "connection refused" /var/log        # ripgrep: far faster than grep -r
$ fd -e conf /etc                            # fd: a friendlier, faster find
$ history | fzf                              # fuzzy-search your command history interactively