Advanced Linux internals & tooling

Correct — On a host that was mostly idle since boot, the since-boot row reads mostly idle. The interval rows, us 100 and id 0 with r at 4 on two CPUs, show saturation now.

Correct — Collection would begin at the next boot. sudo systemctl enable --now sysstat starts it at once, and is-active then prints active.

Correct — The fraction is runnable tasks over all tasks, and the last field is the most recent PID the kernel allocated.

Correct — The lab's virtio disk reports 1 while it is a file on the host's SSD. Measure latency with iostat or fio instead of trusting the flag.

Correct — Bit n-1 stands for signal n, so 0x4002 is SIGINT and SIGTERM. The handler ran and chose to carry on; any line it logs confirms it.

Correct — Both show as D, but TASK_KILLABLE ends on SIGKILL, while TASK_UNINTERRUPTIBLE waits for the I/O to complete with SIGKILL left pending in ShdPnd.

Correct — multipathd.service sets CPUWeight=1000, so Ubuntu's system.slice passes cpu down and nice ranks tasks within one group. Rocky's system.slice passes memory and pids alone.

Correct — EEVDF has a base slice instead (1.4 ms on this 2-CPU VM), readable in /sys/kernel/debug/sched, a debugging interface rather than a supported setting.

Correct — Each worker has a CPU to itself, so pressure and %wait stay low. The load average settles at 2, the CPU count, on a healthy machine.

Correct — some means at least one task in the group waited; full means all of them waited, which two workers sharing one CPU seldom do.

Correct — critical-chain shows the path that gated the target; units off that path can take any time without delaying the boot.

Correct — journald was stopped by systemd-shutdown in the normal order. Next, find who or what asked for the reboot: a user, an update tool or a timer.

Correct — update-grub reads /etc/default/grub, then each file in /etc/default/grub.d/, and the last assignment wins. Put your change in a drop-in that sorts later.

Correct — Requires= pulls db into the transaction but orders nothing, so both started together and app lost the race. With After= the start took five seconds and app connected.

Correct — fail rejects any transaction that conflicts with queued jobs; without the option, replace would have swapped the start for the stop.

Correct — The unit is generated from fstab (SourcePath=/etc/fstab), so it changes only when the generator runs again.

Correct — systemd created the socket and keeps it across service restarts; the kernel completes the handshake and queues the connection, up to the backlog of 4096 that Send-Q shows.

Correct — CONSTRAINT_MEMCG means a cgroup limit was hit, and oom_memcg names that cgroup. A machine-wide OOM says CONSTRAINT_NONE.

Correct — Sequential access let the kernel read ahead, so later faults found their pages cached. Without it each page is a separate major fault, which is how a database whose working set misses the cache becomes disk-bound.

Correct — vm.overcommit_memory=0 is a heuristic: an 8 GiB request, more than RAM and swap together, was refused with ENOMEM. Shortages show up later, when pages are touched.

Correct — 50 ms per 100 ms period is used up every time. Lifting the quota with set-property CPUQuota= let each worker use a full CPU.

Correct — /var/tmp resolved to vda (253:0), 8M means 8,000,000 bytes a second, and io.max is not work-conserving: the read took 8.4 s on an idle disk.

Correct — With --mount-proc, unshare also creates a mount namespace and mounts a new /proc, and ps then sees itself as PID 1.

Correct — The namespace maps UID 1001 alone, so the owner of /etc/shadow shows as the overflow ID 65534 and the capabilities do not apply to it.

Correct — That is why DelegateSubgroup=init put the processes in a child and left the unit's cgroup empty for subtree_control.

Correct — open() creates a description with its own offset (fd 5); dup() and fork() share one, which is also why a child's reads move its parent's position.

Correct — The watch limit, sized to about 1% of memory, fails inotify_add_watch with ENOSPC; the instance limit fails inotify_init with EMFILE instead.

Correct — ext4 and XFS pick physical blocks when the data is written back and they know how much there is, which keeps files contiguous.

Correct — The lab's zeroed directory block passed fsck with status 0 and was found by xfs_repair -n. If the log is dirty, mount and unmount first.

Correct — The result is durable but heavier than intended. gnusync calls fsync() on the file, and the Rust sync --data calls fdatasync().

Correct — The filesystem was made for 200 MiB and does not know the device grew; resize2fs took df to 359M. lvextend --resizefs does both steps.

Correct — Two linear segments concatenate the disks: spanning adds capacity, not redundancy. lvs --segments shows the same layout.

Correct — Four requests are served and four wait for a slot, each for about one service time. Latency above the quiet-time value with flat throughput is the signature.

Correct — Keyed on args.dev, [251, 0] (nullb0) had every request between 8 and 64 ms, and [253, 0] (vda) held the twenty fast ones. biolatency.bt alone keys on the sector and cannot say which device a request went to.

Correct — A child inherits its parent's descriptors, so the blocks are freed when both close it; stopping the writer's unit ends both.

Correct — The columns are processed, dropped and time_squeeze, in hexadecimal. A few squeezes are normal; a count that climbs with traffic, or any drops, means packet processing cannot keep up.

Correct — A qdisc is chosen when an interface comes up, so an interface raised early keeps the built-in pfifo_fast for as long as it stays up.

Correct — RHEL's hosts line lists files, dns and myhostname, in that order; the module answers for the machine's own name and for _gateway, the current default gateway.

Correct — Check ss -ltnp on the server (in the lab the service was bound to 127.0.0.1) and look for a rule that rejects with a TCP reset; a load balancer in front would also answer this way.

Correct — A window of a few kilobytes per round trip gives roughly the throughput measured. Loss does far more damage than its percentage suggests.

Correct — Each counter belongs to one stage. On a real host, loss inside the network is in none of your counters, and TCP retransmissions are the evidence.

Correct — A retransmission count means little until it is compared with TcpOutSegs; nstat without -a gives the change over an interval.

Correct — Nothing is slow: the program asks to sleep. The fix is in the design, a shorter interval or inotify to be woken when a file arrives.

Correct — Ubuntu's perf is built without those BPF skeletons, so it shows pointer values. strace prints the strings, at a much higher cost per call.

Correct — 4 is an Ubuntu addition to the upstream scale, which ends at 2. A one-off sudo leaves the host's policy alone.

Correct — With 21 samples each one is almost 5% of the profile, too coarse to tell 3% from 8%. Aim for hundreds of samples, by running longer, and repeat before trusting small differences.

Correct — On RHEL 10 the bcc tools also compile against the 6.12 kernel, unlike runqlat-bpfcc on Ubuntu 26.04's kernel 7.0.

Correct — fentry reads arguments through BTF, so a one-liner can follow the file to its name; kernel function names and arguments are not a stable interface.

Correct — Each thread holds what the other wants. Making flush_queue take stats_lock before queue_lock, as collect_stats does, removes the cycle.

Correct — systemd-coredump stores the core compressed in /var/lib/systemd/coredump, logs a stack trace in the journal, and coredumpctl list shows it with COREFILE present.

Correct — sysctl.d covers /proc/sys. On a RHEL host running TuneD (a standard install does), a custom profile that includes the active one sets it; otherwise transparent_hugepage= on the kernel command line.

Correct — TuneD detects VMs with virt-what, which does not recognise this hypervisor. Check what an automatic choice was based on, and note that the profile changed several settings at once, none of them measured.