Reading and editing files safely

Read logs, edit config with drop-ins, validate first.

Most of your time on a server is spent reading: logs to find out what happened, configuration to find out what the machine was told to do. Now and then you change a configuration file, and a mistake there can stop a service or lock you out of the machine. This lesson covers the tools for reading files and logs, the bytes you should not let reach your terminal, and a routine for changing configuration that leaves the package's files alone and checks your change before it takes effect.

Reading files and logs

A few examples write practice files, so start by making a directory for them:

cat prints a whole file at once, which suits short files such as this one-line SSH setting from the Ubuntu cloud image:

Logs are longer and more sensitive, so first check whether you may read them. On Ubuntu the main system log is /var/log/syslog and logins, sudo use and SSH events go to /var/log/auth.log.

Both files belong to the syslog user, which the logging daemon runs as, and to the group adm, with mode rw-r-----: the group may read them and nobody else may. Ubuntu puts the first administrator account in adm, and deploy is in it, so it reads the logs without sudo. Adding someone to adm lets an analyst read logs without sudo, which is far less than root, but it is still broad: adm reads every system log on Ubuntu, including auth.log and the systemd journal, where passwords typed into the user-name prompt and secrets printed by applications turn up. Grant it only to people cleared to read all of that. On RHEL the equivalent files are /var/log/messages and /var/log/secure, readable by root only, so you read them with sudo:

New lines are added at the bottom of a log, so tail is the command you want: it prints the last 10 lines, or as many as -n asks for. logger writes a line into the system log, which is a quick way to check that logging works:

The two lines above yours are whatever else the machine logged a moment earlier, so they are left out here. Each line starts with a timestamp in RFC 3339 format (date, time to the microsecond, and the offset from UTC; Ubuntu 26.04 writes this format), then the host name, then the program that logged the line, then the message. head is the mirror image and prints the first lines; wc -l counts lines, which tells you how much you are dealing with before you open a file:

For anything longer than a screen, use less. It opens even a very large file at once, because it reads only the part you are looking at. Space and b move a page forward and back, /word searches forward and n jumps to the next match, G goes to the end and g to the start, &word hides every line that does not contain word, and q quits. The lesson on logs shows the other half of Linux logging, the systemd journal.

Following a log as it grows

tail -f keeps the file open after printing the last lines and prints each new line as it is written; you stop it with Ctrl-C. Logs do not grow forever, though. logrotate renames them on a schedule (weekly for /var/log/syslog on Ubuntu, keeping four old copies) and a new, empty file takes the old name. The next two terminals follow a small application log, ~/cmd-view/app.log. While tail runs, a second session of deploy (another SSH login, or a second terminal on the console) types three commands a second or so apart: echo "request 2" >> ~/cmd-view/app.log adds a line, mv ~/cmd-view/app.log ~/cmd-view/app.log.1 rotates the log the way logrotate does, and echo "request 3, after rotation" > ~/cmd-view/app.log writes to the new file. Ctrl-C is pressed at the end of each terminal.

tail -f follows the file it opened. After the rename that file is app.log.1, which never grows again, so the third request never appears and nothing tells you. tail -F follows the name instead:

It notices the file disappear, opens the new one when it appears and carries on. Use -F whenever you leave a follow running for a while. Inside less, pressing F follows the end of the file in the same way as tail -f, and Ctrl-C returns you to scrolling.

Bytes that are instructions to your terminal

Your terminal does not only display text. Some bytes, control characters and escape sequences, are commands to it: move the cursor, erase a line, change colours or the window title. A log records whatever a client sent, so anyone who can get a line into a log can put those bytes there too. Here is a web access log line built the way such a request would be logged:

cat -v shows control bytes as visible codes: ^M is a carriage return, which moves the cursor back to the start of the line, and ^[[K is an escape sequence that erases from the cursor to the end of the line. With plain cat, your terminal carries both out, and the screen shows only 198.51.100.4 GET /index.html 200: the probe for shell.php from 203.0.113.7 is printed and then wiped before you can see it. file warns about escape sequences, and less shows control characters as caret codes by default. Read untrusted files with less or cat -v, and never with less -r, which passes them through raw.

Editors, and editing as root with sudoedit

Servers have no graphical editor, so you edit in the terminal. On Ubuntu the default is nano: you type normally, and the bottom two lines list the shortcuts, where ^ means Ctrl. Ctrl-O writes the file (press Enter to confirm the name) and Ctrl-X exits. vi is part of the POSIX standard and present on practically every Linux and Unix system; Ubuntu 26.04 ships vim and a smaller vim.tiny, while a minimal RHEL 10 install has only vi:

vi starts in normal mode, where keys are commands rather than text. Press i to insert text and Esc to return to normal mode, then type :wq and Enter to save and quit, or :q! to quit without saving. When a vi session stops making sense, Esc followed by :q! gets you out without changing the file.

Configuration files belong to root, and sudo nano /etc/... runs the whole editor as root, including anything it can do besides editing, such as running shell commands. sudoedit FILE (the same as sudo -e FILE, and supported by sudo-rs) is the safer habit. It copies the file to a temporary file you own, runs your own editor on the copy as you, and when you quit it copies the result back as root. It refuses to edit symbolic links and files in a directory you can write to, which blocks link tricks. It uses the editor named in the variable SUDO_EDITOR, VISUAL or EDITOR (the environment lesson explains variables); with none of them set, Ubuntu's sudo-rs opens /usr/bin/editor, which is nano. SUDO_EDITOR=vim sudoedit FILE picks another for one run.

Changing configuration: drop-in, validate, reload, verify

The filesystem lesson showed that packages own their files and your changes belong in /etc, preferably as a separate drop-in file. The SSH server is the example that matters most, because a mistake there can lock you out of the server. Its main file pulls in the drop-in directory:

grep -n prints the lines that contain a word, with their line numbers (the lesson on finding files and text covers it). The Include is on line 24, near the top, so sshd reads the drop-ins, in name order, before the rest of the main file. For each keyword sshd uses the first value it reads. Now compare what the main file says about password logins with what the server will actually do:

Line 78 is a comment that shows the built-in default; it sets nothing. sudo sshd -T reads the configuration exactly as the server would and prints the value in force for every keyword (for a connection that matches no Match block; -C user=NAME,host=HOST,addr=ADDRESS shows what one particular login would get), and the pipe | hands that output to grep (the next lesson explains pipes). Password logins are off because 60-cloudimg-settings.conf says so, and since drop-ins are read first, uncommenting line 78 and writing yes there would change nothing. This is the most common reason for "I changed sshd_config and nothing happened".

To make a change, create your own drop-in with sudoedit /etc/ssh/sshd_config.d/10-cmd-view.conf. This one makes the server check on idle connections every five minutes (a harmless setting to practise with), with the comment on a line of its own:

# Probe idle SSH clients every 5 minutes; drop them after 3 missed replies.
ClientAliveInterval 300

Before anything uses the file, test it with sudo sshd -t. Suppose the first attempt had a typo, ClientAlivInterval:

The test refuses the file, names the file and the line, and exits with status 255. Nothing has been applied, because a test never touches the running server.

After fixing the typo, the test prints nothing, which means the configuration is valid, and -T confirms the value that will be used:

reload makes the running server read its configuration again, and connections that are already open, including yours, carry on. The file sudoedit created belongs to root with mode 0644, like the other drop-ins. On Ubuntu, the ssh.service unit also tests the configuration itself before every start and reload:

So on Ubuntu a reload with a broken file fails and the old settings stay in force, but a restart stops the running server first, and the failed test then keeps it from starting again: new logins fail until you fix the file. RHEL's sshd.service has no test step, and its reload only sends the hangup signal. sshd answers that signal by starting itself again with the new configuration, and if the file is broken it cannot start and the listener is gone:

Try this

With the lesson's drop-in in place, add a second one, /etc/ssh/sshd_config.d/90-cmd-view.conf, containing ClientAliveInterval 60, and predict what sudo sshd -T | grep clientaliveinterval will print before you run it. It prints 300, because sshd keeps the first value it reads and 10-cmd-view.conf sorts first. Then undo everything: remove both files with sudo rm, run sudo sshd -t and sudo systemctl reload ssh, and confirm that sudo sshd -T | grep clientaliveinterval reports clientaliveinterval 0, the default.

Takeaway

Read untrusted files with less or cat -v, and follow logs with tail -F. Change configuration by adding a drop-in, testing it (sshd -t, visudo -c), reloading rather than restarting, and confirming the value in force (sshd -T), with a second session open.

Related