Keeping secrets off the host's surfaces

Environment leaks and systemd credentials.

A server holds credentials for the systems it talks to: database passwords, API tokens, keys for object storage. Each one sits somewhere on the host, and each place has its own set of readers. In this lesson you find out who can read a secret in an environment variable, on a command line, in shell history and in a configuration file, then move a service's token into a systemd credential, encrypt it at rest with systemd-creds, and rotate it. The goal is a rule you can apply to any service: the secret reaches only the process that needs it, and nothing else on the host can read it. Linux essentials showed how the environment works; this lesson assumes that.

Who can read a secret, by where it sits

The threat is local. An intruder who gets a shell as any account, or a bug that lets one service read another's files, starts by collecting credentials, because they lead to the next system. The question for each place a secret can sit is how many accounts can read it.

Start with the most common pattern: a token passed to a service with Environment=. The demo service runs as its own account, created with sudo useradd --system --no-create-home --shell /usr/sbin/nologin hard-secrets-app, and gets a placeholder token that way. Its program only starts a helper and waits:

#!/bin/sh
# Demo app: starts a helper process and waits. The helper inherits the environment.
sleep infinity &
wait

[Unit]
Description=Demo: a service given its API token through the environment
 
[Service]
Type=exec
User=hard-secrets-app
# Anti-pattern: every local user can read this value.
Environment=API_TOKEN=LAB-ONLY-not-a-real-token-1111
ExecStart=/usr/local/lib/hard-secrets/env-app

Make the program executable (sudo chmod 755), run sudo systemctl daemon-reload and start the unit. Then check what an ordinary user sees, with no sudo:

systemctl show reads unit properties from systemd over D-Bus, and Environment= is one of them, available to every local account. The unit file is world-readable too: 0644 is the normal mode for unit files. The systemd.exec(5) manual is explicit that environment variables are not suitable for secrets: they are exposed to unprivileged clients over D-Bus and they propagate down the process tree. The process itself is better protected:

The kernel lets only the process owner and root read /proc/PID/environ. But the service started a helper, sleep, and the helper carries the token too: every program a service runs (a backup tool, a plugin, a shell script) receives a copy, and any of them may log its environment when it fails. A core dump holds the process memory, environment included; on RHEL, systemd-coredump also stores the environment block in the journal entry for the crash (the COREDUMP_ENVIRON= field in systemd-coredump(8)). Moving the value into a root-only file with EnvironmentFile= hides it from systemctl show, but not from the process. Here /etc/hard-secrets/env is a root-only (0600) file holding one line, TOKEN= and a placeholder, started as a transient unit:

systemctl show now lists only the file's path, while the process environment still holds the token (grep -c counts it without printing it), and so will every child it starts.

Command lines and shell history

A secret passed as an argument is worse. A stand-in sync tool (a script that only sleeps) runs as hard-secrets-app with its password on the command line, the way many tools allow; it was started with sudo systemd-run --unit=hard-secrets-sync -p User=hard-secrets-app /usr/local/lib/hard-secrets/hard-secrets-sync --user=backup --password=...:

deploy sees another account's password without sudo, for as long as the process runs. Every account on the host can do the same, because /proc is mounted without the hidepid option (the mount options show none), which is the Ubuntu and RHEL default. Mounting the host's /proc with hidepid=invisible hides other users' processes from every account, but it also breaks monitoring tools that expect to see every process. systemd's ProtectProc=invisible works the other way round: it hides other users' processes from one service (the sandboxing lesson shows it), which is worth setting, but it does nothing to hide that service's own command line from other accounts. The real fix is at the source: pass the secret in a file (many tools have a --password-file or similar option) or on standard input.

What you type is stored too. bash writes your commands to ~/.bash_history when the shell exits, so a password typed as an argument lands in a plain file in your home directory.

Ubuntu's default ~/.bashrc sets HISTCONTROL=ignoreboth: a command that starts with a space is not saved, and neither is a line identical to the one before it. A leading space is a safety net for a command you had to type with a secret in it, not a habit to rely on. Better, let the program prompt for it, or read it into a variable without echo: read -rs TOKEN prints nothing and stores nothing in history.

A file only the application can read

Most applications read their secrets from a configuration file. The rule is owner root, group of the application's account, mode 0640, in a directory with mode 0750. The application can read the file but not change it, so a compromised application cannot rewrite its own configuration, and no other account can read it at all. Create the directory and an empty file (size 0) with the final owner and mode before any secret goes in:

Then add the value with sudoedit /etc/hard-secrets/app.conf, which edits a temporary copy and writes it back into the existing file, so the owner and mode stay as they are while the size grows. Verify from each side:

The application account reads the file (grep -c prints a count, not the secret), deploy is refused, and the application cannot append to it. The order matters, and this is why:

A file created the usual way gets mode 0644 from root's umask, readable by everyone until someone remembers the chmod. install -m 0640 -o root -g ... /dev/null FILE creates the empty file with the right owner and mode first. The impact of the control is that anything else that needs the file, such as a monitoring check running as another account, needs its own group membership or its own copy. The rollback is chmod and chown; if the file was ever readable by others while untrusted code ran on the host, rotate the secret as well. The pattern is SecOpsLog advice.

systemd credentials

systemd has a mechanism built for this, and it removes the need for the application account to read any file on disk. With LoadCredential=, systemd, running as root, reads the file when the service starts and places a private, read-only copy in a directory whose path it passes to the service in $CREDENTIALS_DIRECTORY. The source file can stay root:root 0600; create it the same way as app.conf (sudo install -m 0600 -o root -g root /dev/null /etc/hard-secrets/token, then sudoedit), and install the demo program below with mode 755.

[Unit]
Description=Demo: a service that reads its API token from a systemd credential
 
[Service]
Type=exec
User=hard-secrets-app
# systemd reads the file as root and gives the service a private, read-only
# copy in $CREDENTIALS_DIRECTORY. The account needs no access to the file.
LoadCredential=api-token:/etc/hard-secrets/token
ExecStart=/usr/local/lib/hard-secrets/demo-app

#!/bin/sh
# Demo app: read the API token from the credential systemd provides and log only a
# fingerprint of it, never the token itself.
set -eu
token=$(cat "$CREDENTIALS_DIRECTORY/api-token")
echo "token loaded from $CREDENTIALS_DIRECTORY, sha256 $(printf %s "$token" | sha256sum | cut -c1-12)"
exec sleep infinity

api-token is the credential's name inside the service, and the path after the colon is where systemd reads it. The demo logs a short SHA-256 fingerprint, which lets you tell two token versions apart without ever writing a token to the journal. journalctl -n 1 --grep sha256 shows the latest matching line; with --grep, -n lists the most recent matches newest first.

The service found its token, and systemctl show now gives an unprivileged user an empty Environment= and not even the path of the credential. Look at the copy systemd made:

The copy is owned by root with an ACL entry that grants read access to hard-secrets-app and nobody else, and deploy cannot even list the directory. findmnt shows why it never reaches the disk: each service's credentials live in their own small tmpfs, mounted read-only and with noswap, so the plaintext is not written to swap either. The directory exists only while the service runs, and backups of the disk never see it.

Encrypted at rest, rotated on purpose

The source file is still plaintext on disk, readable by root and present in every backup. systemd-creds encrypt turns it into a ciphertext that systemd decrypts only when the service starts. The key is either derived from the machine's TPM2 chip, which never lets it leave the hardware, or kept in /var/lib/systemd/credential.secret, or both. The lab VM has no TPM:

partial with -firmware and -driver means no TPM2 device is present (systemd-analyze(1)), so use the host key explicitly. On hardware with a TPM2, the default --with-key=auto binds the credential to both the TPM2 and the host key.

The first use creates the host key, and systemd warns that it is not on encrypted media: anyone who can read the root filesystem offline can read the key, which is the reason to prefer a TPM2, or full-disk encryption as in the disk encryption lesson. The output is Base64 ciphertext; --name=api-token is embedded in it, so the credential cannot be renamed and fed to a different service unnoticed. One more consequence of a key on disk: if you build a golden image and anything in the build encrypts a credential or starts a unit with LoadCredentialEncrypted=, the key is created inside the image, and every clone shares it. A credential encrypted on one host then decrypts on all of them. Delete /var/lib/systemd/credential.secret when you seal the image (as you do with /etc/machine-id and the SSH host keys), let each host create its own, and encrypt credentials on the target host. Now point the unit at the encrypted file, delete the plaintext, and restart:

[Unit]
Description=Demo: a service that reads its API token from a systemd credential
 
[Service]
Type=exec
User=hard-secrets-app
# systemd decrypts the credential as root and gives the service a private,
# read-only plaintext copy in $CREDENTIALS_DIRECTORY.
LoadCredentialEncrypted=api-token:/etc/credstore.encrypted/hard-secrets-token
ExecStart=/usr/local/lib/hard-secrets/demo-app

Same fingerprint on both lines (the newest is on top), so the service received the same token, now from ciphertext. /etc/credstore.encrypted ships with Ubuntu and RHEL as a root-only (0700) directory, and systemd searches it when a credential is named without a path. Credentials are read once, at start. Rotation is therefore two steps: write the new value, then restart the service. Here the new token is a random value that is never displayed; in practice it is the one your provider issued.

The newest line, at the top, has a new fingerprint. Revoke the old token at the provider as well, since rotation on the host does nothing about a copy someone already took. The name check works as described:

Two consequences for the rest of your tooling. Backups: the ciphertext in /etc/credstore.encrypted is useless without /var/lib/systemd/credential.secret, so a backup that contains both contains the secret. SecOpsLog advice is to exclude the host key from backups and re-encrypt credentials from your secret store after a restore (with a TPM2 the key is not on disk at all). Integrity monitoring: AIDE (the integrity lesson) stores checksums and attributes of files, not their contents, so watching /etc/credstore.encrypted shows you when a credential changed without copying it into a report. Keep its database root-only regardless, since a checksum of a short password can be guessed offline. The rollback for this whole section is systemd-creds decrypt into a root-only file and the LoadCredential= line from before.

On RHEL 10

RHEL 10 ships systemd 257, which supports the same LoadCredential=, LoadCredentialEncrypted= and systemd-creds, with the same /etc/credstore.encrypted directory. The same unit works unchanged with SELinux enforcing:

Try this

On your Ubuntu lab machine, run sudo systemd-run --unit=try-env -p User=nobody -p Environment=TOKEN=placeholder sleep 300 and, without sudo, find the token with systemctl show. Stop it. Then write a placeholder into a root-only file with install -m 0600 /dev/null and sudoedit, and start sudo systemd-run --unit=try-cred -p User=nobody -p LoadCredential=token:/path/to/file sleep 300. Confirm that systemctl show -p Environment try-cred is empty, that sudo ls -l /run/credentials/try-cred.service/ shows the copy with an ACL for nobody, and that ls on that directory without sudo is refused. Stop the unit and check that the directory is gone.

Takeaway

Give a service its secret through a systemd credential, encrypted with systemd-creds, and read by the service from $CREDENTIALS_DIRECTORY; never through Environment=, an argument, or a file other accounts can read. Rotate by writing the new value and restarting, and verify with fingerprints, never by printing the secret.

Related