Secrets: where they come from and where they leak

Files, credentials, stdin and env compared, plus argv, log and child-process leaks.

An automation job needs credentials: an API token, a database password, a webhook key. This lesson is about where your program should get them and every place they leak on the way: file permissions, the process list, other processes' environments, service definitions, logs and git history. You will read a secret from a file with a permission check, from standard input and from systemd credentials, watch a second account read a token from another user's command line, redact secrets from log output, and handle a token that was committed. Every secret in the lab is an obvious test value.

Where a secret should come from

Each source hands the secret to your program and to someone else as well; the question is who. This is the ranking the whole track uses, best first:

A secret manager (HashiCorp Vault, a cloud provider's secret store) adds audit logs, central rotation and short-lived credentials; its client library is a pointer here, and scripting-adv covers integrating one. The rest of this lesson works through the other rows on the lab machine. Unpack the lesson files (the box at the top of this page) in your home directory and work in ~/ps-secrets; they hold the scripts listed below. Two steps need sudo: one creates a second account, the other stores a credential under /etc/credstore, and the end of the lesson removes both.

A secret file only you can read

A file is the simplest source, and it is only as private as its permissions. secret_source.py reads one and refuses it when anyone else could read it:

"""Load a secret from the best source available: a systemd credential, a 0600 file, or stdin."""
import getpass
import os
import stat
import sys
from pathlib import Path
 
 
class SecretError(Exception):
    """The secret is missing, empty, or stored where other users can read it."""
 
 
def read_secret_file(path: Path) -> str:
    """Read a secret from a file that only its owner can read (mode 0600 or 0400)."""
    try:
        with open(path, encoding="utf-8") as f:
            info = os.fstat(f.fileno())  # check the file we opened, not whatever the name points to later
            if info.st_uid != os.getuid():
                raise SecretError(f"{path}: owned by uid {info.st_uid}, not by uid {os.getuid()}")
            if info.st_mode & 0o077:
                raise SecretError(f"{path}: mode {stat.filemode(info.st_mode)} lets other users read it")
            value = f.read().strip()
    except OSError as err:
        raise SecretError(f"{path}: {err.strerror}") from err
    if not value:
        raise SecretError(f"{path}: empty")
    return value
 
 
def load_token(name: str, token_file: str | None) -> tuple[str, str]:
    """Return (secret, where it came from), trying the sources in order of preference."""
    cred_dir = os.environ.get("CREDENTIALS_DIRECTORY")  # set by systemd for LoadCredential=
    if cred_dir and Path(cred_dir, name).is_file():
        # systemd made this file: owned by root, readable only by this unit's user (an ACL)
        path = Path(cred_dir, name)
        value, source = path.read_text(encoding="utf-8").strip(), f"systemd credential {path}"
    elif token_file:
        value, source = read_secret_file(Path(token_file)), f"file {token_file}"
    elif sys.stdin.isatty():
        value, source = getpass.getpass(f"{name}: ").strip(), "the terminal (getpass, not echoed)"
    else:
        value, source = sys.stdin.readline().strip(), "standard input"
    if not value:  # from every source: an empty secret is an error, never an empty token
        raise SecretError(f"empty secret from {source}")
    return value, source

os.fstat() asks about the file that is already open, so the check and the read are about the same file even if someone swaps the name in between. st_mode & 0o077 keeps the group and other permission bits: any bit set there means another account can read or change the file. stat.filemode() prints the mode as ls -l does. load_token() tries the sources in the order of the ranking, and refuses an empty value whichever source it came from; the systemd branch is explained in the next section. use_token.py uses it and shows only a fingerprint, the first 12 hex digits of the token's SHA-256, which identifies a token without revealing it. That is safe for a long random token only: a password or a PIN can be found again from its hash by trying candidates, as the hashing lesson shows, so for those log nothing (or an HMAC with a separate key).

"""Load the API token and say where it came from, without printing it.
 
Usage: use_token.py [--token-file PATH]   (otherwise: systemd credential, then stdin)
"""
import hashlib
import sys
 
from secret_source import SecretError, load_token
 
 
def main(argv: list[str]) -> int:
    token_file = argv[1] if len(argv) == 2 and argv[0] == "--token-file" else None
    try:
        token, source = load_token("api-token", token_file)
    except SecretError as err:
        print(f"error: {err}", file=sys.stderr)
        return 1
    # A short hash identifies a random token without revealing it. Never do this for a
    # password: it can be guessed back from its hash (the hashing lesson shows how).
    fingerprint = hashlib.sha256(token.encode()).hexdigest()[:12]
    print(f"token from {source}, sha256 fingerprint {fingerprint}")
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

The first command creates the token file the careless way, readable by everyone, as a file copied from another machine often is:

api-token with mode 0644, readable by every account on the host, was refused with exit status 1. After chmod 600 it loaded. A mistyped path and a file you may not read both end with one clear line instead of a traceback, because OSError is turned into SecretError. Create secret files with the right mode from the start, for example install -m 600 /dev/null api-token before writing to it, or in a subshell with umask 077, so they are never readable, even for a moment.

For a person at a terminal, or a pipe from a password manager, standard input avoids a file altogether:

When standard input is a terminal, load_token() calls getpass.getpass(), which prints the prompt api-token: and turns off echo while the person types, so the value never appears on screen (the lab checked this through a pseudo-terminal). When input is redirected, as here, it reads one line. An empty input is an error, not an empty token.

systemd credentials, and why not Environment=

A scheduled job should not depend on a file in someone's home directory. The host side of this is covered in the Linux hardening lesson "Keeping secrets off the host's surfaces": a token in a unit's Environment= is readable by every local account through systemctl show, and LoadCredential=ID:PATH replaces it. With that setting systemd itself (running as root) reads the file when the service starts and places a copy in a private directory whose path the service finds in $CREDENTIALS_DIRECTORY, so the source file can belong to root. This section is about what your Python code sees. Store the token under /etc/credstore/, the location the systemd.exec manual recommends, and run use_token.py as yourself in a transient service:

Your own account (deploy on the lab machine) cannot read the stored copy. systemd-run creates a unit for one command: --uid="$USER" runs it as you, --wait --pipe keeps the command in your terminal and returns its exit status, --collect removes the unit afterwards even if it fails, and --same-dir runs it in the current directory. The interpreter is given by absolute path because the service does not get your shell's PATH, and the units are named after the lesson so they are easy to clean up.

The service found the token through $CREDENTIALS_DIRECTORY. The last command shows why load_token() does not apply the "owned by me" check there: the copy lives on a small read-only tmpfs created for that one unit, belongs to root with mode 0440, and the + after the mode means an access control list (ACL) grants the service's user read access. systemd has already restricted it to the unit's user and root. For a permanent service, put the same LoadCredential= line in the unit file.

What other users can see: argv and the environment

The same Linux lesson shows which process details other accounts can read. The short version: every local account can read any process's command line, while /proc/PID/environ has mode 0400, so only the process's own user and root can read its environment. Here is the command-line half from your Python tool's side. slow_job.py only sleeps for a minute:

"""Stands in for a long-running job. It ignores its arguments and sleeps for a minute."""
import time
 
time.sleep(60)

Create a second account with no special rights, ps-secrets-other, then start the job the careless way, with the token both in its environment and on its command line (the value is a fake; a real one typed at the prompt would also land in your shell history):

ps-secrets-other read the token from the command line twice: with ps, and from /proc/PID/cmdline, which is where ps gets it (sudo -u is only how the lab acts as that account). The environment is out of that account's reach because of the 0400 mode, but not out of your own processes' reach, and every child process gets a copy. The subprocess lesson showed how to give a child a scrubbed environment with env=. A simpler habit for a token that arrives in the environment is to take it out as you read it:

"""Take the token from the environment once, then keep it away from child processes."""
import os
import subprocess
import sys
 
token = os.environ.pop("API_TOKEN", "")  # read it and remove it: children started later do not inherit it
if not token:
    sys.exit("error: API_TOKEN is not set")
child = subprocess.run(["printenv", "API_TOKEN"], capture_output=True, timeout=5)
print(f"token read ({len(token)} characters); a child process sees it: {child.returncode == 0}")

With os.environ.pop() the child started afterwards did not see the variable (False); the sed copy that uses get() passed it on (True). pop() does not erase the copy the kernel shows in /proc/self/environ, which still holds the value the process started with (the lab checked), so it protects children, not your own process from your own user. Shell history is the last place a typed token lands; the Linux lesson covers HISTCONTROL and why a leading space is a safety net, not a way to handle secrets.

Keeping secrets out of logs

Secrets reach logs by accident: a debug line someone forgot, or an error message that contains a URL with an API key in its query string. A logging filter can replace known secret values in every record before it is written:

"""A logging filter that replaces known secret values before a record is written."""
import logging
 
 
class RedactSecrets(logging.Filter):
    def __init__(self, secrets: list[str]):
        super().__init__()
        self.secrets = [s for s in secrets if s]
 
    def scrub(self, text: str) -> str:
        for secret in self.secrets:
            text = text.replace(secret, "[REDACTED]")
        return text
 
    def filter(self, record: logging.LogRecord) -> bool:
        record.msg = self.scrub(record.getMessage())  # the message with its arguments filled in
        record.args = None
        if record.exc_info:  # the traceback text can hold the secret too
            record.exc_text = self.scrub(logging.Formatter().formatException(record.exc_info))
        return True  # keep the record, now without the secret

A filter is an object whose filter() method sees each log record; returning True keeps the record. getMessage() fills the arguments into the message, so the scrub covers log.info("key %s", token) too. A traceback is formatted separately, so the filter formats it, scrubs it and stores the result in exc_text, which the formatter then uses instead of formatting it again.

"""A sync job whose failure message contains the token, as happens with keys in URLs."""
import logging
import sys
 
from redact import RedactSecrets
from secret_source import SecretError, load_token
 
log = logging.getLogger("sync_job")
 
 
def call_api(token: str) -> None:
    url = f"https://api.example.test/v1/sync?key={token}"  # some APIs take the key in the URL
    raise ConnectionError(f"cannot reach {url}")
 
 
def main() -> int:
    try:
        token, _ = load_token("api-token", "api-token")
    except SecretError as err:
        print(f"error: {err}", file=sys.stderr)
        return 1
    handler = logging.StreamHandler()  # writes to stderr
    handler.addFilter(RedactSecrets([token]))  # on the handler: it sees records from every logger
    logging.basicConfig(level=logging.INFO, handlers=[handler], format="%(levelname)s %(name)s: %(message)s")
    log.info("starting sync with key %s", token)  # a debugging line someone forgot to remove
    try:
        call_api(token)
    except ConnectionError:
        log.error("sync failed", exc_info=True)
        return 3
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main())

With the filter, the token is [REDACTED] in the debug line and in the traceback's last line. Without it (the sed copy deletes the addFilter line), grep finds the token on lines 1 and 9 of the output. Attach the filter to the handler, as sync_job.py does. A filter attached to a logger only sees records logged through that exact logger, not those from its child loggers (the lab checked: a filter on app redacted app's message and missed app.db's). Redaction is a safety net for values you know; the real rule is still to never format a secret into a message.

A committed secret: history and rotation

make_history.sh builds a small repository in which a token was committed and then removed in the next commit, the usual sequence after someone notices:

#!/usr/bin/env bash
# Builds ./notifier: a repository in which a token was committed, then "removed" one commit later.
# Fixed names and dates give the same commit IDs on every run.
set -euo pipefail
export GIT_AUTHOR_NAME=Deploy GIT_AUTHOR_EMAIL=deploy@web01.example
export GIT_COMMITTER_NAME=Deploy GIT_COMMITTER_EMAIL=deploy@web01.example
rm -rf notifier
git init -q -b main notifier
cd notifier
git config core.pager cat  # print output directly (the lesson's terminals cannot page)
git config color.ui false
 
printf '%s\n' 'TOKEN = "lab-token-not-secret-v1"  # TODO: move this out of the code' > notify.py
git add notify.py
GIT_AUTHOR_DATE=2026-09-01T09:00:00Z GIT_COMMITTER_DATE=2026-09-01T09:00:00Z \
  git commit -q -m "Add notifier"
 
printf '%s\n' 'from pathlib import Path' '' \
  'TOKEN = Path("api-token").read_text(encoding="utf-8").strip()' > notify.py
printf '%s\n' 'api-token' > .gitignore
git add notify.py .gitignore
GIT_AUTHOR_DATE=2026-09-02T09:00:00Z GIT_COMMITTER_DATE=2026-09-02T09:00:00Z \
  git commit -q -m "Read the token from a file"
git log --oneline

The fixed dates make the commit IDs reproducible, and the two git config lines only make the output print directly into this lesson's terminals.

git -C notifier runs git as if it had been started inside notifier, so you stay in ~/ps-secrets. git grep finds nothing in the current tree, which is what a reviewer looking at the latest version sees. git log -S (the "pickaxe") lists the commits that added or removed the string, and git show HEAD~1:notify.py prints the file as it was, token included. Every clone, fork and CI cache made since the first commit has that version.

So the fix is rotation, in this order. Issue a new token and revoke the old one at the service that issued it, so the leaked value stops working. Put the new one where the ranking says, here a 0600 file created under umask 077 from secrets.token_urlsafe() (the next lesson covers generating tokens), and ignored by git: git status --ignored marks it !!. Check the issuer's logs for use of the old token since the commit. Rewriting history (for example with git filter-repo) is optional clean-up after that and never a replacement for revoking, because copies you do not control keep the old commits.

Try this

Create a vault/ directory with three secret files and the modes a careless setup leaves behind:

Write audit_secrets.py DIR that checks every entry with Path.lstat() (so a symbolic link is judged as a link) and prints one line per file: ok, not a regular file, owned by uid N, or mode -rw-r--r--, other users can read it when any group or other bit is set. It exits 1 if any file has a problem. Expected first run: vault/api-key (-rw-r--r--) and vault/smtp-password (-rw-r-----) flagged, vault/db-password ok, exit status 1. Break it: test only the "other" bits with 0o007 and see smtp-password pass although every member of its group can read it. Fix the mask, run chmod 600 vault/*, and confirm three ok lines and exit status 0. When you are done, remove the second account (with its home directory) and the stored credential; userdel notes that the account never had a mail spool:

Takeaway

Give a job its secrets through systemd credentials or a secret manager, or a 0600 file your code checks before reading, and take them from standard input when a person is present. Never put a secret in arguments, source code or Environment=; remove it from os.environ once read, keep it out of log messages, and rotate any secret that was ever committed.

Related