SSH keys, FIDO keys, MFA and bastions

Key lifecycle, MFA and a single entry point.

Once passwords are off, an SSH key is the credential that matters, and every copy of a private key is a way in. In this lesson you limit what each key may do on the server, revoke a lost key on every server at once, add a second factor that a stolen key file does not carry, and route access through a single bastion. You also look at the failures these controls bring: a clock that drifts, a phone that is lost, and the drop-in order on RHEL that breaks MFA before it starts. Creating keys and using ssh-agent are covered in Linux essentials; the server baseline is the previous lesson.

Restricting what a key may do

The key lifecycle starts on the client: one key per person and device, ed25519 (ssh-keygen's default type since OpenSSH 9.5), a passphrase so that a copied key file is useless on its own, and ssh-add -t 8h so the agent forgets it at the end of the day. The server side is where you limit the damage a key can do. Each line in authorized_keys can carry options in front of the key (sshd(8), AUTHORIZED_KEYS FILE FORMAT). The threat here is an automation key: it has no passphrase, it sits on a build server, and whoever steals it gets whatever it allows.

The lab's accounts: hard-keys-ci for a build job, hard-keys-mfa for a person with a phone, and hard-keys-dev for a developer with an old and a new laptop, plus a group for the bastion at the end. deploy holds all their private keys. The tests use a second sshd on 127.0.0.1:2222 as in the previous lesson, whose Try this stopped it, so start it again:

Now restrict the CI key:

restrict turns off every forwarding, terminal allocation and ~/.ssh/rc, including restrictions added in future OpenSSH versions. command= fixes the only program the key can run. from= accepts the key only from listed addresses; in the lab the "build server" is this machine, so it lists 127.0.0.1. Try to use the key for something else:

The requested cat /etc/shadow was ignored and the forced command ran instead (the program can read what was asked from SSH_ORIGINAL_COMMAND if it needs to). The forwarding request was refused. -b 127.0.0.2 makes the connection come from another local address, standing in for another host, and the right key was refused with a log line that names the address and the rule. expiry-time="YYYYMMDD" is a further option that makes a key stop working on a date, useful for contractors. The operational cost is that the job can only ever do the one thing, which is the aim; plan a separate key for each task.

Revoking a lost key everywhere

A laptop with a key for hard-keys-dev is lost. Deleting its line from authorized_keys works on the servers you remember. A key revocation list (KRL) works on every server that reads it: RevokedKeys names a file of revoked keys, and a key listed there is refused whatever the authorized_keys files say.

ssh-keygen -k builds the list from the public key; -Q checks keys against it and exits with status 1 because one is revoked. Point sshd at the list in your baseline 00-secopslog.conf (in the lab that file holds only this lesson's lines, so the previous lesson's AllowGroups does not get in the way; on your server, the test accounts go in its group). Put the lines near the top, above any Match block: everything after a Match line belongs to that block, so a line appended at the end would apply to the backup account only.

# Refuse every key in the key revocation list (KRL).
RevokedKeys /etc/ssh/revoked_keys

Check and reload; the lab reloads its test instance with a HUP signal, and on a real server this is sudo systemctl reload ssh:

The old key is refused even though its line is still in authorized_keys, and the journal says which file refused it. Add later keys with -k -u, which updates the list instead of replacing it. Two impacts matter. sshd_config(5) warns that if the file cannot be read, public key authentication is refused for all users, so create the list before the setting and ship both with your configuration management. And the list must reach every server, which is where a fleet outgrows loose keys: SSH certificates (TrustedUserCAKeys) let a CA sign short-lived keys centrally. To roll back, remove the RevokedKeys line, test and reload.

FIDO security keys

An ed25519-sk or ecdsa-sk key keeps its private half inside a FIDO2 token such as a USB security key (supported since OpenSSH 8.2). The file on disk is only a handle, useless without the token, and every login needs a touch; ssh-keygen -t ed25519-sk -O verify-required also asks for the token's PIN. The lab has no token, so it shows only that both sides support the key types and that the server does not demand a PIN unless told to:

pubkeyauthoptions none means sshd accepts a signature made with a touch alone. To require the PIN as well, set PubkeyAuthOptions verify-required, or put verify-required in front of that key in authorized_keys; a client-side option alone is a promise the server never checks. The impact is logistics: tokens cost money, get lost, and each user needs a second one registered as a spare.

Keys plus a TOTP code

A TOTP code (time-based one-time password, RFC 6238) is a six-digit number computed from a shared secret and the current 30-second time step, usually by a phone app. Combined with a key, a stolen key file alone no longer logs in. On Ubuntu the PAM module comes from the universe repository (sudo apt install libpam-google-authenticator), and each user enrols once, as themselves (sudo -iu hard-keys-mfa in the lab):

The secret goes into the user's authenticator app (normally scanned from a QR code, turned off here with --qr-mode=NONE) and into ~/.google_authenticator, readable only by the user. --disallow-reuse makes each code work once, --rate-limit=3 --rate-time=30 allows three attempts per 30 seconds, and --window-size=3 accepts the codes for the previous, current and next step. The two scratch codes are single-use emergency codes. The lab copies the secret (the file's first line) to deploy's ~/.totp-secret and the first scratch code to ~/.scratch-code, standing in for the phone and the printed codes.

Now the PAM side: the keyboard-interactive step runs sshd's auth stack, and on Ubuntu that stack starts with @include common-auth, which would ask for the Unix password first. These accounts have no password, so replace it with the TOTP module. Enrol every person first, yourself included, then deploy with nullok, which lets an account without a secret through, and the pam_permit line the module's README pairs with it (a stack in which every module answers "ignore" fails). Two side effects: /etc/pam.d/sshd is a packaged conffile, so the edit can hold back openssh-server updates (the patching lesson), and without common-auth the PAM lesson's faillock no longer covers SSH:

Then tell sshd that every login needs both methods, in the same 00-secopslog.conf: put the global lines above the Match blocks and replace the baseline's KbdInteractiveAuthentication no, since the first value wins. In AuthenticationMethods, a comma means "then": publickey,keyboard-interactive is a key and then a code, while a space would separate alternatives.

# Every login needs a key and then a code from the PAM stack (TOTP).
KbdInteractiveAuthentication yes
AuthenticationMethods publickey,keyboard-interactive
 
# The CI account has no phone: key only, restricted in authorized_keys.
Match User hard-keys-ci
    AuthenticationMethods publickey

The lab has no terminal to type into, so an askpass program answers the prompt; this one runs oathtool (package oathtool) on the same secret, as the phone app would:

#!/bin/sh
# Stands in for the authenticator app: prints the current code.
oathtool --totp -b "$(cat ~/.totp-secret)"

With the code, the login works. With the key alone, the server answers "Permission denied (keyboard-interactive)": the key was accepted and the second step was not done. The CI key still works, because its Match block asks for the key only. During the rollout, nullok still lets an account that has not enrolled in with its key alone. Once everyone is enrolled, remove it, and that account is refused:

That is the impact on everyone: without a secret, no login. Automation, monitoring and backup accounts need their own exemption, restricted as above.

On RHEL the same lines fail before they start, when the file sorts after 50-redhat.conf, which sets the deprecated alias ChallengeResponseAuthentication no:

Read first, the no wins, and sshd refuses a list that needs a disabled method; on RHEL, systemctl reload sshd with this file would stop new logins. In 00-secopslog.conf it passes. RHEL's auth stack is the password-auth substack, which would ask for the Unix password first, and neither google-authenticator nor oathtool (nor pam_u2f for FIDO keys) is in the Rocky BaseOS, AppStream or Extras repositories. EPEL packages google-authenticator as a community package; Red Hat's supported route is Identity Management (IdM), which offers OTP tokens and FIDO2 passkeys through SSSD.

When the second factor fails, and rolling it back

TOTP depends on the phone and the server agreeing on the time. Here the authenticator's clock is two minutes slow:

#!/bin/sh
# An authenticator whose clock is two minutes slow.
oathtool --totp -b -N '2 minutes ago' "$(cat ~/.totp-secret)"

ssh asked for a code three times. Two codes were outside the window of three steps (about 30 seconds either way), and the third attempt hit the rate limit of three logins per 30 seconds; the successful login half a minute earlier counted. The user sees only a plain "Permission denied". The module can learn a steady offset after several attempts (its README; noskewadj turns that off), but the fix is the server's clock: chrony, covered in the logging lesson, and automatic time on the phones. When a phone is lost, a scratch code gets its owner in once:

The scratch code worked and is gone from the file; one is left. The journal shows the module's verdicts under sshd(pam_google_auth), which is the line to alert on for repeated invalid codes; its first line shows it asking the unenrolled hard-keys-dev for a code anyway, which hides who is enrolled. Plan a break-glass path before you need it: an account exempt from TOTP through its own Match block, whose key is kept offline and whose logins page someone, plus console access with a local password. Test it on a schedule, because the day you need it is the wrong day to find it broken.

Rolling TOTP back touches two files, and they must move together. Remove only AuthenticationMethods and keep the PAM edit, and keyboard-interactive becomes a complete login on its own, checked by nothing but the code:

A current code and no key logged in. Revert both in one change: restore @include common-auth and drop the module, remove KbdInteractiveAuthentication yes and AuthenticationMethods, check and reload. If the halves must go separately, revert the PAM file and KbdInteractiveAuthentication first. dpkg --verify finds no difference from the package, so the conffile is back to its packaged state:

One way in: ProxyJump through a bastion

A bastion is the only host that accepts SSH from outside; everything else accepts SSH only from it. The lab builds this with network namespaces: the client can reach the bastion (203.0.113.10) but has no route to the application network (198.51.100.0/24), where the server app1 (198.51.100.20) runs its own sshd.

Host bastion
    HostName 203.0.113.10
    User hard-keys-dev
    IdentityFile ~/.ssh/dev_ed25519
    IdentitiesOnly yes
 
Host app1
    HostName 198.51.100.20
    User hard-keys-dev
    IdentityFile ~/.ssh/dev_ed25519
    IdentitiesOnly yes
    ProxyJump bastion

# SecOpsLog lab bastion, a second sshd in its own network namespace.
ListenAddress 203.0.113.10:22
PidFile /run/hard-keys-bastion.pid
# Keys are root-owned here, so users cannot add their own.
AuthorizedKeysFile /etc/ssh/hard-keys/authorized_keys/%u
PasswordAuthentication no
KbdInteractiveAuthentication no
UsePAM yes
AllowGroups hard-keys-jump
LogLevel VERBOSE
 
# Jump accounts get no shell and may only reach SSH on the app network.
Match Group hard-keys-jump
    PermitTTY no
    ForceCommand /usr/sbin/nologin
    AllowAgentForwarding no
    X11Forwarding no
    AllowTcpForwarding local
    PermitOpen 198.51.100.20:22
    PermitListen none
    AllowStreamLocalForwarding no

SSH_CONNECTION on app1 shows the connection arriving from the bastion's address, which is what its from="198.51.100.10" key option expects. Without the jump there is no route. On the bastion itself the account gets no shell (ForceCommand runs nologin), and PermitOpen refuses a hop to any other port. PermitOpen covers only forwarding from the client's side; AllowTcpForwarding local with PermitListen none also refuses remote forwards (ssh -R), which would open listeners on the bastion that other users could reach, and AllowStreamLocalForwarding no does the same for Unix sockets:

ProxyJump opens a TCP channel through the bastion and runs the real SSH session inside it, so your key and your agent never leave the laptop. Running ssh on the bastion, or forwarding your agent with ssh -A, lets anyone with root there use your keys. Put MFA on the bastion, the one place every session passes.

Even at LogLevel VERBOSE (the CIS value), the bastion logged the logins, the forced command and the refused hop, but not where the permitted jump went, and the refused remote forward left only its login line; app1 logged the bastion as the source. Reconstructing a session means correlating both logs by time, on a central log server. To roll back the bastion, remove the ProxyJump lines from the clients' configuration and the from= restriction on app1 before you stop the bastion's sshd.

Try this

On an Ubuntu test machine with a test sshd, install a key with restrict,command="/usr/bin/df -h /" for a throwaway user and confirm that ssh ... 'cat /etc/shadow' prints the df output. Revoke a second key with sudo ssh-keygen -k -f /etc/ssh/revoked_keys, add RevokedKeys to your drop-in, reload, and find "revoked by file" in journalctl -t sshd-session. Then enrol the user with google-authenticator, switch the PAM stack and AuthenticationMethods as above, and confirm three results: a current code from oathtool --totp gets in, the key alone gets "Permission denied (keyboard-interactive)", and a code made with -N '2 minutes ago' is refused. Finish by reverting the PAM file and the sshd lines together.

Takeaway

Give every key the narrowest options that still do its job, keep a revocation list that every server reads, and require a second factor where people log in, with the drop-in named so it is read first. Keep clocks synchronised and a tested break-glass path, and let people reach internal hosts only through a bastion with ProxyJump.

Related