Hardening the SSH server

Drop-ins, verification and safe rollout.

The SSH server is how you, your colleagues and your automation reach the machine, so it is also what an attacker with a stolen key or a guessed password reaches. In this lesson you write one small configuration file that switches off password logins and direct root logins, limits who may log in at all and tightens the limits on each connection. You prove the result with sshd's own test modes and with real logins against a second server instance, and you roll it out and back without locking yourself out. Client keys and ~/.ssh are in Linux essentials; keys, MFA and bastions in depth are the next lesson.

Where the server takes its settings from

Linux essentials showed the mechanism: /etc/ssh/sshd_config includes /etc/ssh/sshd_config.d/*.conf near its top (line 24 on Ubuntu 26.04, line 15 on RHEL 10), the drop-ins are read in name order, and for each keyword sshd keeps the first value it reads (sshd_config(5)). sudo sshd -T prints the values that result. So the question is always what is already in that directory.

60-cloudimg-settings.conf comes with Ubuntu's cloud images and turns password logins off. 10-acceptenv-colorterm.conf is added by the lab's VM tool. The effective values are OpenSSH's own defaults plus Debian's choices in the main file: 120 seconds to finish logging in, six authentication attempts per connection, no checks on silent clients, root may log in with a key (prohibit-password), and X11 forwarding on. persourcepenalties is on by default since OpenSSH 9.8: an address that keeps failing to log in is refused new connections for a while. Where no file sets PasswordAuthentication, the value is OpenSSH's default, yes.

RHEL ships more drop-ins, and they are the reason for the file name used in this course:

50-redhat.conf sets X11Forwarding yes, so a hardening file named 60-hardening.conf loses: sshd -t accepts it and sshd -T shows that nothing changed. Renamed to 00-secopslog.conf, it is read first and wins. Other files compete for the same slot. cloud-init writes 50-cloud-init.conf (here PasswordAuthentication no, but yes when a user-data file sets ssh_pwauth: true), and the RHEL installer writes 01-permitrootlogin.conf with PermitRootLogin yes when someone ticks "Allow root SSH login with password". The ComplianceAsCode remediations that the CIS profiles use name their file 00-complianceascode-hardening.conf for the same reason.

A baseline drop-in, line by line

The baseline below limits logins to one group, so create the group first and put in it every account that must keep logging in: your own administrator account (sudo usermod -aG GROUP "$USER"), a cloud image's default user, and the accounts that are not people, such as your configuration management's login, backup, monitoring and a break-glass account. If the file reaches a server before the memberships do, automation is locked out and cannot repair it. In the lab, hard-ssh-admin plays you, hard-ssh-backup the backup job, and hard-ssh-other an account that should not get in:

# SecOpsLog SSH server baseline. The 00- prefix makes sshd read this file
# before every other drop-in, and for each keyword the first value wins.
 
# Keys only: no password prompts of either kind.
PasswordAuthentication no
KbdInteractiveAuthentication no
 
# No direct root logins; administrators log in as themselves.
PermitRootLogin no
 
# Only members of this group may log in at all.
AllowGroups hard-ssh-users
 
# Fewer attempts per connection, less time to finish logging in.
MaxAuthTries 4
LoginGraceTime 60
 
# Probe a silent client after 5 minutes; drop it if it does not answer.
ClientAliveInterval 300
ClientAliveCountMax 1
 
# No X11 forwarding on a server.
X11Forwarding no
 
# The backup job's account: no terminal and no forwarding of any kind.
Match User hard-ssh-backup
    PermitTTY no
    DisableForwarding yes

Each line answers a threat, and it helps to know whose advice it is. Password and keyboard-interactive logins are what guessing attacks need; turning both off is SecOpsLog advice for servers where everyone has a key (CIS only forbids empty passwords). PermitRootLogin no is in the CIS RHEL 10 Level 1 server profile; OpenSSH's default still lets root in with a key, and a shared root login leaves no trace of which person used it. AllowGroups is CIS's "limit users' SSH access": a forgotten service account with a key cannot log in unless it is in the group. The lab's group is hard-ssh-users; pick your own name. MaxAuthTries 4 and LoginGraceTime 60 are the CIS values (the defaults are 6 and 120), limiting guesses per connection and how long an unauthenticated connection holds one of the MaxStartups slots (the limit on connections still logging in, beyond which sshd starts refusing new ones). ClientAliveInterval 300 with ClientAliveCountMax 1 is also CIS: after five minutes of silence sshd asks the client for an answer and drops the connection if none comes. A client that answers stays connected however idle its user is, so to end idle sessions use the shell's TMOUT variable (bash exits after that many idle seconds) or logind's StopIdleSessionSec (logind.conf(5)), which ends idle sessions. X11Forwarding no is the upstream default that Debian and Red Hat turn back on; a server rarely needs it, and a forwarded display gives whoever controls the server a way into the user's desktop session (ssh_config(5) warns about this). The Match block applies only to the backup account; it ends at the end of the file, not at the next line of the main file.

sshd -t prints nothing when the configuration is valid. sshd -T shows every value in force, and with -C it evaluates the Match blocks for a connection you describe (user, host name, address), so the backup account and the administrator get different answers from the same files. Check this way before anything reads the file.

Prove it on a second sshd first

A drop-in in place is not yet in force: the running server read its configuration when it started or was last reloaded. That gap is your chance to test, and also a trap: any reload or restart of the real server (a reboot, a colleague, needrestart after the nightly update) applies the untested file. Test at once, and delete the file if you stop before the real reload. Start a second sshd from the same files on a spare port bound to 127.0.0.1, and try real logins against it:

Options given with -o are read before any file, so the test instance listens only on 127.0.0.1:2222 and writes its own PID file. PerSourcePenalties=no is there because every test comes from 127.0.0.1, and a few deliberate failures would otherwise get your own address refused for a while. For the tests, deploy makes six keys and installs one of them for the administrator, the account outside the group and root, and a separate key for the backup account:

Test your own account first, then the ones that must be refused:

The administrator gets in with a key, which on your server proves you keep access before anything else happens. The other account and root are refused with the same message, and so is a client that offers no key, because the server now lists publickey as the only method. The journal says why, as the next terminal shows. Now the backup account, which may log in but not forward:

-W asks the server to open a TCP connection onward, which is what a jump host does. The backup account is refused ("administratively prohibited") and the administrator is not; the SSH banner shown is the one of port 22 on the same machine.

MaxAuthTries has a side effect. Each key the client offers counts as an attempt, and without further configuration ssh offers the keys held in ssh-agent one after another:

With five keys in the agent and the right one last, the server disconnects after four. The fix belongs on the client: IdentitiesOnly yes with an IdentityFile for the host in ~/.ssh/config, or -i as here, so only the right key is offered. The server's log records all of it:

The lines come from sshd-session, the per-connection process OpenSSH has used since 9.8, not from sshd. Root was refused by AllowGroups, which sshd checks as soon as it knows the user name, before PermitRootLogin is consulted; that second lock matters on the day someone adds root's group to the list. On RHEL the same lines go to /var/log/secure, because 50-redhat.conf sets SyslogFacility AUTHPRIV:

The refusal there lists gssapi-keyex and gssapi-with-mic too, because 50-redhat.conf also enables GSSAPI (Kerberos) logins. The CIS Level 2 profile turns that off where Kerberos is not used.

Cryptography: verify, do not hand-tune

Old hardening guides list Ciphers, MACs and KexAlgorithms lines. On current OpenSSH, check what is negotiated before you change anything:

The connection used mlkem768x25519-sha256, a hybrid of a post-quantum algorithm and X25519 that is the default since OpenSSH 10.0. The server offers only modern key exchanges: 10.0 also disabled the finite-field Diffie-Hellman groups in sshd and removed DSA keys entirely. The client can still speak older algorithms so that it can reach old servers, which is why ssh -Q kex lists SHA-1 groups; the server never offers them. SecOpsLog advice: on Ubuntu, leave these keywords unset unless a named requirement demands a change, and record the reason, because a hand-written list freezes today's choice and misses tomorrow's default.

On RHEL, the system-wide crypto policy owns these settings through 40-redhat-crypto-policies.conf, and you change it with update-crypto-policies (for example --set FUTURE, or a policy module), as Red Hat's documentation describes:

The file's comment says it plainly: a Ciphers line in a file read earlier wins. Put one in 00-secopslog.conf and sshd -T shows a single cipher; this server has quietly left the policy, and a later policy change will not reach it. Keep crypto lines out of your baseline on RHEL.

Rolling out, and rolling back

On Ubuntu the listening socket belongs to systemd:

ssh.socket holds port 22 and starts ssh.service on the first connection, which is why the service shows disabled. The settings in this lesson apply with sudo sshd -t && sudo systemctl reload ssh (sshd on RHEL), and sessions already open stay open. Port and ListenAddress are different: the socket unit owns them. openssh-server's README.Debian says that a generator turns them into the socket's ListenStream settings, and that they take effect only after sudo systemctl daemon-reload and sudo systemctl restart ssh.socket. Binding sshd to a management address is a real control; moving it to another port is not, because a port scan finds it in seconds. ssh.socket sets FreeBind=yes, so a mistyped or not yet assigned address binds without an error and nothing can ever connect to it. Arm an undo first, sudo systemd-run --on-active=5min --unit=undo-ssh sh -c 'rm /etc/ssh/sshd_config.d/00-secopslog.conf; systemctl daemon-reload; systemctl restart ssh.socket', connect to the new address from another host, then sudo systemctl stop undo-ssh.timer. On RHEL a new Port also needs an SELinux port label (semanage port -a -t ssh_port_t -p tcp PORT) and a firewalld rule, or sshd cannot bind it. Repeated failures from one address are already slowed down by PerSourcePenalties, which leaves fail2ban little to add on a keys-only server.

Rolling back is the same routine in reverse: remove the drop-in, test, reload. On the test instance, a HUP signal is the reload:

With the file gone, the account outside the group gets in again, and so does root, with a key: that is OpenSSH's default prohibit-password, and the reason PermitRootLogin no belongs in the baseline.

Try this

On an Ubuntu test machine, with a second session open as root (sudo -i) and your own account in the group, copy the baseline to /etc/ssh/sshd_config.d/00-secopslog.conf with a group name of your own, and create a second file, 99-late.conf, containing MaxAuthTries 10. Predict what sudo sshd -T | grep maxauthtries prints, then check (4: the first value wins). Start a test instance with sudo /usr/sbin/sshd -o ListenAddress=127.0.0.1:2222 -o PidFile=/run/sshd-2222.pid -o PerSourcePenalties=no, and confirm that a user outside the group gets "Permission denied (publickey)" while the journal says "not allowed because none of user's groups are listed in AllowGroups". Load five keys into ssh-agent and reproduce "Too many authentication failures", then fix it with IdentitiesOnly. Finish with sudo kill "$(cat /run/sshd-2222.pid)" and remove both files.

Takeaway

Put your SSH settings in one drop-in that sorts first, verify them with sshd -t and sshd -T -C, and prove them with real logins against a test instance before you reload the real server with a session still open. Leave the cryptography to OpenSSH's defaults or RHEL's crypto policy.

Related