Connecting with SSH

Keys, known_hosts, config and debugging.

SSH is how you reach a Linux server: an encrypted connection that gives you a shell on the remote machine, runs single commands there and copies files. This lesson covers the side you use every day, the client: the host key question on a first connection and why it matters, logging in with a key instead of a password, the agent that saves you from retyping a passphrase, a ~/.ssh/config file that shortens every command, copying files with scp and rsync, and reading the errors when a connection fails. The examples connect from web01 to a second server, app01 at 192.0.2.30, which the lab simulates on the same virtual machine, as the account ess-ssh-app.

If you have typed the earlier lessons at your practice machine's console, this is where you move to SSH, which is how servers are normally run: at the end of the lesson you log in to the practice machine from your own computer. If you have been connecting with ssh from the start, this lesson explains what that command was doing.

The first connection and the host key

Every SSH server has its own key pair, the host key, and proves its identity with it on every connection. The first time you connect, your client has nothing to compare the key with, so it shows you the key's fingerprint, a short hash of it, and asks. Get the fingerprint through another channel before you answer: from the server's console, from the boot log your cloud provider shows, or from an administrator, who prints it on the server like this:

(In this lab app01 uses the same host key file as web01, so the command runs here. The text after the fingerprint is only the key's comment.) Now connect. The part before @ is the account on the server:

The fingerprint in the question matches the one the administrator printed, so answering yes is safe. You can also paste the fingerprint you were given instead of typing yes, and ssh compares the two for you. ssh then saves the host key and asks for the account's password, which is not shown as you type. After the login banner (left out here; a first login has no "Last login" line yet) you are in a shell on app01, as the prompt shows; exit closes the session.

The key was saved in ~/.ssh/known_hosts, and every later connection is checked against it without asking. That check is what protects you from a machine in the middle: someone on the network who impersonates the server to capture what you type, including your password.

Ubuntu's client configuration sets HashKnownHosts yes, so the file stores a hash of each host name instead of the name itself, and a stolen copy does not list the servers you use. ssh-keygen -F finds the entry for a host. Comparing fingerprints by hand works for a handful of servers; organisations with many publish their host keys instead (a centrally managed known_hosts file, or SSHFP records in DNS) or sign them as SSH certificates, so that nobody has to answer the question.

app01 accepts passwords, which is OpenSSH's default and the state of many servers built with the Ubuntu installer. Ubuntu's cloud images turn password logins off (the 60-cloudimg-settings.conf file shown in "Reading and editing files safely") and install your public key when the machine is created, which is where the next section ends up.

Logging in with a key

A key pair replaces the password. The private key stays on your machine; the public key goes into ~/.ssh/authorized_keys of the account on the server. At login the server asks your client to sign some data with the private key and checks the signature with the public key, so the private key never crosses the network. ssh-keygen creates the pair; ed25519 is the default key type, and -t ed25519 only makes that explicit. -C sets a comment that tells you later whose key it is:

Pressing Enter at the first question keeps the default file name. The passphrase encrypts the private key file, so a copy of the file is useless without it; it is typed twice and never shown. The fingerprint identifies the key, and the randomart box is the same fingerprint drawn as a picture that is easier to compare by eye.

id_ed25519 is the private key, readable by you alone (-rw-------); id_ed25519.pub is the public key, which is safe to show anyone. ssh-copy-id logs in with the password one last time and appends your public key to authorized_keys on the server:

This time ssh asked for the key's passphrase, which unlocks the private key on your machine and is never sent to the server, and ran hostname on app01. Given a command after the host, ssh runs just that command remotely and returns. With keys working, app01's administrator turns password logins off, as cloud images do. A login that offers no key (-o PubkeyAuthentication=no makes ssh skip its keys) is now refused, and the brackets list the methods the server still accepts:

ssh also refuses to use a private key that other accounts could read:

The agent and ~/.ssh/config

Typing the passphrase for every connection gets tiring. ssh-agent is a small program that holds your unlocked keys in memory for the rest of your session, and ssh asks it to sign instead of reading the key file. Desktop sessions usually start one for you; in a plain terminal session, ssh-add -l shows whether one is running, and eval "$(ssh-agent -s)" starts one (the eval sets the variables that tell ssh where to find it). ssh-add then asks for the passphrase once:

The agent keeps running in the background until eval "$(ssh-agent -k)" stops it. The following commands run in the same session, which finds the agent through SSH_AUTH_SOCK, the path of its socket under ~/.ssh/agent/:

No passphrase prompt this time. The agent can also be forwarded to a server with ssh -A, but anyone with root access on that server can then use your keys while you are connected, so forward it only to machines you trust.

Every line of authorized_keys is a working way into that account, and keys tend to outlive the people and laptops they belonged to. ssh-keygen -lf prints the fingerprint and comment of each line, which tells you whose keys they are:

Typing the account and address every time is tedious and error-prone. ~/.ssh/config gives servers short names:

Host app01
    HostName 192.0.2.30
    User ess-ssh-app

Host is the name you type, HostName the real address or DNS name and User the account. Other common settings are Port, IdentityFile for a particular key and ProxyJump to go through a bastion host (the hardening course uses it). scp, rsync and every other program that runs over ssh read the same file.

Copying files: scp and rsync

scp copies files over SSH the way cp copies them locally. A remote location is written host:path, and a bare host: means the home directory on that host:

On a terminal scp also draws a progress line for each file. Since OpenSSH 9.0 it transfers files with the SFTP protocol, and scp -O falls back to the old protocol for servers that lack SFTP. rsync is the better tool for directories and repeated copies: it compares both sides and sends only what changed. -a copies recursively and keeps permissions and times, -v lists what it sends, and the slash after site/ means "the contents of site":

The second run sent only index.html, the file that changed. rsync must be installed on both machines; "Archives, downloads and checksums" goes further with it. Remove the practice files on both sides when you are done:

When a connection fails

The last line of an ssh error names the layer that failed, and ssh -v shows each step on the way. The most common error is a refused key:

The server does not tell the client why, but it writes the reason to its log. On Ubuntu that is sudo journalctl -u ssh (on RHEL -u sshd); app01's SSH server runs in this lab as a unit called ess-ssh-server:

The account deploy is not allowed to log in to app01 at all. Since OpenSSH 9.8 these lines come from sshd-session, the process that serves one connection. When you cannot read the server's log, -v shows how far the client got:

The connection was made, the host key matched, and the key was offered and refused: the problem is on the server side, such as the wrong account, a key missing from that account's authorized_keys, or a rule like the one above.

The loudest error is a changed host key. Here app01 has been rebuilt with a new one:

ssh refuses to connect, because this is exactly what a machine in the middle would look like. A rebuilt or reinstalled server is the usual, innocent cause, but confirm it: get the new fingerprint from the administrator (for this lab, from the rebuilt key file) and only then remove the old entry and connect again:

The remaining errors come from the network and name resolution, which the previous two lessons explain: a server whose SSH service is stopped, a firewall that drops SSH, and a name that does not exist:

On the server side, Ubuntu keeps port 22 open through ssh.socket, which starts ssh.service on the first connection ("Services with systemd"), while RHEL runs sshd.service directly. Configuring the server itself is the subject of the hardening course.

Try this

Match your key on both ends of a login, with your own computer as the client and your practice machine as the server. On your computer, create a key with ssh-keygen as above and install it on the practice machine: ssh-copy-id USER@ADDRESS works if the server still accepts passwords. A cloud image does not, so there, paste the single line of your .pub file into ~/.ssh/authorized_keys through the console (the directory ~/.ssh with mode 700 and the file with mode 600, both owned by you). If your computer cannot reach the machine's address, which happens with some local VM networks, use a second VM as the client. Print your key's fingerprint with ssh-keygen -lf ~/.ssh/id_ed25519.pub, log in with the key, then on the server run sudo journalctl -u ssh -g "Accepted publickey" -n 1 (on RHEL -u sshd). The SHA256: fingerprint at the end of that line is the same as your key's, which is how an administrator tells which key was used for a login. Then connect with ssh -v and find the Server accepts key line that names the same key.

Takeaway

Check a host key's fingerprint the first time you connect and whenever ssh says it has changed. Log in with a passphrase-protected key held by the agent, and when a connection fails, read the last line of the error, then ssh -v and the server's log.

Related