The control node & the SSH path

Host keys, bastions, and agent forwarding.

Every Ansible run is a locksmith walking a route. The control node carries the master keys. The route is SSH (Secure Shell, the encrypted remote login protocol). At the far end sits a machine where that locksmith has root, the account that can do anything on a Linux box. Most Ansible security advice is about what happens after arrival: which tasks run, which secrets they read, who approved the change. This lesson is about the route, and about the locksmith. Two halves: the path Ansible travels, and the machine it travels from.

Look At The Connection Ansible Actually Builds

Ansible does not speak SSH itself. The default connection plugin shells out to the OpenSSH client already sitting on the control node, and assembles a long command line out of three sources: built-in defaults, your ansible.cfg, and variables from inventory (Ansible's list of the machines it manages, with settings attached to each). Any task run with -vvv prints that command line word for word before it runs. When you want to know whether a setting took effect, read the exec line, not the config file. Here the inventory has already set ansible_user and ansible_ssh_private_key_file, which is why a user and a key show up.

$ ansible web1.internal.example.com -i inventory.ini -m ansible.builtin.ping -vvv

ansible [core 2.16.6]
  config file = /home/deploy/fleet/ansible.cfg
  python version = 3.11.9
Using /home/deploy/fleet/ansible.cfg as config file
<web1.internal.example.com> ESTABLISH SSH CONNECTION FOR USER: deploy
<web1.internal.example.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s
  -o 'IdentityFile="/home/deploy/.ssh/id_ed25519_fleet"'
  -o KbdInteractiveAuthentication=no
  -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
  -o PasswordAuthentication=no -o 'User="deploy"' -o ConnectTimeout=10
  -o 'ControlPath="/home/deploy/.ansible/cp/8f2a1c9de4"'
  web1.internal.example.com '/bin/sh -c '"'"'echo ~deploy && sleep 0'"'"''
web1.internal.example.com | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

Three things there matter. -C -o ControlMaster=auto -o ControlPersist=60s is the stock value of ssh_args: compression on, plus the connection reuse we come back to later. The ControlPath names the socket file that reuse depends on. And notice what is missing. There is no StrictHostKeyChecking option anywhere on that line. Ansible only adds one when you turn host key checking off, so while checking is on, OpenSSH's own default and whatever your ~/.ssh/config says are left to decide. (The exec line is one physical line. It is wrapped here so you can read it.)

The Fingerprint Check At The Door

A locksmith turns up in the right van, wearing the right uniform. The uniform proves nothing. What proves something is the reference number you were given when you booked, checked against the card in his hand. A host key is that reference number. Every SSH server holds a key pair, proves on connect that it owns the private half, and your client compares the public half against the copy already on file in known_hosts. That one comparison is what stands between your root-capable session and whoever happens to answer.

In ansible-core, host_key_checking defaults to True, which means Ansible passes nothing and OpenSSH applies its own default of ask. What happens next depends on whether the ssh client can reach a terminal. On a CI runner, under cron, or in any session with no terminal to prompt on, an unvouched-for host fails outright and you get the message below. Run the same play from your laptop and the ssh client can still find your terminal, so instead of failing it stops and waits for a yes or no that no automation is going to type. Either way the run does not proceed, which is the behaviour you want and the behaviour people find annoying.

$ ansible-playbook -i inventory.ini site.yml --limit web3.internal.example.com

PLAY [web] *********************************************************************
 
TASK [Gathering Facts] *********************************************************
fatal: [web3.internal.example.com]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Host key verification failed.",
    "unreachable": true
}
 
PLAY RECAP *********************************************************************
web3.internal.example.com  : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

Run the same connection by hand and you get the question everyone recognises. That question, multiplied by two hundred fresh instances a week, is how host_key_checking = False ends up in somebody's ansible.cfg.

$ ssh [email protected]

The authenticity of host 'web3.internal.example.com (10.20.4.7)' can't be established.
ED25519 key fingerprint is SHA256:LP3F4dQ1t5s8kZ2xYb0nR7cV9mA6jH1wQeT4uI8oP0k.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Here is the exact trade you make by switching the check off. Ansible adds one option to every ssh command it builds, and OpenSSH does the rest.

$ ANSIBLE_HOST_KEY_CHECKING=False ansible web3.internal.example.com \
    -i inventory.ini -m ansible.builtin.ping -vvv 2>&1 \
    | grep -o 'StrictHostKeyChecking=[a-z]*' | sort -u

StrictHostKeyChecking=no

With StrictHostKeyChecking=no, OpenSSH records unknown host keys silently and lets the connection proceed even when a key it already had on file has changed. Anyone who can answer for that name or address now gets an SSH session on which you are about to run privileged tasks. On that session you will hand decrypted secrets to a module, meaning one of the small programs Ansible ships out to do a job on the target. Those secrets came out of Ansible Vault, the built-in encryption for values kept in the repository. The traffic is still encrypted. It is encrypted to them. And the positions that let someone answer are ordinary ones. A poisoned DNS (Domain Name System, the internet's name-to-address lookup) record. ARP spoofing on a flat management network, ARP being the Address Resolution Protocol that machines use to find each other on a local network, which is trivially lied to. A recycled network address handed to a different box. A host rebuilt by someone who is not you. With checking on, each of those is a loud failure. With it off, each is a green run.

Manage known_hosts Instead Of Switching The Check Off

The prompt is a symptom. The cure is to already have the right key, not to stop looking. Keep checking on, and give the fleet its own known_hosts file separate from your personal one. Use StrictHostKeyChecking accept-new, available since OpenSSH 7.6: a host you have never seen gets recorded without a question, a host whose key has changed is still refused. That kills the interruption and keeps the alarm. If one scratch environment genuinely needs an exception, scope it instead of globalising it, because the ssh connection plugin reads ansible_ssh_host_key_checking as a host or group variable.

Host bastion
    HostName bastion.example.com
    User jump
    IdentityFile ~/.ssh/id_ed25519_bastion
    ForwardAgent no
 
Host *.internal.example.com 10.20.*.*
    User deploy
    IdentityFile ~/.ssh/id_ed25519_fleet
    ProxyJump bastion
    ForwardAgent no
    StrictHostKeyChecking accept-new
    UserKnownHostsFile ~/.ansible/known_hosts_fleet

The part people skip is where the keys come from in the first place. ssh-keyscan asks the host for its key over the same network you have not verified yet. If someone is already sitting in the path, keyscan collects their key, you write it down, and a temporary weakness becomes a permanent one. Trustworthy sources sit outside that path. Most cloud providers print host key fingerprints in the instance console output. A golden image (the standard machine image you build once and stamp out copies of) can ship a key generated at build time. And SSHFP records (host key fingerprints published in DNS) verified through DNSSEC (DNS answers signed cryptographically, so you can tell forgeries from the real zone) let OpenSSH check the fingerprint against the signed zone when you set VerifyHostKeyDNS yes.

Once the keys come from a source you trust, ansible.builtin.known_hosts writes them out. Point the play at localhost so the file lands on the control node, which is the machine doing the checking, rather than on the targets.

- name: Pin verified fleet host keys on the control node
  hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: Write host keys collected out of band
      ansible.builtin.known_hosts:
        name: "{{ item.host }}"
        # key takes a full known_hosts line: <host> <keytype> <base64 key>
        key: "{{ item.host }} {{ item.keytype }} {{ item.pubkey }}"
        path: ~/.ansible/known_hosts_fleet
        state: present
      loop: "{{ verified_host_keys }}"

Through The Bastion, Not Around It

A bastion, or jump host, is the guard in the lobby of a building you cannot walk into directly. There are two ways past a guard. Hand him your keys and let him go up for you, or have him walk you to the door so you open it yourself. ProxyJump is the second one.

ProxyJump (OpenSSH 7.3 and later, -J on the command line) tells your client to open an SSH connection to the bastion, ask the bastion to forward a plain network connection onward to the real target, then run a complete second SSH handshake with that target inside the tunnel. Your private key is used on the control node and nowhere else. The bastion relays ciphertext it cannot read. Both host keys, the bastion's and the target's, are checked separately, so a bastion that has been taken over cannot quietly become the endpoint.

ProxyCommand ssh -W %h:%p bastion is the older spelling of the same design and behaves the same way. The pattern to retire is the one still sitting in old runbooks: ProxyCommand ssh bastion ssh %h, which logs into the bastion and starts a second ssh client there. Now the bastion needs its own key to the fleet, and the inner session terminates on the bastion where the bastion can watch it. That is handing the guard your keys.

Ansible picks up ~/.ssh/config for free, because it is calling the same ssh binary you do. If you would rather keep the settings in inventory so they travel with the repository, ansible_ssh_common_args is appended to the ssh, sftp and scp commands Ansible builds (sftp and scp being the file-copy tools that ride the same connection).

ansible_user: deploy
ansible_ssh_private_key_file: ~/.ssh/id_ed25519_fleet
ansible_ssh_common_args: >-
  -o [email protected]
  -o ForwardAgent=no
  -o StrictHostKeyChecking=accept-new
  -o UserKnownHostsFile=~/.ansible/known_hosts_fleet

To check your work, ssh -G prints the options OpenSSH will actually use for a host once every config file and match rule has been applied, and it does that without opening a connection. It answers "did my Host block match?" in about a second.

$ ssh -G web1.internal.example.com \
    | grep -E '^(user|forwardagent|stricthostkeychecking|identityfile|userknownhostsfile|proxyjump) '

user deploy
stricthostkeychecking accept-new
identityfile ~/.ssh/id_ed25519_fleet
userknownhostsfile /home/deploy/.ansible/known_hosts_fleet
forwardagent no
proxyjump bastion

ControlPersist And The Socket That Outlives It

Opening an SSH connection costs a key exchange, an authentication, and several round trips across the network. A fifty-task play against two hundred hosts would pay that ten thousand times. Connection multiplexing is the wedge that holds the door open: the first connection to a host becomes a master and creates a socket file under ~/.ansible/cp/, and every later ssh, sftp or scp call for the same user, host and port rides the connection that is already up. ControlPersist=60s keeps that master alive for a minute after the last thing using it lets go.

It speeds up getting in, not doing the work. A play that spends its time compiling something on the target will not notice. A play made of many small tasks gets dramatically faster.

The failure mode is a socket file that is a promise nobody is keeping. Your laptop sleeps, the VPN (virtual private network, the encrypted tunnel into the corporate network) reconnects, a run gets killed, and the master process dies while its socket stays behind on disk. The next run tries to ride a corpse. Ansible retries automatically for exactly one flavour of this, the one where the write fails, and the read side below is not it, so you see the failure.

$ ansible-playbook -i inventory.ini site.yml --limit web

fatal: [web1.internal.example.com]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: mux_client_hello_exchange: read from master failed: Broken pipe",
    "unreachable": true
}

$ ls ~/.ansible/cp/
$ rm -f ~/.ansible/cp/*
$ ansible-playbook -i inventory.ini site.yml --limit web

8f2a1c9de4  b71e0c33a9  d0c9a2f18b
 
PLAY [web] *********************************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web1.internal.example.com]
ok: [web2.internal.example.com]

There is a security point hiding in that directory. A live control socket is an already-authenticated, root-capable session to a managed host, sitting as a file in your home directory, usable by anyone who can open it without presenting a key at all. File permissions are the only guard on it, and root on the control node is not stopped by file permissions. On a shared automation box, that is a free lateral move for anyone who can become root locally.

Pipelining Keeps The Module Off The Target's Disk

Without pipelining, every task is a courier run. Ansible generates a small Python program, copies it into ~/.ansible/tmp/ on the target, runs it, then deletes it. With pipelining, Ansible feeds that program down the connection that is already open, straight into the standard input of the remote Python interpreter. Nothing is written to the target's disk.

That is a speed win, and a quieter one for secrets. Module arguments routinely carry the exact values you bothered to encrypt: a database password on its way into a template, an API token (a password-like string that lets one program call another) on its way into a systemd unit file, the recipe that tells Linux how to start a service. With pipelining off, those arguments sit in a file on the managed host for the length of the task, and for longer if the run is killed before cleanup, or if somebody set ANSIBLE_KEEP_REMOTE_FILES=1 to debug something last month and never unset it. That variable also silently disables pipelining, so the two never protect you at the same time.

# default: pipelining off. Watch for the PUT line.
$ ansible web1.internal.example.com -i inventory.ini \
    -m ansible.builtin.setup -vvv 2>&1 | grep -E 'PUT |/usr/bin/python3 '

<web1.internal.example.com> PUT /home/deploy/.ansible/tmp/ansible-local-4471k2p9f8/tmpv3xq1m TO /home/deploy/.ansible/tmp/ansible-tmp-1784625612.31-4489-2716/AnsiballZ_setup.py
<web1.internal.example.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s ... web1.internal.example.com '/bin/sh -c '"'"'/usr/bin/python3 /home/deploy/.ansible/tmp/ansible-tmp-1784625612.31-4489-2716/AnsiballZ_setup.py && sleep 0'"'"''

$ ANSIBLE_PIPELINING=True ansible web1.internal.example.com -i inventory.ini \
    -m ansible.builtin.setup -vvv 2>&1 | grep -E 'PUT |/usr/bin/python3 '

<web1.internal.example.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s ... web1.internal.example.com '/bin/sh -c '"'"'/usr/bin/python3 && sleep 0'"'"''

No PUT line at all, and the remote command is a bare python3 with the module handed to it over the wire. Turn it on in config once you have tested it.

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=120s
pipelining = True

Pipelining ships off by default for one reason: requiretty. Some hardened sudoers files carry Defaults requiretty, which makes sudo (the command that runs something as another user, normally root) refuse to work unless it has a TTY (teletype, the old name for an interactive terminal). Pipelined execution has no TTY, so privilege escalation fails. Current RHEL (Red Hat Enterprise Linux) and Ubuntu do not set it, but CIS (Center for Internet Security) hardening baselines and elderly golden images often do. Test on a canary host, one machine you break first on purpose, rather than assuming. This is what it looks like when you are wrong.

$ ansible-playbook -i inventory.ini site.yml --limit canary1.internal.example.com

fatal: [canary1.internal.example.com]: FAILED! => {
    "changed": false,
    "module_stderr": "sudo: sorry, you must have a tty to run sudo\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

Agent Forwarding Is A Loaded Gun

An SSH agent is a keyring that signs on request. You type the passphrase into it once, and every ssh command afterwards borrows it without ever seeing the key itself. Agent forwarding stretches that keyring down the wire to the machine you logged into, where it appears as a socket file, so a command running over there can ask your agent back home to sign something. The private key never moves. That is exactly why forwarding feels safe, and exactly why it is not safe.

The agent does not ask what a signature is for. It signs whatever challenge is put in front of it, for anyone who can open the socket. The socket lives in a directory only your user can enter, which sounds fine until you remember who else is on a managed host. Root can open any socket on the box.

# whoami
# ls -ld /tmp/ssh-XkP2mQg1
# ls -l /tmp/ssh-XkP2mQg1/agent.31427
# SSH_AUTH_SOCK=/tmp/ssh-XkP2mQg1/agent.31427 ssh-add -l
# SSH_AUTH_SOCK=/tmp/ssh-XkP2mQg1/agent.31427 ssh [email protected]

root
drwx------ 2 deploy deploy 60 Jul 21 09:14 /tmp/ssh-XkP2mQg1
srwxr-xr-x 1 deploy deploy  0 Jul 21 09:14 /tmp/ssh-XkP2mQg1/agent.31427
256 SHA256:mZ8kQ2vX7pR1sT4uY9wA6bC3dE5fG0hJ2kL8nP4qRsT deploy@control (ED25519)
Last login: Tue Jul 21 08:58:11 2026 from 10.20.1.4
deploy@db1:~$

Root on one web server has now authenticated to a database server as you, using a key it never held, in a way that looks in every log like an ordinary login by you. If the forwarded key is the fleet key, one compromised host becomes the whole fleet. Ansible sharpens this compared with the interactive case, because ForwardAgent=yes in ansible_ssh_common_args forwards to every host in the run at the same time.

The fixes, in the order to reach for them. Do not forward: set ForwardAgent no explicitly in ~/.ssh/config, because a stray ForwardAgent yes under Host * is a classic inheritance accident that nobody notices for years. If the reason you wanted forwarding was reaching a network you cannot route to, use ProxyJump, which solves that without lending out the keyring. If a task on the managed host genuinely needs to authenticate somewhere, pulling source code from an internal Git server being the usual case, give that host its own scoped read-only credential, or fetch the artifact on the control node and copy it out.

$ ssh-add -c ~/.ssh/id_ed25519_admin

Enter passphrase for /home/deploy/.ssh/id_ed25519_admin:
Identity added: /home/deploy/.ssh/id_ed25519_admin (deploy@control)
The user must confirm each use of the key

ssh-add -c marks a key so the agent asks for confirmation on every signature, turning a silent theft into a dialog box you were not expecting. OpenSSH 8.9 went further and added destination constraints through ssh-add -h, which binds a key to a specific chain of hops so a forwarded agent cannot spend it anywhere else. Both are excellent on a laptop. Neither survives a two hundred host run, where one prompt per connection stops being automation.

The Control Node Is The Crown Jewel

Add up what lives on the control node. A private key that authenticates as a user with sudo on every host. A vault password that decrypts every secret in the repository. An inventory, which is a labelled map of everything worth attacking. And the playbooks that will run as root tomorrow morning. No single managed host is worth that much. Taking the control node does not even require an exploit. Reading four files is enough.

That is the argument for making it disposable. A CI (continuous integration) runner that starts as a fresh container, pulls the SSH key and the vault password from a secret manager into memory at job start, runs the play, and is destroyed, leaves no standing credential to steal between runs, offers nobody an interactive shell, and puts a commit behind every change. Compare that with the long-lived automation box where five people have sudo, the key has sat at the same path since 2021, and nobody can say who ran last Tuesday's play.

The honest cost: a disposable runner loses its known_hosts and its multiplexed connections every single time, so verified host keys have to be baked into the image or fetched from a trusted store instead of accepted on first connect, and short plays feel slower. Doing that work is the point. It forces a real answer to the question of where your host keys come from.

Whatever your control node is, look at the permissions on the two files that decide everything. OpenSSH refuses to use a private key that other local users can read, and it tells you so at some length. That warning comes back to you wrapped in Ansible's own error, because Ansible is reporting what the ssh client printed.

$ ls -l ~/.ssh/id_ed25519_fleet ~/.vault_pass
$ ansible web1.internal.example.com -i inventory.ini -m ansible.builtin.ping

-rw-r--r-- 1 deploy deploy 411 Jul 18 11:02 /home/deploy/.ssh/id_ed25519_fleet
-rw-r--r-- 1 deploy deploy  33 Jul 18 11:02 /home/deploy/.vault_pass
web1.internal.example.com | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\nPermissions 0644 for '/home/deploy/.ssh/id_ed25519_fleet' are too open.\r\nIt is required that your private key files are NOT accessible by others.\r\nThis private key will be ignored.\r\[email protected]: Permission denied (publickey).",
    "unreachable": true
}

Notice the asymmetry. OpenSSH stopped you cold over a world-readable key. Ansible said nothing whatsoever about ~/.vault_pass being world readable, and it never will, because it has no opinion about that file's mode. A world-readable vault password means every local account on the box can decrypt every secret in the repository, with no banner to warn anyone. Fix both, then go hunting for the copies you forgot about.

$ chmod 600 ~/.ssh/id_ed25519_fleet ~/.vault_pass
$ find ~ -type f \( -name 'id_*' -o -name '*vault*pass*' \) \
    ! -name '*.pub' -perm /077 -printf '%M %p\n'

-rw-rw-r-- /home/deploy/projects/old-fleet/.vault_pass
-rw-r--r-- /home/deploy/backup/id_rsa_legacy

The ansible.cfg Under Your Feet

A recipe box where you stop at the first card you find and cook only from that one. That is how Ansible chooses its configuration. It looks in four places and stops at the first file that exists: the ANSIBLE_CONFIG environment variable, then ansible.cfg in the current working directory, then ~/.ansible.cfg, then /etc/ansible/ansible.cfg. The first match wins the whole file. They do not merge, so a config sitting in a repository does not extend your settings. It replaces them.

Sit with what that second entry means. You clone a role from a stranger to read through it, cd into the directory, and run a playbook of your own. That repository's ansible.cfg now decides where Ansible loads plugins from, what your inventory is, and whether host keys get checked.

# shipped inside the cloned repository, three lines, no tasks required
[defaults]
inventory         = ./inventory/hosts.sh
callback_plugins  = ./.plugins/callback
host_key_checking = False

The first two lines are code execution. A dynamic inventory source can be an executable script, and if the file arrived from git with its executable bit set, Ansible runs it to collect hosts before a single task starts, as you, with your keys on disk. A callback plugin is Python, and Ansible imports every callback it finds on that path into its own process at startup to see what each one is, before it ever decides whether that callback is enabled. Importing Python runs it. It runs with everything the ansible-playbook process has, including the vault password once that is loaded. The third line switches off the check from the top of this lesson. None of this needs you to run a task from the cloned repository. Running any playbook while standing in that directory is enough.

Ansible does tell you which file it picked. ansible --version prints it as config file =, and ansible-config dump --only-changed prints every setting that differs from the default, with the source of each in brackets.

$ cd ~/clones/handy-nginx-role
$ ansible-config dump --only-changed

CONFIG_FILE() = /home/deploy/clones/handy-nginx-role/ansible.cfg
DEFAULT_CALLBACK_PLUGIN_PATH(/home/deploy/clones/handy-nginx-role/ansible.cfg) = ['/home/deploy/clones/handy-nginx-role/.plugins/callback']
DEFAULT_HOST_LIST(/home/deploy/clones/handy-nginx-role/ansible.cfg) = ['/home/deploy/clones/handy-nginx-role/inventory/hosts.sh']
HOST_KEY_CHECKING(/home/deploy/clones/handy-nginx-role/ansible.cfg) = False

There is one built-in protection, and it is narrower than it sounds. Ansible refuses to read ansible.cfg out of a current directory that is world writable, and it says so loudly.

$ cd /tmp/shared-work && ansible-config dump --only-changed | head -1

[WARNING]: Ansible is being run in a world writable directory (/tmp/shared-work),
ignoring it as an ansible.cfg source. For more information see
https://docs.ansible.com/ansible/devel/reference_appendices/config.html#cfg-in-world-writable-dir
CONFIG_FILE() = /etc/ansible/ansible.cfg

That world-writable rule covers /tmp. It does not cover a repository in your own home directory, which is not world writable and is precisely where you clone things. The durable fix is to stop letting the current directory have a vote: pin ANSIBLE_CONFIG in the CI job definition and in the shell profile on every control node, so the first lookup always wins and no directory you happen to be standing in can move your plugin paths.

$ export ANSIBLE_CONFIG=/etc/ansible/fleet.cfg
$ cd ~/clones/handy-nginx-role && ansible-config dump --only-changed | head -2

CONFIG_FILE() = /etc/ansible/fleet.cfg
DEFAULT_HOST_LIST(/etc/ansible/fleet.cfg) = ['/opt/fleet/inventory']

Related