Declarative vs imperative

Describe the end state, not the steps.

There are two ways to get a driver to your front door. You can read out turn-by-turn directions: left at the lights, second right, past the pub, stop after the postbox. Or you can hand over the address and let the driver work it out. Directions only work if the car starts exactly where you assumed it was parked. An address works from anywhere. From halfway there. From the wrong side of town. Steps versus destination. That split runs through every infrastructure tool you will ever touch, and it decides what happens when reality is not what your script expected.

Imperative means you write the actions and the order they happen in. Create a server, install nginx (a web server), start it. The tool performs each action exactly as written and holds no opinion about what was already there. Declarative means you write the end state. There should be one server, running nginx, listening on port 443. The tool reads your description, goes and looks at what actually exists right now, works out the gap, and closes only the gap.

Terraform (a tool that builds cloud resources from a description you keep in git), Kubernetes manifests, CloudFormation and systemd unit files are declarative. A bash script, aws ec2 run-instances and kubectl create are imperative. Ansible sits between the two: a playbook reads top to bottom like a list of steps, but each step is written to look before it leaps, so in practice it behaves close to declarative.

The two-run test

There is a fast way to tell which kind of thing you are holding. Run it twice. If the second run reports nothing to do and changes nothing, it is declarative in effect. If the second run duplicates something, falls over, or quietly makes things worse, you are holding directions. The property you are testing has a name: idempotence, meaning running the same thing five times leaves the machine in the same state as running it once. Here is a hardening script of the kind that exists at almost every company.

#!/usr/bin/env bash
set -euo pipefail
 
# Lock down SSH (secure shell, the service you log into a server through).
# On Debian and Ubuntu the unit is called ssh.service; on RHEL it is sshd.service.
echo 'PermitRootLogin no' >> /etc/ssh/sshd_config
systemctl reload ssh
 

Run it once and root login looks disabled. Run it twice, or let a nightly config job run it every night for a month, and the file keeps growing.

sudo bash harden.sh
sudo bash harden.sh
grep -n 'PermitRootLogin' /etc/ssh/sshd_config

33:#PermitRootLogin prohibit-password
58:PermitRootLogin yes
125:PermitRootLogin no
126:PermitRootLogin no

Two problems here, and the second one is the one that bites. The obvious problem is the duplicate line, which becomes three hundred duplicate lines after a year of nightly runs. The real problem is line 58. Somebody put that there in 2019 and nobody remembers why. For most settings, sshd (the background program on the server that answers SSH logins) takes the first value it finds and ignores every later one. A few keywords stack up instead, like HostKey and ListenAddress, but PermitRootLogin is not one of them. So your appended no sits at the bottom of the file doing absolutely nothing. The script exits 0 both times. Nothing lands in the logs. Root login is still open, and your change ticket says it is closed.

Stop reading the file and start reading the daemon. The -T flag puts sshd in extended test mode: it parses the config the way it would at startup, follows the includes, applies first-value-wins, and prints the settings it actually ended up with.

sudo sshd -T | grep -i permitrootlogin

permitrootlogin yes

That one command is the whole difference between believing you hardened a box and knowing it. It is the declarative idea done by hand: stop describing what you did, go read what is true. Ordering matters one level up as well. Debian 12 and Ubuntu 22.04 both ship an Include /etc/ssh/sshd_config.d/*.conf line at the very top of sshd_config, and that pattern expands in sorted order. First value wins, and the include comes first, so a file dropped into that directory beats every line further down the main file. That directory is where hardening belongs. Watch which number you prefix, though. Ubuntu cloud images ship a file in there called 60-cloudimg-settings.conf that turns password logins back on, so 70-hardening.conf loses to it and 50-hardening.conf wins.

Own the whole file, not a line in it

Appending a line to somebody else's config is like scribbling a correction in the margin of a recipe and hoping the cook reads your handwriting instead of the printed line above it. The declarative move is to own a complete object outright: this file, this exact content, this owner, this mode, with no reference at all to what was there before. In Ansible (a tool that connects to your servers over SSH and makes them match a description you wrote) that is one task. The validate line runs a syntax check on the new file before it is put in place, so a typo cannot lock you out of the box.

- name: SSH baseline
  hosts: all
  become: true
 
  tasks:
    - name: Ship SSH hardening drop-in
      ansible.builtin.copy:
        dest: /etc/ssh/sshd_config.d/50-hardening.conf
        owner: root
        group: root
        mode: '0644'
        content: |
          PermitRootLogin no
          PasswordAuthentication no
          KbdInteractiveAuthentication no
        validate: /usr/sbin/sshd -t -f %s
      notify: Reload sshd
 
  handlers:
    - name: Reload sshd
      ansible.builtin.systemd:
        name: ssh.service
        state: reloaded

ansible-playbook -i 'web01.internal,' harden.yml

PLAY [SSH baseline] ************************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web01.internal]
 
TASK [Ship SSH hardening drop-in] **********************************************
changed: [web01.internal]
 
RUNNING HANDLER [Reload sshd] **************************************************
changed: [web01.internal]
 
PLAY RECAP *********************************************************************
web01.internal             : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Now run the identical command again, and check the answer the way the daemon sees it rather than the way the file looks. Here is the tail of that second run.

ansible-playbook -i 'web01.internal,' harden.yml
ssh web01.internal 'sudo sshd -T' | grep -iE 'permitrootlogin|passwordauthentication'

PLAY RECAP *********************************************************************
web01.internal             : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
 
permitrootlogin no
passwordauthentication no

Notice what did not happen. The handler, which is a task that only fires when some other task actually changed something, stayed quiet. Nothing changed, because the end state was already correct. That changed=0 is a fact you can build an alert on, and this is the operational payoff most people walk straight past. Run this playbook against every host on a schedule and the recap line becomes a monitor. Zeros everywhere means nothing moved since yesterday. A single changed=1 on one host means something on that box drifted, and you want to know who moved it and why before you let the tool quietly move it back. A raw list of shell commands rarely gives you that signal, because it does its thing every single time and never finds out whether it needed to.

The reconciler you are already running

A thermostat is never handed a list of steps. It is handed a number. It reads the room, compares against the number, acts, then does it again, forever. Every declarative system is a thermostat with a different sensor. You are already running one: systemd, the program with process ID 1 that starts and supervises everything else on a modern Linux box. A unit file does not describe a startup procedure. It describes a state that should hold.

[Unit]
Description=Internal API
After=network-online.target
Wants=network-online.target
 
[Service]
ExecStart=/usr/local/bin/api --listen 127.0.0.1:8080
User=api
Restart=always
RestartSec=5
NoNewPrivileges=true
ProtectSystem=strict
PrivateTmp=true
 
[Install]
WantedBy=multi-user.target

Restart=always is the loop. systemd will not launch this once and walk away. It watches the process, and when the process dies it brings it back after RestartSec seconds. Enabling the unit writes the desired boot state to disk as a symlink, and enabling it a second time does nothing at all.

sudo systemctl daemon-reload
sudo systemctl enable --now api.service
sudo systemctl enable --now api.service

Created symlink /etc/systemd/system/multi-user.target.wants/api.service → /etc/systemd/system/api.service.

The second command printed nothing whatsoever, because what it asked for was already true. That is idempotence in its smallest possible package. Now kill the process the way a crash, an out-of-memory kill, or an attacker tidying up after a payload would kill it.

sudo pkill -f /usr/local/bin/api
sleep 6
systemctl is-active api.service
systemctl show api.service -p NRestarts

active
NRestarts=1

You never told systemd to start it again. You told it once what should be true, and it converged. NRestarts is a counter worth scraping into your metrics, because convergence cuts both ways for a defender. A service restarting forty times an hour is either broken or being hit, and the reconcile loop is hiding the symptom by repairing it every time. The same property that keeps your API up will keep quietly restoring a service an attacker keeps crashing, and the only trace is a number nobody is graphing.

Diff, then converge

Terraform makes those middle steps visible, which is exactly what terraform plan is for: refresh, compare, print the difference, apply nothing. Take a security group (a firewall attached to your cloud machines) whose complete rule set lives inside one resource block. The vpc_id points at your virtual private cloud, which is your own walled-off network inside the provider. The 0.0.0.0/0 means every address on the internet.

resource "aws_security_group" "web" {
  name        = "web-sg"
  description = "Public web tier"
  vpc_id      = var.vpc_id
 
  # This block owns the COMPLETE ingress rule set for the group.
  ingress {
    description = "HTTPS from anywhere"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
 
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

At 02:00 during an incident, somebody opens port 22 to the whole internet through the web console so they can get in and fix things. Nobody writes it down. Here is what the next plan says.

terraform plan

aws_security_group.web: Refreshing state... [id=sg-0a1b2c3d4e5f60718]
 
Note: Objects have changed outside of Terraform
 
Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:
 
  # aws_security_group.web has changed
  ~ resource "aws_security_group" "web" {
        id      = "sg-0a1b2c3d4e5f60718"
      ~ ingress = [
          + {
              + cidr_blocks      = ["0.0.0.0/0"]
              + description      = ""
              + from_port        = 22
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 22
            },
            # (1 unchanged element hidden)
        ]
        name    = "web-sg"
        # (8 unchanged attributes hidden)
    }
 
Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.
 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place
 
Terraform will perform the following actions:
 
  # aws_security_group.web will be updated in-place
  ~ resource "aws_security_group" "web" {
        id      = "sg-0a1b2c3d4e5f60718"
      ~ ingress = [
          - {
              - cidr_blocks      = ["0.0.0.0/0"]
              - description      = ""
              - from_port        = 22
              - ipv6_cidr_blocks = []
              - prefix_list_ids  = []
              - protocol         = "tcp"
              - security_groups  = []
              - self             = false
              - to_port          = 22
            },
            # (1 unchanged element hidden)
        ]
        # (7 unchanged attributes hidden)
    }
 
Plan: 0 to add, 1 to change, 0 to destroy.

Read the top half. That is the security-relevant part, and it is the part everyone scrolls past on the way to the yes/no prompt. Objects have changed outside of Terraform is the tool telling you what it found when it went and looked at reality: a rule allowing the entire internet to reach port 22, which your code never asked for. The bottom half is the consequence. The next apply takes it away. Detection and remediation arrive in the same command, from a tool most people file under deployment.

Better still, do not depend on a human reading it. The -detailed-exitcode flag turns a plan into a signal a machine can act on.

terraform plan -detailed-exitcode -no-color > plan.txt
echo "exit=$?"
grep '^Plan:' plan.txt

exit=2
Plan: 0 to add, 1 to change, 0 to destroy.

Exit 0 means reality matches the code. Exit 1 means the run itself failed. Exit 2 means there is a difference. Put that on a timer against every environment and alert on 2. A plan that comes back dirty on a stack nobody deployed to has a short list of possible causes: a teammate clicked something, a controller or autoscaler wrote a field back, or somebody holding credentials you did not expect made a change. You want to be looking at all three within the hour, not at the next release. Keep in mind that a plan needs read access to everything it manages, so the identity running it is a real credential and belongs in your threat model too.

What it will happily destroy

Convergence runs in both directions, and this is where declarative tooling surprises people. If your code says a thing should exist, the tool creates it. If your code stops mentioning a thing the tool created, the tool concludes that thing should not exist. Delete a resource block during a tidy-up, and the plan reads like this. aws_db_instance is a managed relational database, the kind with your customers in it.

terraform plan

aws_db_instance.analytics: Refreshing state... [id=analytics]
 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  - destroy
 
Terraform will perform the following actions:
 
  # aws_db_instance.analytics will be destroyed
  # (because aws_db_instance.analytics is not in configuration)
  - resource "aws_db_instance" "analytics" {
      - allocated_storage       = 200 -> null
      - backup_retention_period = 7 -> null
      - engine                  = "postgres" -> null
      - engine_version          = "15.5" -> null
      - identifier              = "analytics" -> null
      - instance_class          = "db.m6g.large" -> null
      - multi_az                = true -> null
      - skip_final_snapshot     = true -> null
        # (37 unchanged attributes hidden)
    }
 
Plan: 0 to add, 0 to change, 1 to destroy.

What a plan cannot see

A declarative tool's view of reality is not your whole cloud account. It is a ledger, and like any ledger it only knows about entries somebody wrote in it. That ledger is the state file: a record of the objects the tool created, and what each of them looked like last time it checked. Anything not in the ledger is invisible. The port 22 drift above was caught only because the aws_security_group resource owns the group's complete rule list, so an extra rule shows up as a difference inside an object Terraform already tracks. Write those same rules as separate one-rule-per-resource blocks and a hand-added rule becomes a brand new object the ledger has never heard of. Every plan stays clean. Port 22 stays open.

So run both kinds of check. A scheduled terraform plan -detailed-exitcode tells you when something you manage stopped matching its description, which catches modification and deletion. An independent inventory sweep across the whole account, whether that is cloud-native config rules, a scheduled query, or a cloud security posture management product, tells you when something exists that nobody ever described, which catches creation. Modification and creation are different attacks, and an intruder only needs the one you are not watching for.

Go find the oldest hardening script you own. Run it twice on a throwaway copy of the box it normally runs against, diff every file it touched, then read the result back through the system rather than the file: sshd -T, iptables -S, systemctl is-enabled. Whatever came out different on the second run was never a description of an end state. It was a set of directions, written for a car you assumed was still parked outside.

Try this

Run sudo bash harden.sh on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: check mode rehearses, it does not prove. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related