The IaC & automation tool landscape

Provisioning, config, templating, policy, GitOps.

A house gets built by five different crews, and not one of them can do another crew's job. Groundworks digs the hole and pours the foundation. Electricians and plumbers fit out the shell once it is standing. The kitchen arrives as a flat-pack kit with holes pre-drilled at several shelf heights, then gets adjusted to fit this particular room. The building inspector checks the finished work against a code book before anyone signs it off. And the site foreman builds what is on the approved drawings, never what somebody shouted over the fence.

Infrastructure as code, usually shortened to IaC, means describing your servers, networks and cloud accounts in text files that a tool then applies for you. The tools split along those same five lines. Provisioning makes resources exist. Configuration management sets up what runs inside them. Templating shapes one definition for many environments. Policy and scanning check the work before it lands. GitOps delivery applies it from a Git repository (the shared, permanent history of every change your team has made) that nobody can go around. Tools compete inside a category. Across categories, they stack.

Care about the categories rather than the brand names, because each family holds a different set of keys and does a different amount of damage when it misfires. Blast radius is the plain word for that: how much breaks if this one thing goes wrong. Your provisioning tool can delete a production database. Your configuration management tool has root (the all-powerful administrator account on a Linux machine) on every host you own. Your GitOps controller can change what runs in a Kubernetes cluster, Kubernetes being the system that schedules and runs containers across a fleet of machines, at three in the morning with nobody logged in. Knowing which is which tells you where to look when something changed and nobody will admit to it.

Provisioning Makes the Resources Exist

Provisioning is the groundworks crew. It talks to a cloud API (Application Programming Interface, the machine-to-machine front door a cloud exposes so software can create and delete things without a human clicking around) and makes resources appear: virtual machines, networks, load balancers, databases, DNS records (Domain Name System, the phone book that turns names into addresses), and IAM roles (Identity and Access Management, the rules that decide who may do what).

Terraform is the one to learn first. You write what you want in HCL (HashiCorp Configuration Language, a readable format built for describing resources). Terraform loads a provider for each platform you touch, a provider being a plugin that knows one platform's API. It then keeps a state file, an index in JSON (JavaScript Object Notation, a machine-readable text format) that maps every resource in your code to the real object's identifier in the cloud. Two things follow. The state file is a target in its own right, because it often stores passwords and keys in clear text, so guard the bucket that holds it like a password vault. And because Terraform knows what already exists, it can compare your code against reality, work out the difference, and show you the result before touching anything.

resource "aws_security_group" "bastion" {
  name   = "bastion-sg"
  vpc_id = var.vpc_id
 
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# What would this actually do to the real account?
terraform plan -out=tfplan

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
 
Terraform will perform the following actions:
 
  # aws_security_group.bastion will be created
  + resource "aws_security_group" "bastion" {
      + arn                    = (known after apply)
      + description            = "Managed by Terraform"
      + egress                 = (known after apply)
      + id                     = (known after apply)
      + ingress                = [
          + {
              + cidr_blocks      = [
                  + "0.0.0.0/0",
                ]
              + description      = ""
              + from_port        = 22
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 22
            },
        ]
      + name                   = "bastion-sg"
      + name_prefix            = (known after apply)
      + owner_id               = (known after apply)
      + revoke_rules_on_delete = false
      + tags_all               = (known after apply)
      + vpc_id                 = "vpc-0f1a2b3c4d5e6f7a8"
    }
 
Plan: 1 to add, 0 to change, 0 to destroy.
 
Saved the plan to: tfplan
 
To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

Read that output as a security review, because that is what it is. The Plan line, 1 to add and 0 to destroy, is the blast radius in one sentence. The ingress block spells out from_port 22 with cidr_blocks of 0.0.0.0/0. That is SSH (Secure Shell, the encrypted remote-login protocol) open to every address on the internet, because 0.0.0.0/0 is CIDR notation (Classless Inter-Domain Routing, the shorthand for a range of addresses) meaning all of them. Saving the plan with -out earns its keep too. Your CI system (continuous integration, the automated pipeline that tests and ships changes) applies that exact file, so what runs is what was reviewed, not a fresh plan computed after somebody pushed one more commit.

The rest of the family has the same shape. OpenTofu is the fork that appeared after HashiCorp moved Terraform to the Business Source License in August 2023; it stayed on the older open-source terms, the Mozilla Public License 2.0, and speaks the same HCL with the same providers, so every concept carries over. Pulumi lets you write the same declarations in TypeScript, Python, Go or C#. CloudFormation is Amazon's own version, YAML or JSON, organised into stacks. Crossplane moves the whole job inside Kubernetes, where cloud resources become cluster objects that a controller keeps pushing back into shape. Whichever you run holds the strongest credentials in the building, because anything that can create can also destroy. Scope that role to what the repository actually manages. Then watch for applies that did not come from your pipeline: the AWS provider stamps a Terraform user agent on every API call it makes, so CloudTrail (Amazon's log of API calls in the account) shows it in the userAgent field next to a source address. Terraform traffic from a home broadband address at 2am is worth asking about out loud. Treat it as a lead and not as proof, though, because the client chooses that string and anyone can change it.

Configuration Management Owns the Inside of the Box

Provisioning hands you an empty room with power at the walls. Configuration management is the electrician and the plumber. It installs packages, writes config files, sets permissions, switches services on, and restarts them when a file underneath changes.

Ansible is agentless. A control node opens an SSH session to each host, copies a small bundle of Python code into a temporary directory, runs it, reads the result and deletes it, so nothing stays behind. The host does need a Python interpreter already present, which every mainstream Linux ships. Chef and Puppet work the other way round, with an agent on every host: a background program that wakes on a timer and pulls its desired state from a central server. Salt does either. The difference that matters is exposure, not elegance. Agentless means one control node holding SSH keys to your whole estate. Agent-based means a long-running daemon (a program that sits in the background waiting for work) on every host, plus a server that every one of those hosts trusts. Either way, that machine is the prize, and it deserves the protection you would give a domain controller, the server that holds the keys to a Windows network.

- name: SSH baseline
  hosts: web
  become: true
  tasks:
    - name: Disable direct root login
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^#?PermitRootLogin'
        line: 'PermitRootLogin no'
        validate: /usr/sbin/sshd -t -f %s
      notify: Restart ssh
 
  handlers:
    - name: Restart ssh
      ansible.builtin.service:
        name: ssh
        state: restarted

# Dry run: show me the exact lines you would change, change nothing
ansible-playbook -i inventory.ini ops/baseline.yml --check --diff

PLAY [SSH baseline] ************************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web-01]
 
TASK [Disable direct root login] ***********************************************
--- before: /etc/ssh/sshd_config (content)
+++ after: /etc/ssh/sshd_config (content)
@@ -30,7 +30,7 @@
 # Authentication:
 
 #LoginGraceTime 2m
-#PermitRootLogin prohibit-password
+PermitRootLogin no
 #StrictModes yes
 #MaxAuthTries 6
 #MaxSessions 10
 
changed: [web-01]
 
RUNNING HANDLER [Restart ssh] **************************************************
changed: [web-01]
 
PLAY RECAP *********************************************************************
web-01                     : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Two details in that run earn their keep. The validate line makes Ansible hand the candidate file to sshd's own parser before it replaces the live one, and %s is where Ansible substitutes the path of the temporary copy. A typo fails the task instead of locking every engineer out of the host. The second detail is --check --diff, which prints the exact before-and-after lines while writing nothing. That is the gap between a change you can review and a change you have to take on faith.

One thing the playbook does not cover, and it bites people. On Debian 12 and Ubuntu 22.04 the top of /etc/ssh/sshd_config is an Include of /etc/ssh/sshd_config.d/*.conf, and for most settings sshd keeps the first value it reads. A drop-in file left there by your cloud image can quietly outrank the line you edited. Look in that directory before you call the host hardened.

Now the trick most teams miss. Run that same check-mode play on a timer against machines you hardened weeks ago. A run that reports changed=0 means nothing on those boxes has wandered off the baseline, and the handler never fires. The moment it reports changed=1 on a host nobody touched, something edited sshd_config by hand, and you have a hostname, a filename, a diff and a timestamp to chase. Your configuration tool is now a drift detector as well as a build tool, running on kit you already own with rules you already trust.

Keep the limits of that in view. Check mode skips most command and shell tasks outright, because Ansible cannot know what an arbitrary binary would have done. Worse, any task whose input depends on an earlier task's result can report the wrong answer, since that earlier change never actually happened. A clean check run is strong evidence. It is not proof, and it should never be the only gate in front of production.

Templating Shapes One Definition for Many Environments

The flat-pack shelf ships as one kit with several sets of holes already drilled, so the same box fits four different rooms. Kubernetes templating is that kit. You keep one description of your application and adjust it per environment, instead of maintaining four near-identical copies of the same YAML (the indented plain-text format Kubernetes reads) that quietly drift apart inside a month. Helm packages your manifests, a manifest being one file that describes one object you want in the cluster, into a chart with a values file for the parts that change, then renders them through Go templates. Kustomize refuses templating altogether: a base directory of ordinary manifests plus overlays that patch it. It already ships inside kubectl, as kubectl kustomize and kubectl apply -k.

# Render locally, then look at the fields that decide security posture
helm template checkout ./chart -f values/prod.yaml \
  | grep -E '^kind:|replicas:|image:|runAsNonRoot:'

kind: ServiceAccount
kind: Service
kind: Deployment
  replicas: 4
          image: "registry.example.com/checkout:1.8.2"
            runAsNonRoot: true

helm template renders on your machine and never contacts the cluster unless you ask it to with --validate, which is exactly where its security value comes from. A values file is one of the easiest hiding places in a repository. Flipping securityContext.privileged to true, or repointing image.repository at a registry you do not control, is a one-line diff that looks about as interesting as a version bump. Render first, scan the rendered output, then review. What a human reads should be what the cluster is actually going to get.

Policy and Scanning Are the Inspector

The inspector turns up with a code book and checks the work against it. Two kinds of inspector exist here, and mature teams hire both. Scanners arrive with the code book already written and need no setup: Checkov, Trivy (which took over tfsec's misconfiguration rules when that project wound down), Terrascan, KICS. They know the mistakes everybody makes, across Terraform, CloudFormation, Helm charts and Kubernetes YAML. Policy engines are the other kind, where you write the rules yourself. OPA (Open Policy Agent, a general-purpose engine for deciding whether something is allowed) uses a language called Rego, and Conftest is the small wrapper that points OPA at files on disk. HashiCorp's Sentinel does the same job inside HCP Terraform, formerly called Terraform Cloud, and Terraform Enterprise. Scanners catch what everyone gets wrong. Policy engines catch what your organisation specifically forbids.

checkov -f infra/main.tf --compact --quiet

terraform scan results:
 
Passed checks: 3, Failed checks: 2, Skipped checks: 0
 
Check: CKV_AWS_24: "Ensure no security groups allow ingress from 0.0.0.0:0 to port 22"
	FAILED for resource: aws_security_group.bastion
	File: /infra/main.tf:1-11
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/networking-policies/networking-1-port-security
 
Check: CKV_AWS_23: "Ensure every security group and rule has a description"
	FAILED for resource: aws_security_group.bastion
	File: /infra/main.tf:1-11

Checkov exits with a non-zero status when a check fails, and a passing run exits 0. That exit code is the thing that actually stops the pipeline. Look at the second finding as well. CKV_AWS_23 wants a description on every rule, which reads like paperwork right up until you are the person at 2am working out why port 22 is open and which ticket asked for it.

There is a trap in scanning source files, and it is why experienced teams do not stop here. Checkov does resolve the variables it can see, including defaults and a terraform.tfvars sitting beside the code. What it cannot know is what your pipeline passes at run time with -var-file=prod.tfvars, what a data source will return (a data source is a read-only lookup that fetches a real value from the cloud while the plan runs), or any attribute the cloud computes during the apply. Those are precisely the values that decide whether a rule is broken. Pointing the scanner at the right file with --var-file helps. Gating on the plan instead of the source fixes it properly, because a saved plan already has variables resolved, modules expanded and every knowable attribute filled in.

package main
 
import rego.v1
 
deny contains msg if {
	some rc in input.resource_changes
	rc.type == "aws_security_group"
	"create" in rc.change.actions
	some rule in rc.change.after.ingress
	rule.from_port <= 22
	rule.to_port >= 22
	"0.0.0.0/0" in rule.cidr_blocks
	msg := sprintf("%s opens port 22 to the whole internet", [rc.address])
}

# Turn the saved plan into JSON, then judge the plan, not the source
terraform show -json tfplan > tfplan.json
conftest test --policy policy/ tfplan.json

FAIL - tfplan.json - main - aws_security_group.bastion opens port 22 to the whole internet
 
1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions

That JSON holds resource_changes, a list of every resource the apply would touch, each with the actions it would take and its fully resolved after state. Rules written against it see real values instead of placeholders. It is also why one bundle of policy can run in your pipeline and again inside whatever applies your changes, with nothing rewritten: both are reading the same machine-readable description of what is about to happen to production.

GitOps Puts One Door on Production

The site foreman builds what is on the approved drawings and nothing else. GitOps makes that rule mechanical. The repository is the only accepted source of change, and a machine does the applying. Atlantis does this for Terraform. It watches pull requests (a pull request, or PR, is a proposed change that other people review before it merges), runs plan and apply on its own server when someone comments on the PR, and posts the output back into the thread. The cloud credentials live on that server, so nobody needs them on a laptop. Argo CD and Flux do the same for Kubernetes, with a controller that watches Git and keeps steering the cluster back toward what the repository says. Flux re-applies the desired state on every reconcile interval, so a hand edit gets overwritten on the next pass. Argo CD only does that when you switch on self-heal in the sync policy; leave it off and Argo marks the application OutOfSync and waits for a human. Find out which of those two you have before you rely on it.

flux get kustomizations --all-namespaces

NAMESPACE  	NAME       	REVISION                                        	SUSPENDED	READY	MESSAGE
flux-system	apps       	main@sha1:9f3c2ad1c5b7e4d80a6f1b2c3d4e5f60718293a4	False    	True 	Applied revision: main@sha1:9f3c2ad1c5b7e4d80a6f1b2c3d4e5f60718293a4
flux-system	flux-system	main@sha1:9f3c2ad1c5b7e4d80a6f1b2c3d4e5f60718293a4	False    	True 	Applied revision: main@sha1:9f3c2ad1c5b7e4d80a6f1b2c3d4e5f60718293a4

That REVISION column is the exact commit running in the cluster right now, which answers the question every audit and every incident opens with: what is deployed, and who approved it? The hash points straight at a diff, a reviewer and an author. When Flux corrects a hand edit, it also records a Kubernetes event saying so, and your log pipeline can alert on that. Drift correction and drift detection turn out to be one feature wearing two hats.

Reading a Stack You Did Not Build

Sooner or later you inherit a repository nobody on your team wrote. The quickest way in is to ask which categories are present, because filenames are honest here. Every tool insists on its own.

find . -maxdepth 4 \( -name '*.tf' -o -name 'Chart.yaml' -o -name 'kustomization.yaml' \
  -o -name '*.rego' -o -name 'atlantis.yaml' -o -name 'site.yml' \) \
  -not -path './.terraform/*' | sort

./ansible/site.yml
./atlantis.yaml
./charts/checkout/Chart.yaml
./infra/main.tf
./infra/variables.tf
./k8s/overlays/prod/kustomization.yaml
./policy/security_groups.rego

Seven files, five categories, and now you know every door into production: Terraform under infra/, applied by Atlantis from pull requests; Ansible configuring the hosts; a Helm chart and a Kustomize overlay shaping what runs in the cluster; and Rego policy that is supposed to be gating all of it. The follow-up question is the one that finds real gaps. Does every one of those paths actually pass through the policy step, or is one of them a way in that skips the inspector?

Put an ownership line in the README of every directory that holds infrastructure code, naming the one tool allowed to apply it. Then make a clean plan part of your morning. Run terraform plan -detailed-exitcode on a schedule in CI: it exits 0 when nothing has changed, 1 on an error, and 2 when a difference exists. A scheduled plan that starts returning 2 with nobody having pushed a commit means the real infrastructure moved without the code, and that is either drift you need to explain or a person you need to find.

Try this

Run terraform plan -out=tfplan on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: two tools, one resource, endless fight. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related