What Infrastructure as Code is
Manage infrastructure like source code.
A recipe card and a meal are two different things. The meal gets eaten. The recipe survives, gets reviewed by other cooks, gets corrected when someone works out the oven runs hot, and can be handed to a person who has never set foot in your kitchen. Infrastructure as Code (IaC, the practice of writing down what your servers and cloud resources should look like in files, then letting a tool build reality to match) is the recipe card. The running server is the meal.
The old way was cooking without a recipe. Somebody logged into a web console, clicked through eleven screens, picked an instance size, opened a firewall port, attached a disk, and moved on. Six months later the disk fills up and nobody alive knows why it was sized that way. With IaC you write the desired state into a text file, put that file in Git (a version control system that records every change, who made it, and when), get it reviewed like any other code change, and run a tool that makes reality match the file.
For a security or operations person, that shift matters more than it does for anyone else on the team. Your job is knowing what exists, who changed it, and whether the change was safe. IaC turns all three of those from archaeology into a git log.
Desired State, Not Instructions
Here is the idea that trips up newcomers. A shell script is a list of steps: do this, then this, then this. Run it twice and you may get two of everything, or an error, or a mess. IaC tools work the other way around. You describe the end state you want, and the tool works out the steps to get there from wherever things currently stand. That property has a name: idempotence (running the same thing repeatedly leaves you in the same place, instead of stacking up side effects).
A thermostat is the everyday version. You do not tell a thermostat "turn on the heat for nine minutes." You tell it "21 degrees." It reads the room, compares that to your number, and acts on the difference. Set it to 21 a hundred times in a row and nothing bad happens. IaC tools are thermostats for infrastructure.
Here is a small, real example using Terraform (a widely used provisioning tool from HashiCorp). The file below says there should be a security group, which is a cloud firewall rule set attached to your servers, with exactly these properties. Read it as a noun, not a verb.
terraform {required_providers {aws = {source = "hashicorp/aws"version = "~> 5.0"}}}provider "aws" {region = "eu-west-1"}resource "aws_security_group" "web" {name = "web-sg"description = "HTTPS in, everything out"ingress {description = "HTTPS from the internet"from_port = 443to_port = 443protocol = "tcp"cidr_blocks = ["0.0.0.0/0"]}egress {from_port = 0to_port = 0protocol = "-1"cidr_blocks = ["0.0.0.0/0"]}tags = {Owner = "platform-team"}}
Nothing in that file says "create." It says what should be true: 443 inbound, everything outbound, and no other way in. The 0.0.0.0/0 is CIDR notation (Classless Inter-Domain Routing, the standard shorthand for a range of network addresses), and that particular range means every address on the internet. The protocol = "-1" on the egress rule is the cloud provider's way of saying "any protocol." The tool works out on its own whether making that true means creating a group, editing one, or doing nothing at all.
The Plan Step Is Your Best Security Control
Think of a builder walking you round the site with the drawings before anyone picks up a hammer. Every serious IaC tool has that walk-round built in, a dry-run mode that tells you what it is about to do before it does it. In Terraform the command is terraform plan. This is the most useful habit in the whole practice, and it is where a reviewer catches the change that would have opened SSH (Secure Shell, the remote login protocol that listens on port 22) to the entire internet.
terraform init -input=false && terraform plan -out=tfplan
Initializing the backend...Initializing provider plugins...- Finding hashicorp/aws versions matching "~> 5.0"...- Installing hashicorp/aws v5.62.0...- Installed hashicorp/aws v5.62.0 (signed by HashiCorp)Terraform has created a lock file .terraform.lock.hcl to record the providerselections it made above. Include this file in your version control repositoryso that Terraform can guarantee to make the same selections by default whenyou run "terraform init" in the future.Terraform has been successfully initialized!Terraform used the selected providers to generate the following executionplan. Resource actions are indicated with the following symbols:+ createTerraform will perform the following actions:# aws_security_group.web will be created+ resource "aws_security_group" "web" {+ arn = (known after apply)+ description = "HTTPS in, everything out"+ egress = [+ {+ cidr_blocks = [+ "0.0.0.0/0",]+ description = ""+ from_port = 0+ ipv6_cidr_blocks = []+ prefix_list_ids = []+ protocol = "-1"+ security_groups = []+ self = false+ to_port = 0},]+ id = (known after apply)+ ingress = [+ {+ cidr_blocks = [+ "0.0.0.0/0",]+ description = "HTTPS from the internet"+ from_port = 443+ ipv6_cidr_blocks = []+ prefix_list_ids = []+ protocol = "tcp"+ security_groups = []+ self = false+ to_port = 443},]+ name = "web-sg"+ name_prefix = (known after apply)+ owner_id = (known after apply)+ revoke_rules_on_delete = false+ tags = {+ "Owner" = "platform-team"}+ tags_all = {+ "Owner" = "platform-team"}+ vpc_id = (known after apply)}Plan: 1 to add, 0 to change, 0 to destroy.Saved the plan to: tfplanTo perform exactly these actions, run the following command to apply:terraform apply "tfplan"
Read the last line of that summary out loud on every change: "1 to add, 0 to change, 0 to destroy." A pull request (a proposed code change, opened for other people to review before it lands) that claims to add a tag but shows 1 to destroy is telling you something its description did not. Databases and disks get deleted this way. The -out=tfplan part saves the plan to a file so that terraform apply tfplan runs exactly what you reviewed, rather than a freshly recomputed plan that drifted while the review sat in a queue.
terraform show tfplan redacts sensitive values, but terraform show -json tfplan does not, and neither does the raw file. Never commit tfplan, terraform.tfstate, or *.tfstate.backup to Git, never paste plan output into a public ticket, and keep remote state in a backend with encryption at rest and tight access control. Add those patterns to .gitignore on day one, not after the first leak.Provisioning Versus Configuration
Two different jobs hide under the IaC umbrella, and mixing them up causes real confusion later. Back to the kitchen for a second, because the split is the same one a builder and a chef would recognise.
Provisioning is building the kitchen. Servers, networks, load balancers, managed databases, DNS records (Domain Name System, the address book that turns a name like example.net into an IP address), IAM roles (Identity and Access Management, the rules for who is allowed to do what). That is Terraform's job, or OpenTofu's, or CloudFormation's, or Pulumi's.
Configuration management is stocking that kitchen and setting the stove. Installing packages, writing config files, enabling services, applying a hardening baseline. That is Ansible's job, or Chef's, or Puppet's. Terraform builds the machine. Ansible then turns it into a web server.
Ansible describes desired state too. Its recipe file is called a playbook, its unit of work is a task with a name, a module, and some arguments, and every module checks the current state before it touches anything.
---- name: Baseline hardening for web hostshosts: webbecome: truetasks:- name: Install nginxansible.builtin.apt:name: nginxstate: presentupdate_cache: true- name: Drop in SSH hardening that wins on load orderansible.builtin.copy:dest: /etc/ssh/sshd_config.d/00-hardening.confowner: rootgroup: rootmode: '0644'content: |PasswordAuthentication noKbdInteractiveAuthentication novalidate: 'sshd -t -f %s'notify: Restart sshd- name: Ensure nginx is enabled and runningansible.builtin.systemd:name: nginxstate: startedenabled: truehandlers:- name: Restart sshdansible.builtin.systemd:name: sshstate: restarted
Three details in there are worth stealing. The validate: 'sshd -t -f %s' line runs a syntax check against the candidate file before Ansible moves it into place, so a typo cannot lock you and everyone else out of the fleet. The notify only fires the restart when the file actually changed, which means a no-op run does not bounce your SSH daemon for nothing (a handler is Ansible's word for a job that runs at the end, and only if something told it to). And KbdInteractiveAuthentication no closes the side door: turning off PasswordAuthentication alone can still leave password prompts reachable through the keyboard-interactive path.
Ansible has its own walk-round with the drawings. --check reports what would change, and --diff shows you the exact lines.
ansible-playbook -i inventory.ini harden.yml --check --diff
PLAY [Baseline hardening for web hosts] ****************************************TASK [Gathering Facts] *********************************************************ok: [web01.example.net]TASK [Install nginx] ***********************************************************ok: [web01.example.net]TASK [Drop in SSH hardening that wins on load order] ***************************--- before+++ after: /etc/ssh/sshd_config.d/00-hardening.conf@@ -0,0 +1,2 @@+PasswordAuthentication no+KbdInteractiveAuthentication nochanged: [web01.example.net]TASK [Ensure nginx is enabled and running] *************************************ok: [web01.example.net]RUNNING HANDLER [Restart sshd] *************************************************changed: [web01.example.net]PLAY RECAP *********************************************************************web01.example.net : ok=5 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Note that the handler ran even in check mode. Ansible notifies and executes handlers on a dry run, so the counts include it. The line that matters to you is the one above the recap: a changed on the hardening task, against a machine you believed was already hardened, is a finding. Either the baseline was never applied here, or somebody undid it.
Drift Is the Thing You Are Actually Defending Against
Drift is when reality stops matching the file. Somebody opened port 22 during an outage at 3am and never closed it. A vendor's setup script quietly edited a security group. An attacker with stolen console credentials added an inbound rule and walked out with your data. In all three cases the code in Git still says the door is locked. The door is not locked.
This is the real security payoff, and it goes well past "reproducible builds." Once your code is the declared truth, drift detection turns into an intrusion detection signal. You run the plan on a schedule against production and it answers exactly one question: has anything changed out from under us? A clean plan is a quiet night.
terraform plan -detailed-exitcode -refresh-only; echo "exit=$?"
aws_security_group.web: Refreshing state... [id=sg-0a1b2c3d4e5f67890]Note: Objects have changed outside of TerraformTerraform detected the following changes made outside of Terraform since thelast "terraform apply" which may have affected this plan:# aws_security_group.web has changed~ resource "aws_security_group" "web" {id = "sg-0a1b2c3d4e5f67890"~ ingress = [+ {+ cidr_blocks = [+ "0.0.0.0/0",]+ description = ""+ from_port = 22+ ipv6_cidr_blocks = []+ prefix_list_ids = []+ protocol = "tcp"+ security_groups = []+ self = false+ to_port = 22},# (1 unchanged element hidden)]name = "web-sg"# (8 unchanged attributes hidden)}This is a refresh-only plan, so Terraform will not take any actions to undothese. If you were expecting these changes then you can apply this plan torecord the updated values in the Terraform state without changing any remoteobjects.exit=2
There it is. Somebody opened SSH to the world. The -detailed-exitcode flag is what makes this usable in automation: exit code 0 means no changes, 1 means the command itself errored, 2 means there is a difference. An exit code is the number a command hands back when it finishes, and any scheduler can act on it. Wire a nightly job to alert on a 2 and you have a cheap, high-signal control that catches careless humans and intruders with the same trap. Then take the resource ID from that output and go find the who and the when in your cloud provider's audit log (CloudTrail on AWS, Activity Log on Azure, Cloud Audit Logs on Google Cloud).
terraform apply and slam the door shut. Resist it, at least for ten minutes. Applying overwrites the evidence, and if this was an intrusion it tells the attacker you noticed. Save the plan output somewhere safe, pull the audit-log entries for that resource ID, and confirm it was a tired engineer rather than a session that also minted access keys or an IAM role you have never heard of. Contain first, reconcile second. Reverting one rule takes thirty seconds. Reconstructing who added it after you erased the trail takes days.Verifying On the Box Itself
IaC tells you what should be true. Trust it, then go and check, the way you would rattle the handle after locking up. On a modern systemd Linux (systemd is the program that starts and supervises services on most distributions today, including Ubuntu 22.04 and Debian 12) the verification for that hardening play is two commands. First, confirm the running SSH daemon actually has the setting loaded, rather than confirming that some file on disk contains the line.
sudo sshd -T | grep -iE '^(password|kbdinteractive)authentication'
passwordauthentication nokbdinteractiveauthentication no
sshd -T prints the effective configuration after every Include has been resolved. Grepping sshd_config by hand can lie to you, because Debian and Ubuntu put Include /etc/ssh/sshd_config.d/*.conf near the top of that file, those included files are expanded in alphabetical order, and for most keywords SSH keeps the first value it obtains rather than the last. A drop-in file pulled in at line one therefore beats your careful edit two hundred lines further down. That is why the playbook writes 00-hardening.conf: the leading zeros put it first in the queue on purpose. One caveat on sshd -T itself, and it catches people out. It does not evaluate Match blocks unless you hand it connection details, so a Match Address or Match User rule that re-enables passwords for one network stays invisible in the output above. Ask about that case directly with sudo sshd -T -C user=deploy,host=web01,addr=10.0.4.7.
systemctl is-enabled nginx && systemctl show nginx -p ActiveState,SubState,MainPID
enabledActiveState=activeSubState=runningMainPID=1417
is-enabled answers "will this come back after a reboot," which is a completely different question from "is it running right now." A surprising number of 4am incidents trace back to a service somebody started by hand and never enabled, so it quietly vanished the next time the host restarted. IaC catches that, because enabled: true is written down in the file. One thing to watch on newer releases: Ubuntu 24.04 ships SSH under socket activation, where ssh.socket listens and spawns a fresh sshd per connection. Restarting ssh.service there is harmless but beside the point, and changing the listening port means editing the socket unit rather than sshd_config.
The Console Is for Looking, Not for Changing
Once you adopt IaC, the cardinal rule is that you stop making changes by hand. Use the web console to observe, read, and debug. Make every change through the code, through review, through the pipeline. A team half-committed to IaC, still clicking "only this once" during incidents, ends up with the worst of both worlds: the code is not trustworthy and neither is anyone's memory.
You can enforce that instead of hoping for it. Give the humans read-only roles in production and let only the pipeline's own identity hold write permissions. Now the console is physically incapable of changing anything, and the nightly drift check stops being a nag and becomes an alarm, because any exit code 2 in production means somebody used credentials that should not have been able to write in the first place.
What Breaks, and What To Do About It
Three failure modes show up in every team's first year. The first is damage to the state file. Terraform keeps a state file that maps the resources in your code to the real IDs in the cloud, and if two people apply at the same moment it can end up corrupted or with duplicated resources. Use a remote backend with state locking, which is a shared store plus the equivalent of a key hanging by the door, so the second person gets a clear "state is locked" error instead of a race. On AWS that is S3 (Simple Storage Service, the object store) with use_lockfile = true; the older DynamoDB locking table still works but has been deprecated since Terraform 1.11, and HCP Terraform (formerly Terraform Cloud) handles it for you.
The second is secrets in the repository. Variables feel like a convenient place to park a database password. They are not. A .tfvars file (the file that supplies values for your variables), the state file, and the saved plan all end up storing that value in the clear. Pull secrets from a dedicated store at apply time and keep the literal string out of Git entirely.
The third is the resource that gets replaced when you expected an edit. Some attributes cannot be changed in place, so the tool deletes the thing and builds a new one. The plan does tell you, on a line that is very easy to skim past: # forces replacement.
# aws_db_instance.main must be replaced-/+ resource "aws_db_instance" "main" {~ address = "main-postgres.abc123.eu-west-1.rds.amazonaws.com" -> (known after apply)~ availability_zone = "eu-west-1a" -> "eu-west-1b" # forces replacement~ endpoint = "main-postgres.abc123.eu-west-1.rds.amazonaws.com:5432" -> (known after apply)identifier = "main-postgres"~ resource_id = "db-XKPT4H2RQ7VZ3MNBGL5YU6WCEA" -> (known after apply)# (41 unchanged attributes hidden)}Plan: 1 to add, 0 to change, 1 to destroy.
That plan reads like a one-word edit and behaves like deleting your production database. Search every plan for forces replacement and must be replaced before you approve it, and put lifecycle { prevent_destroy = true } on the resources you cannot afford to lose, which makes the plan itself fail rather than politely offering to delete them.
terraform plan -detailed-exitcode -refresh-only against production returns exit code 2, showing a new inbound rule allowing 0.0.0.0/0 on port 3389 (Remote Desktop). What is the right first move?sudo sshd -T and reading the daemon's effective settings. What is the one blind spot it warns about in that command's output?sshd -T resolves every Include and prints the merged result, which is exactly why the lesson trusts it over grepping the file by hand.sudo sshd -T -C user=deploy,host=web01,addr=10.0.4.7.-T prints the effective configuration the daemon would apply at startup, which is more trustworthy than reading the file.terraform plan ends with: Plan: 1 to add, 0 to change, 1 to destroy. What should you do?0 to destroy, so this rationalises away the exact red flag you should catch.1 to destroy means a real resource is deleted on apply, which is how databases and disks quietly vanish.Start smaller than feels useful. Pick one thing you already own by hand, a DNS zone or a single security group, and write it down. Run terraform plan and read every line of the output until nothing in it surprises you. Then get the nightly drift check running before you widen the scope, because on an incident call, a small amount of infrastructure you can prove has not changed is worth more than a large amount you merely believe has not.
Try this
Run terraform init -input=false && terraform plan -out=tfplan on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.
Takeaway
The trap worth remembering here: a plan file is a secrets file. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.