>_SecOpsLog_
CoursesInfrastructure as Code & automationCloudFormation & Crossplane

CloudFormation & Crossplane

Cloud-native and Kubernetes-native IaC.

Intermediate12 min · lesson 14 of 23

AWS CloudFormation (Amazon Web Services' own infrastructure-as-code service) works like a builder's merchant that keeps your paperwork for you. You hand over a written order. The shop builds everything on it, files the order under one name, and keeps the only copy. You go home with nothing to file. The written order is a template, written in YAML or JSON (two plain-text formats for describing structured data). The filed order is a stack. AWS itself records what got built and what each piece looks like right now, so there is no state file to park in an S3 bucket (Amazon's object storage service), no lock table to configure, no backend to encrypt and guard. The ledger never leaves AWS.

That closeness is the whole trade. CloudFormation plugs straight into AWS machinery you would otherwise build yourself: rollback that fires automatically when an update fails, stack policies that refuse updates to protected resources, termination protection that blocks stack deletion, Service Catalog, plus SAM (Serverless Application Model, a shorthand syntax that CloudFormation expands at deploy time) and the CDK (Cloud Development Kit, which turns TypeScript or Python into a template). The cost is that it speaks only AWS. Coverage is not instant either, despite being first-party. New services and new resource properties routinely land in CloudFormation weeks or months after the API itself, which is one reason plenty of AWS-only shops still keep a second tool around. You will meet CloudFormation constantly regardless, because AWS documentation, quick-starts and one-click deploy buttons all hand you a template and expect you to read it.

template.yaml
AWSTemplateFormatVersion: '2010-09-09'
 
Parameters:
  AdminCidr:
    Type: String
    Default: 10.0.0.0/16            # CIDR: shorthand for a range of IP addresses
 
Resources:
  WebServer:
    Type: AWS::EC2::Instance        # AWS-native resource types
    Properties:
      ImageId: ami-0ff8a91507f77f867
      InstanceType: t3.micro
      SecurityGroupIds: [!Ref WebSG]
      Tags:
        - Key: Name
          Value: web-server
 
  WebSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: web tier
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 22
          ToPort: 22
          CidrIp: !Ref AdminCidr    # SSH (remote login) from the office range only
 
  WebRole:
    Type: AWS::IAM::Role            # creating IAM changes how you must deploy
    Properties:
      RoleName: web-server-role     # a NAMED IAM resource -> CAPABILITY_NAMED_IAM
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: { Service: ec2.amazonaws.com }
            Action: sts:AssumeRole
 
# deployed as a "stack": AWS keeps the record, so there is no state file to manage

The stack is the unit of everything. You create, update, delete and roll back at stack level, and CloudFormation stamps every resource that supports tagging with three of its own tags: aws:cloudformation:stack-name, aws:cloudformation:stack-id and aws:cloudformation:logical-id. That is how you answer "what built this thing?" six months later, from the console or a tag report, without asking around.

Secrets need care. A parameter marked NoEcho: true comes back masked as asterisks in describe-stacks output and in stack events, which stops the obvious leak. It does not mask a value you paste into the template's Metadata section or into a resource's Metadata attribute, and it does nothing at all about the template file itself, which is usually sitting in Git for anyone with read access. Keep the secret out of the template and let CloudFormation fetch it at deploy time with a dynamic reference.

template.yaml (secrets)
  Database:
    Type: AWS::RDS::DBInstance      # RDS: Relational Database Service, managed databases
    Properties:
      Engine: postgres
      DBInstanceClass: db.t3.micro
      AllocatedStorage: '20'
      MasterUsername: appuser
      # CloudFormation resolves this at deploy time: no secret in Git, none in the template
      MasterUserPassword: '{{resolve:secretsmanager:prod/web/db:SecretString:password}}'
      # an SSM Parameter Store SecureString works too, pinned to a parameter version:
      #   '{{resolve:ssm-secure:/prod/web/db-password:3}}'

The Two Flags That Decide Who Holds the Power

terminal
$ aws cloudformation deploy --template-file template.yaml --stack-name web-prod
output
 
Waiting for changeset to be created..
 
An error occurred (InsufficientCapabilitiesException) when calling the CreateChangeSet
operation: Requires capabilities : [CAPABILITY_NAMED_IAM]

That refusal is a safety catch, not red tape. Think of it as the till asking for a manager's key. IAM (Identity and Access Management) is the service that decides who can do what inside an AWS account, so any template that creates or changes IAM has to be deployed with an explicit acknowledgement: CAPABILITY_IAM for roles and policies CloudFormation names itself, CAPABILITY_NAMED_IAM when the template hard-codes the name, and CAPABILITY_AUTO_EXPAND when the template uses macros or transforms that can generate more template at deploy time (every SAM template needs that last one). When you are reviewing a template somebody else wrote, that flag is your cue to stop and read the IAM section line by line. A role with AdministratorAccess and a trust policy pointing at an account number nobody recognizes looks like ordinary YAML until you actually read it.

terminal
$ aws cloudformation deploy \
    --template-file template.yaml \
    --stack-name web-prod \
    --capabilities CAPABILITY_NAMED_IAM \
    --role-arn arn:aws:iam::123456789012:role/cfn-web-prod-exec
output
 
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - web-prod

The second flag, --role-arn, is the one most teams skip, and it decides who really holds the power. Without it, CloudFormation makes every API call as you. Anyone who can trigger a stack update borrows the full reach of whatever identity runs the pipeline. With it, the stack acts as a role you scoped on purpose: this stack can build the handful of things it owns and nothing else. That ARN (Amazon Resource Name, the unique identifier for any AWS thing) sticks to the stack for later updates too, so the limit holds after you walk away. Setting it requires iam:PassRole on that role, which is a useful gate in itself. It tidies the audit trail as well. In CloudTrail (the AWS record of every API call), calls made by the stack carry the execution role as the actor and invokedBy set to cloudformation.amazonaws.com, so you can tell automation apart from a human at a keyboard.

terminal
# jq is a command-line filter for JSON; CloudTrailEvent arrives as a JSON string
$ aws cloudtrail lookup-events \
    --lookup-attributes AttributeKey=EventName,AttributeValue=CreateRole \
    --max-results 1 --query 'Events[0].CloudTrailEvent' --output text \
  | jq '.userIdentity.invokedBy, .userIdentity.sessionContext.sessionIssuer.arn'
output
"cloudformation.amazonaws.com"
"arn:aws:iam::123456789012:role/cfn-web-prod-exec"

Change Sets Are the Plan You Get to Read

CloudFormation's version of terraform plan is the change set. It is the written quote a builder hands you before anyone picks up a hammer: here is what I will add, what I will alter, and what I will knock down. The deploy command builds one every time and runs it straight away, unless you tell it to stop and let you read.

terminal
$ aws cloudformation deploy --template-file template.yaml --stack-name web-prod \
    --capabilities CAPABILITY_NAMED_IAM --no-execute-changeset
output
 
Waiting for changeset to be created..
Changeset created successfully. Run the following command to review changes:
aws cloudformation describe-change-set --change-set-name arn:aws:cloudformation:us-east-1:123456789012:changeSet/awscli-cloudformation-package-deploy-1784030112/9c1f5b6a-0d3e-4c77-9a21-5f0a2b8e77c4
terminal
$ aws cloudformation describe-change-set \
    --change-set-name arn:aws:cloudformation:us-east-1:123456789012:changeSet/awscli-cloudformation-package-deploy-1784030112/9c1f5b6a-0d3e-4c77-9a21-5f0a2b8e77c4 \
    --query 'Changes[].ResourceChange.[Action,LogicalResourceId,ResourceType,Replacement]' \
    --output text
output
Modify   WebServer   AWS::EC2::Instance        True
Add      WebRole     AWS::IAM::Role            None
Modify   WebSG       AWS::EC2::SecurityGroup   False

Replacement is the column that matters. True means CloudFormation cannot change that property in place, so it builds a replacement resource and deletes the old one. For an EC2 instance (Elastic Compute Cloud, an AWS virtual machine) that means a new machine, a new root disk, a new private IP address, and anything written to the old disk is gone. False means an in-place update. Conditional means it depends on values only known once the update actually runs, and you should read it as True until somebody proves otherwise. On a database, or any volume holding data people care about, that one word is the difference between a rolling change and an outage. Make it part of review: no stack update touching a stateful resource gets approved until a human has read its change set.

Drift Detection Finds the Console Edits

Here is the scenario that pays for this whole section. Someone with console access widens the security group to debug a Friday evening incident, sets the source to 0.0.0.0/0, and forgets. Your template still says 10.0.0.0/16. Your Git history looks spotless. Port 22 is open to the internet. CloudFormation will not tell you, because nothing is watching. You have to ask.

terminal
$ aws cloudformation detect-stack-drift --stack-name web-prod
output
{
    "StackDriftDetectionId": "7f13a0c2-6ba5-11f0-9c1b-0a7e2b45c9d3"
}
terminal
$ aws cloudformation describe-stack-drift-detection-status \
    --stack-drift-detection-id 7f13a0c2-6ba5-11f0-9c1b-0a7e2b45c9d3
output
{
    "StackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/web-prod/2c8b41e0-6b9f-11f0-a1d4-0e5c2f9a11bb",
    "StackDriftDetectionId": "7f13a0c2-6ba5-11f0-9c1b-0a7e2b45c9d3",
    "StackDriftStatus": "DRIFTED",
    "DetectionStatus": "DETECTION_COMPLETE",
    "DriftedStackResourceCount": 1,
    "Timestamp": "2026-07-21T09:14:22.104000+00:00"
}
terminal
$ aws cloudformation describe-stack-resource-drifts --stack-name web-prod \
    --stack-resource-drift-status-filters MODIFIED \
    --query 'StackResourceDrifts[].PropertyDifferences[]'
output
[
    {
        "PropertyPath": "/SecurityGroupIngress/0/CidrIp",
        "ExpectedValue": "10.0.0.0/16",
        "ActualValue": "0.0.0.0/0",
        "DifferenceType": "NOT_EQUAL"
    }
]
Drift detection reports; it never repairs
Finding the drift is not fixing it. CloudFormation works out an update by comparing your new template against the last template it stored, never against the live resource, so running aws cloudformation deploy again with the same file answers "No changes to deploy. Stack web-prod is up to date" and leaves port 22 open to the world. To close it you have to hand CloudFormation a genuine change to that property: edit the rule in the template, or change the parameter feeding it. Better, stop the drift at the source with an IAM policy or a service control policy (an organization-wide rule that caps what any account can do) denying humans ec2:AuthorizeSecurityGroupIngress in production, and run detection on a schedule with the AWS Config managed rule cloudformation-stack-drift-detection-check so nobody has to remember. One more limit: detection only covers the resource types and properties AWS supports for it, so read a clean result as "nothing found", not "nothing changed".

CloudFormation or Terraform on AWS

On AWS the choice is a real one. CloudFormation keeps the record for you, rolls back a failed update on its own, and needs nothing installed beyond the AWS CLI (command line interface). Terraform is cloud-agnostic, so one tool and one workflow covers AWS, Azure, Google Cloud, Kubernetes, Datadog and hundreds of other providers; its module ecosystem is far bigger; support for brand-new AWS features often arrives sooner than CloudFormation ships it; and most engineers find HCL (HashiCorp Configuration Language) and plan output easier to read than a change set. The price is a state file you have to store, lock, encrypt and guard, because that file contains every value your infrastructure knows. Teams living entirely inside AWS often stay with CloudFormation, frequently through the CDK, which generates the template from real code. Teams that want one workflow everywhere pick Terraform or OpenTofu (the open-source fork). Both are declarative, both compute a diff before acting, and the habits carry over either way.

Crossplane Turns the Cluster Into the Control Plane

Terraform and CloudFormation are contractors you phone. They turn up, do the work written on the order, and drive off. Crossplane is the building superintendent who lives on site with a copy of the blueprint and walks the floors all day. You install Crossplane into a Kubernetes cluster, add a provider package for the platform you want, and the cluster's API grows new resource types for cloud things. A bucket, a database, a VPC (Virtual Private Cloud, your own private network inside AWS) become Kubernetes objects you create with kubectl apply, and a controller keeps checking them against the real cloud.

There is no state file for you to store or secure. The desired state is the object sitting in the Kubernetes API (etcd, the database behind that API, is the ledger now) and the actual state is whatever the cloud reports when the controller looks. It looks on a timer. That poll interval is a provider flag, commonly somewhere between one and ten minutes, so a bucket someone deletes by hand comes back on its own in minutes rather than waiting to be noticed on the next apply. That is the real break from run-on-demand tools: enforcement never stops. It also means everything you already built around Kubernetes now applies to cloud infrastructure with no new tooling, including GitOps (keeping desired state in Git and letting a controller apply it), admission control, audit logging, and RBAC (role-based access control, the rules for who may do what to which objects). Platform teams push it further with Compositions and Composite Resource Definitions, so a developer asks for a "PostgresInstance" and your house rules decide what that actually builds.

bucket.yaml
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
  name: acme-logs-prod
spec:
  forProvider:
    region: us-east-1
  providerConfigRef:
    name: default            # which cloud credentials this object provisions with
  deletionPolicy: Orphan     # kubectl delete removes the K8s object, NOT the bucket
terminal
$ kubectl apply -f bucket.yaml
$ kubectl get managed
output
bucket.s3.aws.upbound.io/acme-logs-prod created
 
NAME                                      READY   SYNCED   EXTERNAL-NAME    AGE
bucket.s3.aws.upbound.io/acme-logs-prod   True    True     acme-logs-prod   47s
terminal
# delete it out from under Crossplane, the way an attacker or a tired human would
# (this provider is configured to poll every minute)
$ aws s3 rb s3://acme-logs-prod --force
$ kubectl get bucket acme-logs-prod -w
output
remove_bucket: acme-logs-prod
NAME             READY   SYNCED   EXTERNAL-NAME    AGE
acme-logs-prod   True    True     acme-logs-prod   3m12s
acme-logs-prod   False   True     acme-logs-prod   3m41s
acme-logs-prod   True    True     acme-logs-prod   4m02s
Where the record lives, and who fixes drift
CloudFormation
stack in AWS
AWS keeps the record
runs on deploy
change set, then exits
drift on demand
reports, never repairs
Terraform / OpenTofu
state file
a backend you secure yourself
runs on apply
plan, then exits
drift on next plan
apply puts it back
Crossplane
objects in etcd
the cluster is the record
never exits
a controller, not a command
drift on every poll
puts itself back
Same declarative idea in all three. What changes is who holds the record, and how long drift gets to live.

In Crossplane, Kubernetes RBAC Is Cloud IAM

This is the part teams miss. The provider holds credentials that can create and destroy real infrastructure, and it uses them on behalf of whoever managed to create the object. So "who can make an S3 bucket in this account" stops being an AWS IAM question and becomes a Kubernetes RBAC question. The cluster's permission model is now a front door into your cloud account. Check it the way you would check any other lock.

terminal
$ kubectl auth can-i create buckets.s3.aws.upbound.io \
    --as=system:serviceaccount:team-web:deployer
$ kubectl get providerconfig default -o jsonpath='{.spec.credentials.source}{"\n"}'
$ kubectl get providerconfig default \
    -o jsonpath='{.spec.credentials.secretRef.namespace}/{.spec.credentials.secretRef.name}{"\n"}'
output
yes
Secret
crossplane-system/aws-creds

Two findings in three commands. A service account in a team namespace can create cloud storage, which is either the self-service you designed on purpose or an accident of a ClusterRole written too broadly. And the provider authenticates from a static Secret, which means long-lived AWS access keys are sitting in etcd. On EKS (Elastic Kubernetes Service), switch that to IRSA (IAM Roles for Service Accounts, which mints short-lived credentials for the pod) or to EKS Pod Identity, so there are no permanent keys left to steal. From then on, treat read access to the crossplane-system namespace as read access to your cloud account. Those provider Secrets are what an attacker who lands a shell in the cluster goes looking for, long before they care about your pods.

kubectl delete can destroy real infrastructure
A managed resource defaults to deletionPolicy: Delete, so removing the Kubernetes object deletes the actual cloud resource. A stray kubectl delete -f, an over-eager GitOps prune, or a namespace teardown can take a production database with it, and no amount of Git history brings the data back. Set deletionPolicy: Orphan on anything stateful (or the newer spec.managementPolicies, leaving Delete out of the list), and when you need to change a resource by hand, annotate it crossplane.io/paused: "true" so the controller stops reconciling instead of fighting your change back out.

Pick the One Your Team Can Actually Operate

These tools are less better-or-worse than fit-for-context. CloudFormation suits deep AWS shops that want native integration, automatic rollback and nothing of their own to run. Terraform and OpenTofu suit teams who want one workflow across every provider. Crossplane suits platform teams already running Kubernetes who want to hand developers a self-service API with the house rules baked in. The failure mode is picking by fashion. Adopt Crossplane because it is the interesting choice, and one day your production database depends on a stuck controller in a cluster nobody on the rota can debug. Fight CloudFormation in a pure-AWS shop, and you burn weeks hardening a state backend you never needed. The principles are identical in all three, so pick the one whose failure modes your team can handle at 3am.

Quick check
01detect-stack-drift reports your security group as MODIFIED: someone widened it in the console. You re-run aws cloudformation deploy with the unchanged template. What happens to the drift?
Correct — detection only reports, and only an update that genuinely changes the property closes it.
Incorrect — It never auto-corrects drift. Re-enforcing on every pass is Crossplane's model, not CloudFormation's.
Incorrect — No such gate exists. Drift is invisible to the deploy path.
Incorrect — That state comes from a rollback that could not finish, not from drift.
02In a CloudFormation change set, the Replacement column for a resource reads 'Conditional'. How should you treat that value when reviewing an update to a production database?
Incorrect — Conditional is not False; it means the outcome is not yet decided, which is the opposite of a guaranteed in-place update.
Incorrect — the change set still executes; Conditional is a warning about uncertainty, not a gate that blocks it.
Incorrect — Replacement applies to any resource and matters most on stateful ones like databases.
Correct — a runtime-dependent replacement can build a new resource and delete the old one, destroying the database's data, so you review it as a possible replace-and-delete.
03Your platform team manages a production database as a Crossplane managed resource. During a namespace cleanup someone runs kubectl delete -f on the directory and the real cloud database is deleted, data and all. Which configuration would have prevented this?
Incorrect — a read-only credential would break every provision and reconcile, and is not the mechanism the lesson uses.
Incorrect — the managed resource defaults to deletionPolicy: Delete, so Crossplane itself issues the delete regardless of a console setting you never wired in.
Correct — Orphan makes kubectl delete remove only the Kubernetes object and leave the database standing.
Incorrect — the namespace does not protect the resource; the delete policy on the object decides whether the cloud database survives.

Two checks worth running this week. If you deploy with CloudFormation, list the stacks with no execution role, because those are the ones borrowing whatever identity happens to run them. If you run Crossplane, ask what a developer's service account can already do to your cloud account through the cluster.

terminal
$ aws cloudformation describe-stacks \
    --query 'Stacks[?RoleARN==`null`].StackName' --output text
$ kubectl auth can-i --list --as=system:serviceaccount:team-web:deployer \
  | grep upbound.io
output
logging-prod   ci-artifacts   vpc-shared
buckets.s3.aws.upbound.io    []    []    [create delete get list patch update watch]

Try this

Run aws cloudformation deploy --template-file template.yaml --stack-name web-prod on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: drift detection reports; it never repairs. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

OpenTofu & PulumiHelm: the Kubernetes package…

Related

What Terraform is & how it worksTerraformState files are secretsTerraform security and complianceWhy Terragrunt existsTerragruntWhat OpenTofu is & why it forkedOpenTofu