Change sets & safe updates

Preview a change before it happens.

Your security team files a one-line ticket: encrypt the payments database at rest. Sounds like a five-minute job. AWS CloudFormation (Amazon Web Services' service for building cloud infrastructure from a written blueprint instead of clicking through a console) keeps your whole setup described in one plain-text file called a template. You open it, find the database, set a single property, StorageEncrypted: true, and push. Do that blind and CloudFormation does something the ticket never asked for. It deletes your running database and builds a fresh, empty one in its place. Encryption on an RDS instance (Relational Database Service, Amazon's managed database engine) can only be turned on at the moment the instance is created, never flipped on later, so the only way CloudFormation can honor your edit is to throw the old database away and make a new one. Every payment record goes with it.

A change set is the safety check that catches this before it happens. Print preview does the same thing for a document: it shows you the forty-page misprint before you send it to the printer and waste the toner. Or picture a demolition crew that walks the building first and writes up which walls come down, flagging the one holding up the roof before anyone swings a hammer. A change set is that written preview for your infrastructure. You hand CloudFormation the edited template, it compares that against what is actually running right now, and it hands back a precise list of what it would add, change, or destroy. Nothing real moves until you say the word.

What a change set actually does

Creating a change set changes nothing in your account. None of your resources move. It is a dry run with a name attached, so you can create it now, look at it, and come back to it later. You point it at your edited template, and CloudFormation goes off to compute the plan.

aws cloudformation create-change-set \
  --stack-name payments-api \
  --change-set-name encrypt-db \
  --template-body file://template.yaml \
  --capabilities CAPABILITY_IAM

{
    "Id": "arn:aws:cloudformation:us-east-1:123456789012:changeSet/encrypt-db/6f2a1c9d-8b4e-4a77-9e21-3c5d0b7a4e10",
    "StackId": "arn:aws:cloudformation:us-east-1:123456789012:stack/payments-api/1e7b0a40-2f3c-11ef-9c1a-0aa1b2c3d4e5"
}

The --capabilities CAPABILITY_IAM flag is you signing off that this template is allowed to touch IAM (Identity and Access Management, the service that controls who can do what in AWS) resources; CloudFormation refuses to guess about permissions on your behalf. The call returns right away with two identifiers, both in Amazon Resource Name (ARN) form, which is AWS's unique home address for any single thing it manages: the change set Id and the StackId. Notice the plan itself is not in that response. It is still being built in the background. So you wait for it to finish before you try to read it, with aws cloudformation wait change-set-create-complete.

One sharp edge lives here: an empty plan counts as a failure, not a quiet success. If your edited template has no real differences from what is already running, CloudFormation does not hand you a friendly empty change set. It creates one in FAILED state with the reason "The submitted information didn't contain changes." The wait command then exits with a non-zero code, and that can break a pipeline that assumed a zero meant everything worked. Handle it on purpose: treat "no changes" as a normal outcome, delete the empty change set, and carry on.

Reading the Replacement column

Once the plan is ready, you read it. Every entry has an Action, one of Add, Modify, or Remove, and for anything being changed it also carries a Replacement value. Replacement is the single field that tells you whether you are looking at a safe edit or a demolition.

aws cloudformation describe-change-set \
  --stack-name payments-api --change-set-name encrypt-db \
  --query "Changes[].ResourceChange.[Action,LogicalResourceId,ResourceType,Replacement]" \
  --output table

--------------------------------------------------------------------------
|                            DescribeChangeSet                            |
+---------+--------------+-------------------------------+---------------+
|  Modify |  PaymentsDb  |  AWS::RDS::DBInstance          |  True         |
|  Modify |  AppServer   |  AWS::EC2::Instance           |  False        |
+---------+--------------+-------------------------------+---------------+

Two resources show up, because in the same edit you also nudged the application server's configuration. The application server (AWS::EC2::Instance, a virtual machine running on Elastic Compute Cloud, EC2) is a plain Modify with Replacement False, so CloudFormation edits it in place and the machine keeps its identity and its disk. The payments database is a Modify with Replacement True, and that word is the alarm. True means CloudFormation cannot make this change on the resource that exists, so it will destroy the current one and build a replacement from scratch. You will also meet a third value, Conditional. It means CloudFormation might replace the resource, depending on values it cannot work out until the update actually runs. Treat Conditional with exactly the same fear as True.

What forced the replacement

Replacement True tells you a resource will be recreated. It does not tell you why. For that, you drill into the same change set and pull the full record for that one resource.

aws cloudformation describe-change-set \
  --stack-name payments-api --change-set-name encrypt-db \
  --query "Changes[?ResourceChange.LogicalResourceId=='PaymentsDb'].ResourceChange | [0]"

{
    "Action": "Modify",
    "LogicalResourceId": "PaymentsDb",
    "PhysicalResourceId": "payments-db-prod",
    "ResourceType": "AWS::RDS::DBInstance",
    "Replacement": "True",
    "Scope": [
        "Properties"
    ],
    "Details": [
        {
            "Target": {
                "Attribute": "Properties",
                "Name": "StorageEncrypted",
                "RequiresRecreation": "Always"
            },
            "Evaluation": "Static",
            "ChangeSource": "DirectModification"
        }
    ]
}

The Details section names the exact cause. Target.Name is the property you touched, StorageEncrypted. RequiresRecreation reads Always, and that is what rolls up into Replacement True at the resource level. That field has three settings worth knowing: Never (the property can be updated in place), Conditionally (it might need a recreation), and Always (it always forces one). Evaluation reads Static here because CloudFormation already knows the new value straight from your template. When a value is only worked out at run time, say a name built from another resource's attribute, Evaluation reads Dynamic instead, CloudFormation cannot be certain ahead of time, and the resource shows up as Conditional rather than a hard True. ChangeSource of DirectModification means you edited this property yourself, as opposed to a parameter or a linked resource shifting underneath you.

Execute the plan or discard it

A change set only ever ends one of two ways. execute-change-set carries out the plan exactly as you saw it. delete-change-set discards the whole thing and touches nothing. In our story the plan showed Replacement True on the payments database, so executing it is precisely the disaster we set out to avoid. The right move is to delete this change set and take a path that keeps the data: snapshot the database, restore that snapshot into a brand-new encrypted instance, then point the application at the new one. If you like this preview step but want it folded into a single command, aws cloudformation deploy --no-execute-changeset creates a change set and then stops, so you can inspect it before you commit.

Change sets also go stale, and this one catches people out. A plan is computed against the stack exactly as it looked at that one instant. Execute a different change set, or let the stack change some other way, and every plan you did not run turns OBSOLETE and can never be executed. Two separate fields track a change set's life, and beginners mix them up constantly. Status is about building the diff: CREATE_COMPLETE means the plan finished and is ready to read. ExecutionStatus is about running it: AVAILABLE means you can still execute it, OBSOLETE means the plan is stale and dead.

aws cloudformation list-change-sets --stack-name payments-api \
  --query "Summaries[].[ChangeSetName,Status,ExecutionStatus]" \
  --output table

-------------------------------------------------------
|                    ListChangeSets                   |
+---------------------+------------------+------------+
|  old-instance-type  |  CREATE_COMPLETE |  OBSOLETE  |
|  bump-storage       |  CREATE_COMPLETE |  AVAILABLE |
+---------------------+------------------+------------+

Make replacement safe by default

Reading the plan protects you on the day you are paying attention. The next two controls protect you on the day you are not. Start with two resource attributes that decide what happens to a resource's data when it gets torn down. They cover different events, like two separate insurance policies on the same building. DeletionPolicy covers removal: the resource being deleted from the stack, or the whole stack being deleted. UpdateReplacePolicy covers replacement: what happens to the old resource when an update swaps it out for a new one. They are completely independent of each other. Set both.

Resources:
  PaymentsDb:
    Type: AWS::RDS::DBInstance
    DeletionPolicy: Snapshot
    UpdateReplacePolicy: Snapshot
    Properties:
      DBInstanceIdentifier: payments-db-prod
      Engine: postgres
      DBInstanceClass: db.t3.medium
      AllocatedStorage: "100"
      StorageEncrypted: true

With UpdateReplacePolicy: Snapshot in place, if any future change ever forces the database to be replaced, CloudFormation takes a snapshot of the old instance before it deletes it, so the data outlives the resource. Here is the trap that bites people. They set DeletionPolicy alone and assume it covers replacement too. It does not. On a replacement, DeletionPolicy sits there doing nothing, and UpdateReplacePolicy is the attribute that actually fires. Want a safety snapshot on both a straight deletion and a replacement? Write both attributes, exactly as above.

The second control does not rely on anyone remembering to read a diff, which is exactly why it is the strongest one. A stack policy is a small JSON (JavaScript Object Notation, a plain-text format for structured data) document you attach to the stack, and it spells out which resources may be updated and how. With no policy set, everything is allowed. You can write one that denies replacement and deletion of your most valuable resource, and CloudFormation will refuse the action even when an update explicitly asks for it. It works like the lock on a breaker box: routine switches flip freely, but nobody kills the main power by accident.

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "Update:*",
      "Principal": "*",
      "Resource": "*"
    },
    {
      "Effect": "Deny",
      "Action": ["Update:Replace", "Update:Delete"],
      "Principal": "*",
      "Resource": "LogicalResourceId/PaymentsDb"
    }
  ]
}

aws cloudformation set-stack-policy \
  --stack-name payments-api \
  --stack-policy-body file://protect-db.json
 
# a later change tries to replace the DB and is refused:
aws cloudformation execute-change-set \
  --stack-name payments-api --change-set-name rebuild-db
aws cloudformation describe-stack-events --stack-name payments-api \
  --query "StackEvents[?LogicalResourceId=='PaymentsDb']|[0].[LogicalResourceId,ResourceStatus,ResourceStatusReason]" \
  --output table

-----------------------------------------------------------------
|                      DescribeStackEvents                      |
+-------------+------------------+--------------------------------+
|  PaymentsDb |  UPDATE_FAILED   |  Action denied by stack policy |
+-------------+------------------+--------------------------------+

To make a protected change on purpose, you set a looser policy for the length of that one update, run it, then put the guard back. These two habits reinforce each other. The change set diff is your audit trail of exactly what a template edit would do, line by line, which beats "trust me, it is a tiny change" every time. The stack policy makes sure the destructive lines physically cannot fire unless a human takes the lock off first. If you are the defender reviewing an infrastructure change, that diff is where you catch the careless edit, or the malicious one, before it ever lands in production.

When you run the safe version of this change, prove it landed the way the plan promised. Read the database's PhysicalResourceId before and after: if it changed, a replacement happened, and you had better have that snapshot in hand. Confirm encryption is actually on with aws rds describe-db-instances and check the StorageEncrypted field. Then run drift detection (CloudFormation comparing its own record of the stack against the live resources) so the stack's story matches reality. The change set told you what should happen. These checks prove what did.

Try this

Work through “Make replacement safe by default” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: replacement on a stateful resource is a stop sign. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related