StackSets, nesting & drift

Multi-account/Region IaC and drift detection.

A coffee chain does not let each store design itself. Head office keeps one master blueprint: where the counter goes, what the signage says, which door is the fire exit. Every new store gets built from that blueprint. Revise it once and the change reaches all 400 stores in one motion, and when a manager quietly paints over a fire exit, the inspector walking in with the blueprint in hand spots the difference in seconds. CloudFormation, the AWS service that builds infrastructure from a text file you write, gives you the same three moves at professional scale. Nested stacks are the reusable sections of the blueprint. StackSets push one template into many accounts and Regions from a single operation. Drift detection flags anything a human changed by hand. Wire all three together and your infrastructure stays reproducible even as the estate grows past a hundred accounts.

Nested stacks: breaking one huge template apart

One template can only hold so much. AWS caps it at 500 resources, 200 parameters, 200 outputs, 200 mappings, and 51,200 bytes if you pass the template inline in the API call (1 MB if you upload it to S3, Amazon's object storage, first). You will hit a human limit long before the AWS one. Past a few dozen resources a flat template stops being readable, and nobody wants to be the person editing line 900 on a Friday. A nested stack is a stack that exists as a resource inside another stack. The parent declares the child with an AWS::CloudFormation::Stack resource, points TemplateURL at the child template sitting in S3, passes values down as Parameters, and reads values back out of the child's Outputs with Fn::GetAtt. So you write a networking stack once, a database stack once, an app stack once, get each reviewed properly, then assemble them in a parent. Each child appears as its own stack in the console with its own event log and its own rollback, which makes a failure far easier to pin down.

A child template sitting on your laptop is no use to CloudFormation until it reaches S3. aws cloudformation package does that upload for you and rewrites every TemplateURL in the parent to point at the object it created, so you still deploy the whole thing as one unit. A nested hierarchy can build up to 2,500 resources in a single operation, well past the flat limit of 500. What you pay for that is welded lifecycles. Delete the parent and every child goes with it. An update that fails three levels down rolls the entire tree back. If all you want is to hand one value (a VPC ID, the identifier of a virtual private cloud, or an ARN, an Amazon Resource Name) from one stack to another, and the two stacks should otherwise live separate lives, reach for cross-stack references instead: Export in the stack that produces the value, Fn::ImportValue in the one that consumes it. Those keep the lifecycles apart, with one catch. AWS refuses to delete an exporting stack while anything still imports from it.

# Parent template references each child as a resource:
#   NetworkStack:
#     Type: AWS::CloudFormation::Stack
#     Properties:
#       TemplateURL: ./network.yaml          # local path -> rewritten by 'package'
#       Parameters: { VpcCidr: 10.0.0.0/16 }
 
# 'package' uploads local child templates to S3 and rewrites every TemplateURL:
aws cloudformation package \
  --template-file parent.yaml \
  --s3-bucket my-cfn-artifacts \
  --output-template-file parent.packaged.yaml
# Uploading to a1b2c3d4e5f6...  4531 / 4531  (100.00%)
# Successfully packaged artifacts and wrote output template to parent.packaged.yaml.
 
aws cloudformation deploy \
  --template-file parent.packaged.yaml \
  --stack-name prod-platform \
  --capabilities CAPABILITY_NAMED_IAM
# Waiting for changeset to be created..
# Waiting for stack create/update to complete
# Successfully created/updated stack - prod-platform

StackSets: one operation, every account you own

A StackSet is the mailing list for your blueprint. You define the template once in an administrator account, and CloudFormation creates a stack instance, one running copy of that stack for one account in one Region, everywhere on the list, from a single action. Two permission models decide how the admin account is allowed to reach into the targets. SELF_MANAGED is the do-it-yourself route: you create an AWSCloudFormationStackSetAdministrationRole in the admin account, plus an AWSCloudFormationStackSetExecutionRole that trusts it inside every target account. Use it when the targets sit outside AWS Organizations (the AWS feature that groups accounts under one policy root and one bill), or when they straddle organizational boundaries. SERVICE_MANAGED hands that IAM (Identity and Access Management, the AWS permissions system) plumbing to CloudFormation once you switch on trusted access with Organizations. After that you target whole organizational units (OUs, folders of accounts) instead of typing out account IDs, and with AutoDeployment enabled, any account somebody later moves into the OU picks up the baseline on its own. The quotas are roomy: 1,000 stack sets per admin account and 100,000 instances per stack set. Only 10,000 instance operations may run at the same time in a Region, though.

# SERVICE_MANAGED needs trusted access enabled once for the whole org:
aws cloudformation activate-organizations-access
 
aws cloudformation create-stack-set \
  --stack-set-name org-security-baseline \
  --template-body file://baseline.yaml \
  --permission-model SERVICE_MANAGED \
  --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false \
  --capabilities CAPABILITY_NAMED_IAM
# {
#     "StackSetId": "org-security-baseline:8f2a1c9e-3b4d-4e6f-a1b2-c3d4e5f6a7b8"
# }
 
# One operation -> baseline stack into every account in the OU, across 3 Regions:
aws cloudformation create-stack-instances \
  --stack-set-name org-security-baseline \
  --deployment-targets OrganizationalUnitIds=ou-r0a1-1x2y3z4w \
  --regions eu-west-1 us-east-1 ap-southeast-1 \
  --operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=25,FailureTolerancePercentage=10
# {
#     "OperationId": "1d7f0e2a-9c8b-4a6d-b5e3-2f1a0c9d8e7f"
# }

Keeping the blast radius small

Rolling one template into 300 accounts at once is exactly how a bad template breaks 300 accounts at once. --operation-preferences is the dial that stops that. MaxConcurrentPercentage (or MaxConcurrentCount) caps how many accounts deploy in parallel. FailureTolerancePercentage sets how many instances are allowed to fail before CloudFormation halts the operation instead of pressing on. RegionConcurrencyType=SEQUENTIAL works through Regions one at a time, so a problem that surfaces in us-east-1 never reaches eu-west-1. A sane default pairs a low failure tolerance with modest concurrency, so the rollout stalls after the first handful of failures rather than faithfully copying your mistake into every account you own. Try the change in one throwaway account first. Then widen it in stages.

Drift detection: finding the changes nobody wrote down

Drift is the gap between what the template says and what is actually running. Someone opens the console at 2am, widens a security group to clear an incident, never comes back to update the code, and now your template is fiction. Detection runs in three legs, and it runs asynchronously: you start it, AWS works in the background, you check back. detect-stack-drift hands you a StackDriftDetectionId token straight away. You poll describe-stack-drift-detection-status with that token until DetectionStatus reads DETECTION_COMPLETE. Then describe-stack-resource-drifts names the offenders. A stack comes back as DRIFTED, IN_SYNC, or NOT_CHECKED under StackDriftStatus, and each individual resource comes back as MODIFIED, DELETED, IN_SYNC, or NOT_CHECKED. Two sharp edges are worth memorising. CloudFormation compares only the properties you set explicitly, so anything left at its default is invisible to the check. And while an unsupported resource type at least gets flagged NOT_CHECKED, a few properties drop out of the comparison silently: KMSKeyId (the Key Management Service encryption key) on any resource, and the source code of a Lambda function, the AWS service that runs your code without a server. Those never surface as drift, and they never surface as NOT_CHECKED either.

# 1. Kick off async detection; you get a token back immediately:
aws cloudformation detect-stack-drift --stack-name prod-platform
# {
#     "StackDriftDetectionId": "b4e0f8c2-1a3d-4c5e-9f6a-7b8c9d0e1f2a"
# }
 
# 2. Poll until DetectionStatus is DETECTION_COMPLETE:
aws cloudformation describe-stack-drift-detection-status \
  --stack-drift-detection-id b4e0f8c2-1a3d-4c5e-9f6a-7b8c9d0e1f2a
# {
#     "StackId": "arn:aws:cloudformation:eu-west-1:111122223333:stack/prod-platform/...",
#     "StackDriftStatus": "DRIFTED",
#     "DetectionStatus": "DETECTION_COMPLETE",
#     "DriftedStackResourceCount": 1
# }
 
# 3. List only the resources that actually drifted:
aws cloudformation describe-stack-resource-drifts \
  --stack-name prod-platform \
  --stack-resource-drift-status-filters MODIFIED DELETED
# {
#     "StackResourceDrifts": [{
#         "LogicalResourceId": "AppSecurityGroup",
#         "ResourceType": "AWS::EC2::SecurityGroup",
#         "StackResourceDriftStatus": "MODIFIED",
#         "PropertyDifferences": [{
#             "PropertyPath": "/SecurityGroupIngress/0/CidrIp",
#             "ExpectedValue": "10.0.0.0/16",
#             "ActualValue": "0.0.0.0/0",
#             "DifferenceType": "NOT_EQUAL"
#         }]
#     }]
# }

The same trick scales to the whole estate. detect-stack-set-drift fans detection out across every instance in the stack set, and describe-stack-set reports the rolled-up DriftStatus along with a count of how many instances have drifted. Run it on a schedule. Drift detection costs nothing, but it is rate-limited, so a nightly sweep fired by EventBridge (the AWS event bus, which can trigger things on a cron schedule) is about right, while a tight polling loop will get you throttled. CloudFormation and StackSets carry no charge of their own. You pay for the resources the templates create, plus a small per-handler-operation fee for third-party or private registry extensions once you pass the first 1,000 operations in a month.

# Drift across the whole estate — every instance in the stack set:
aws cloudformation detect-stack-set-drift --stack-set-name org-security-baseline
# {
#     "OperationId": "9a8b7c6d-5e4f-3a2b-1c0d-9e8f7a6b5c4d"
# }
 
aws cloudformation describe-stack-set --stack-set-name org-security-baseline \
  --query 'StackSet.StackSetDriftDetectionDetails.{Status:DriftStatus,Drifted:DriftedStackInstancesCount,Total:TotalStackInstancesCount}'
# {
#     "Status": "DRIFTED",
#     "Drifted": 2,
#     "Total": 47
# }

Make code the only way in

Detection is worth little if drift is routine, so the real work is making out-of-band change hard in the first place. Preventive guardrails come first: Service Control Policies (organization-wide rules that cap what an account may do, whatever its own IAM policies say) and permission boundaries that deny console edits on managed resources to everyone except a break-glass role, the emergency identity you use loudly and rarely. That leaves IaC (infrastructure as code, describing what you run in files anyone can review) as the only sanctioned route in. Next, catch bad templates before they ship: cfn-lint, cfn-nag, cfn-guard, or Checkov running inside the pipeline will flag misconfigurations at build time. Scheduled drift detection is the backstop behind both, and unexplained drift deserves a page like any other incident. You reconcile it one of two ways: update the template to bless the change, or put the resource back the way the code describes it. If somebody created a resource by hand and you want it under management without recreating it, an import operation folds it into the stack in place, which is the clean way to close out a MODIFIED or DELETED finding.

So the division of labour reads like this. Nested stacks cut one oversized template into modules with clear inputs and outputs. StackSets take a module and push it across accounts and Regions, and the operation preferences you set decide whether one failing OU stops the rollout dead or the rollout carries on without it.

Drift reports on a StackSet double as a quiet census of who is still working in the console. Detect on a schedule, and when an instance has drifted, fix it by redeploying that stack instance. Editing production by hand until it "matches" the template is the same habit that produced the drift.

Service-managed StackSets aimed at an OU with AutoDeployment enabled will onboard new accounts without anyone asking you first. That is the whole point of them, and it is also the risk: a mistake in the baseline template lands everywhere, including in accounts that did not exist when you wrote it. Pair the stack set with Service Control Policies so a template trying to create public S3 buckets in every OU cannot succeed even when it deploys.

Try this

List the stack sets you have and check the status of their instances. Then, if you have permission somewhere harmless, start drift detection on a lab stack and read what comes back.

aws cloudformation list-stack-sets --query 'Summaries[].{Name:StackSetName,Status:Status}' --output table
aws cloudformation list-stack-instances --stack-set-name org-guardrails \
  --query 'Summaries[].{Acct:Account,Region:Region,Status:Status}' --output table
aws cloudformation detect-stack-drift --stack-name lab-net
aws cloudformation describe-stack-drift-detection-status --stack-drift-detection-id 1234 \
  --query '{Status:DetectionStatus,Drift:StackDriftStatus}' --output table

org-guardrails | ACTIVE
111122223333 | us-east-1 | CURRENT
111122223333 | eu-west-1 | CURRENT
-----------------------------
| DescribeStackDrift...     |
+-----------+---------------+
| Detection | Drift         |
+-----------+---------------+
| DETECTION_COMPLETE | DRIFTED |
+-----------+---------------+

Takeaway

StackSets are how one template reaches every account you own. Drift detection is how you find out that reality has wandered away from that template. What matters is what you do next: make the template win again, on purpose, in code.

For your next shift: turn on scheduled drift detection for one production stack, then raise a ticket for every DRIFTED resource a human changed rather than a pipeline.

Detection tells you *what* changed. It changes nothing back, and a detection pipeline with no remediation path behind it becomes one more alert stream nobody reads. Turning "someone should reconcile that" into an automatic response, taking inventory of the fleet, patching it, and running a remediation the moment drift appears, is the job of AWS Systems Manager, which is the next lesson. Its runbooks are how you close the gap between spotting a divergence and correcting it across hundreds of machines.

Related