Remediation & prevention

Auto-fix the safe cases; guardrail the rest.

A fire door propped open with a wedge is still a fire door. It stops being one at the exact moment it matters. That is what happened to a bucket called acme-prod-logs. An assessor is sampling evidence for a SOC 2 audit, short for Service Organization Control 2, the report software vendors hand their customers to show their controls hold. One control on the sample list covers S3, Simple Storage Service, Amazon's file storage: "S3 buckets must block all public access." That line is CIS AWS 2.1.5, one numbered item in the Center for Internet Security's hardening benchmark. So the assessor pulls three weeks of history out of AWS Config, the Amazon Web Services service that records how every resource is configured and how that configuration changed over time. One bucket has a gap. An on-call engineer stripped its public-access block at 2 a.m. to share a log export during an incident, and never put it back. The control failed silently for twenty-one days. The auditor then asks two questions, and those two questions are the whole of this lesson. How fast can you close a violation once something detects it? And how do you stop that entire class of violation from ever happening again?

From finding to closed control

A finding is the smoke alarm going off. Remediation is somebody actually putting the fire out, and it is what lets you mark a control resolved instead of merely observed. Compliance as code treats that second part as a program you can run, not a paragraph in a runbook nobody has opened since 2021. There are two families of it. Detect-then-fix is reactive: the resource already exists, it has drifted away from policy, and something spots it. That something is a detective control, which might be an AWS Config rule, a CSPM (Cloud Security Posture Management, a scanner that continuously checks cloud accounts against a rule set), or a scheduled query. A remediation action then pushes the resource back to baseline after the fact. Prevent-at-admission is proactive: a preventive control stands at the door, inspects every create-or-update request before the resource is ever written down, and either turns it away or repairs it on the spot, so the broken version never exists at all. One closes gaps that already opened. The other refuses to let them open. A serious program runs both, because each is blind exactly where the other sees, and knowing which one to reach for is the real skill.

Detect-then-fix: AWS Config plus SSM Automation

AWS Config is the building inspector who walks the floor every time somebody moves a wall, and also does a scheduled round anyway. It evaluates your resources against rules, event-driven when a resource's configuration changes, or on a timer. When a rule marks a resource NON_COMPLIANT, you attach a remediation configuration to that rule, and the configuration runs an SSM Automation document against whatever got flagged. SSM is AWS Systems Manager, and a document here is a runbook: an ordered list of AWS API (Application Programming Interface, the machine-facing way of telling AWS to do something) calls. AWS ships managed documents for the common fixes, so you will rarely write one yourself. Set Automatic to true and the fix fires with nobody in the loop. MaximumAutomaticAttempts and RetryAttemptSeconds put a ceiling on how hard it keeps retrying a resource that refuses to settle. The piece that makes this scale to a whole account is the RESOURCE_ID placeholder: Config drops the flagged resource's id into the document's BucketName parameter at run time, so one remediation config repairs every offender you have today and every one you acquire next quarter, rather than one hard-coded bucket. The document runs under an IAM (Identity and Access Management) role you hand it as AutomationAssumeRole. That role has to trust ssm.amazonaws.com, because Systems Manager assumes it on your behalf, and for this job it needs exactly one write permission: s3:PutBucketPublicAccessBlock. Give each rule its own role rather than sharing one, because a shared remediation role quietly collects the write permissions of every rule you attach it to until it can change most of production. Start by confirming the control really is failing, and on which resource.

# Which resources is Config flagging for this rule right now?
aws configservice get-compliance-details-by-config-rule \
  --config-rule-name s3-bucket-level-public-access-prohibited \
  --compliance-types NON_COMPLIANT \
  --query 'EvaluationResults[].EvaluationResultIdentifier.EvaluationResultQualifier.ResourceId' \
  --output json

[
    "acme-prod-logs"
]

That id is the target. Now write the remediation down as code: which SSM document to run, which IAM role it borrows, and the parameters that spell out what compliant looks like once it finishes. Reach for the bucket-level runbook, AWSConfigRemediation-ConfigureS3BucketPublicAccessBlock, whose required BucketName parameter takes the id of the flagged bucket. There is a near-identical sibling, AWSConfigRemediation-ConfigureS3PublicAccessBlock, that works at the account level and requires an AccountId. It defines no BucketName at all, so pointing a bucket rule at it does not fail quietly. The execution errors out on parameters, the bucket stays NON_COMPLIANT, and the Config console shows a failed remediation action rather than a fix that ran and did nothing. The special value RESOURCE_ID is what tells Config to inject whichever bucket the rule flagged, which is why this one file covers the whole account and not only today's offender.

[
  {
    "ConfigRuleName": "s3-bucket-level-public-access-prohibited",
    "TargetType": "SSM_DOCUMENT",
    "TargetId": "AWSConfigRemediation-ConfigureS3BucketPublicAccessBlock",
    "TargetVersion": "1",
    "Automatic": true,
    "MaximumAutomaticAttempts": 5,
    "RetryAttemptSeconds": 60,
    "Parameters": {
      "AutomationAssumeRole": {
        "StaticValue": { "Values": ["arn:aws:iam::111122223333:role/ConfigRemediationRole"] }
      },
      "BucketName":            { "ResourceValue": { "Value": "RESOURCE_ID" } },
      "BlockPublicAcls":       { "StaticValue": { "Values": ["true"] } },
      "IgnorePublicAcls":      { "StaticValue": { "Values": ["true"] } },
      "BlockPublicPolicy":     { "StaticValue": { "Values": ["true"] } },
      "RestrictPublicBuckets": { "StaticValue": { "Values": ["true"] } }
    }
  }
]

# Attach the remediation to the rule...
aws configservice put-remediation-configurations \
  --remediation-configurations file://remediation.json
 
# ...and force it now for the flagged bucket (Automatic would fire on next eval).
aws configservice start-remediation-execution \
  --config-rule-name s3-bucket-level-public-access-prohibited \
  --resource-keys resourceType=AWS::S3::Bucket,resourceId=acme-prod-logs

{
    "FailedBatches": []
}
{
    "FailedItems": []
}

With Automatic set to true, this would have fired on its own at the next evaluation. The start-remediation-execution call forces it right now so you are not sitting around waiting for the demo. The automation puts the public-access block back. Read the resource straight from the API to confirm the control passes. Before, the block was missing entirely. After, all four settings are true.

aws s3api get-public-access-block --bucket acme-prod-logs

{
    "PublicAccessBlockConfiguration": {
        "BlockPublicAcls": true,
        "IgnorePublicAcls": true,
        "BlockPublicPolicy": true,
        "RestrictPublicBuckets": true
    }
}

To an auditor, the fix matters less than the receipt for the fix. Every remediation execution leaves two trails: the SSM Automation run history, and the Config timeline entry where the resource flips from NON_COMPLIANT back to COMPLIANT. Lay those side by side and you have a closed loop with a real number attached, a mean time to remediate you can measure, instead of somebody's verbal assurance that the team keeps an eye on things. A control that provably heals itself in minutes and can produce its own paperwork is far stronger evidence than one a human swears they watch. Stay precise about the word evidence, though. A passing check is evidence toward the control, not a certification of it, and the auditor still wants your written narrative and the sampling method behind those numbers.

Prevent-at-admission: Kyverno mutate

Detect-then-fix always leaves a window. Twenty-one days in the story above, or at best the few minutes between the drift and the next evaluation. Prevention shrinks that window to zero by moving the check to the moment of creation. Kubernetes keeps a bouncer at the door for exactly this. An admission controller sees every create and update before the object is persisted to etcd, the key-value store where a cluster keeps its real state. A validating policy turns a non-compliant manifest away outright. A mutate policy is the better tool for the safe, boring, well-understood fixes, because it patches the resource in flight, so the version that actually gets stored is the compliant one. It is the same auto-fix-the-safe-case instinct from the AWS side, applied one step earlier in the resource's life. Kyverno's +() anchor adds a field only when that field is absent, so you can inject a secure default without clobbering a team that set the value on purpose. The policy below guarantees every Pod carries a seccomp profile (seccomp is a Linux kernel feature that limits which system calls a container is allowed to make), which Pod Security Standards Restricted requires, even when whoever wrote the manifest forgot. The same requirement can shift left, too. Run kyverno apply against the manifests in CI (continuous integration, the automated build that runs on every pull request) and diff its mutated output against the source, and a Pod that shipped without the profile shows up as a patched field right there in the pull request. One catch. A mutate on its own will not fail your build, because kyverno apply exits non-zero only on a validate failure. Pair it with a validating twin when you want CI to hard-block rather than quietly repair.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-seccomp-profile
  annotations:
    policies.kyverno.io/title: Add default seccomp profile
spec:
  rules:
    - name: set-seccomp-runtimedefault
      match:
        any:
          - resources:
              kinds:
                - Pod
      mutate:
        patchStrategicMerge:
          spec:
            +(securityContext):
              +(seccompProfile):
                +(type): RuntimeDefault

kubectl apply -f add-seccomp-profile.yaml
 
# This Pod sets NO seccompProfile of its own:
kubectl run audit-test --image=nginx:1.27 --restart=Never
 
# Read back what was actually stored in the cluster:
kubectl get pod audit-test -o jsonpath='{.spec.securityContext.seccompProfile.type}'

clusterpolicy.kyverno.io/add-seccomp-profile created
pod/audit-test created
RuntimeDefault

Picking a lane, and closing the loop

Neither approach wins outright. Detect-then-fix reaches everything a gate never sees: infrastructure that was already running, changes made out of band, third-party automation, somebody clicking around in the console at midnight. It also always trails reality, and it can pick a fight with an engineer making a perfectly legitimate change in the middle of an incident. Prevent-at-admission is airtight for anything that comes through the door and completely blind to everything that does not. A mutate that quietly rewrites a manifest can also leave a developer baffled about why their YAML (the text format Kubernetes manifests are written in) grew a field they never typed, so ship it with a clear policies.kyverno.io/title annotation and a validating counterpart that says out loud what the rule wants. A grown-up program runs both lanes and adds a third. Fixes with real blast radius never auto-fire: deleting a resource, revoking a role, tearing down a security group. Those become tracked tickets with a named owner and a due date, which is how time-to-remediate turns into a number you can report rather than a hope. Then any violation class that keeps coming back gets designed out with an admission policy or an SCP (Service Control Policy, an AWS guardrail that caps what an entire account is permitted to do at all). Detect, remediate, prevent. Each class fixed once, then gated shut behind you.

Try this

Run both lanes end to end on things you can throw away: a sandbox AWS account and a cluster you are happy to delete. For the AWS half, turn on the s3-bucket-level-public-access-prohibited Config rule, then create a bucket and strip its public-access block to reproduce the 2 a.m. mistake from the story. Now work the walkthrough above in order: get-compliance-details-by-config-rule to watch it get flagged, put-remediation-configurations with your own remediation role in place of the 111122223333 example, start-remediation-execution to force the fix, and get-public-access-block to confirm all four settings came back true. For the Kubernetes half, install Kyverno, apply add-seccomp-profile.yaml, run the bare Pod, and read the stored seccompProfile back. Then break each one on purpose. Point TargetId at the account-level sibling while leaving BucketName in place, and read the parameter error the failed execution hands back. Drop the +() anchors from the mutate and watch it overwrite a profile a team set deliberately.

# A bucket-level public access block is on by default, so remove it to open the gap.
BUCKET=acme-sandbox-logs-$(date +%s)
aws s3api create-bucket --bucket "$BUCKET" --region us-east-1
aws s3api delete-public-access-block --bucket "$BUCKET"
 
# Config re-evaluates on its own schedule; nudge it if you do not want to wait.
aws configservice start-config-rules-evaluation \
  --config-rule-names s3-bucket-level-public-access-prohibited

Takeaway

Sort every control by what the wrong fix would cost, not by how easy it is to automate. Reversible and well understood belongs in detect-then-fix. Anything that arrives through an admission gate belongs in prevent-at-admission, where the broken version never gets written down at all. Anything that deletes, revokes or tears down belongs on a ticket with a name and a date on it. Write that sorting down beside your rule list before you switch anything to Automatic:true, so the argument your team has is which lane a control belongs in, and not why the automation removed a gateway at 3 a.m.

Related