CoursesCompliance as codeEvidence automation

Evidence automation

Controls that evidence themselves, mapped.

Expert35 min · lesson 7 of 15

A photo of the smoke alarm on your ceiling proves the alarm exists. It says nothing about whether it worked every night for the past year. That gap is the whole problem an audit puts in front of you. It is week two of a SOC 2 Type II audit (Service Organization Control 2, the report where an outside assessor watches your controls across a period of months rather than on one day) and the assessor asks something that sounds easy: show me that every S3 bucket (Simple Storage Service, Amazon's object store) was encrypted at rest, on every day of the twelve-month period. You have the control. An AWS (Amazon Web Services) Config rule and a CI (continuous integration) policy both enforce encryption. But the auditor is not asking whether the control exists. They are asking you to prove it ran, over and over, for a year. A screenshot taken this morning answers none of that. Evidence automation is what answers it: make each control emit its own proof every time it runs, tag that proof with the requirement it satisfies, and store it where nobody can edit it. Then audit readiness is a state you live in rather than a scramble you survive.

What actually counts as evidence

Three words have to stay separate in your head. A control is the rule you promised to follow: "S3 buckets must be encrypted at rest." Evidence is the record that the rule was applied, meaning what ran, what the result was, which resources it covered, and when. The third idea is the one auditors live by, design versus operating effectiveness. A SOC 2 Type I report says the control looked correct on a single day, the way a restaurant inspection certificate says the kitchen was clean on the morning of the visit. A Type II report, the one your customers actually ask for, says the control worked across six to twelve months, which is closer to a year of daily temperature logs. You cannot prove a year with a snapshot. Machine-made evidence dissolves the problem, because the control's own runs are the record. Every scanner run, every Config-rule evaluation, every admission decision is already a timestamped, scoped, pass-or-fail fact. Evidence automation is the discipline of collecting those facts, tagging them with control IDs, and keeping them somewhere they cannot be doctored. Auditors trust tool-generated, locked-down evidence far more than a spreadsheet an engineer assembled, because a hand-built spreadsheet is exactly the artifact they are trained to distrust.

Step 1: generate the evidence with a real scanner

The cheapest evidence to trust is evidence a tool produced, not evidence a person typed. Prowler is an open-source scanner that runs hundreds of checks against a cloud account and writes what it finds in OCSF (Open Cybersecurity Schema Framework), a shared layout for security findings expressed as JSON (JavaScript Object Notation, structured plain text that programs can read), so tools from different vendors describe the same thing the same way. Every Prowler finding carries its framework mappings already baked in. Install it in an isolated environment and point it at the account. The run below scans only S3 and writes two things: a CSV (comma-separated values, the spreadsheet-friendly copy) a human can skim, and the machine-grade OCSF JSON that becomes the archived artifact.

generate evidence (Prowler 5.x)
# isolated install
pipx install prowler
# scan only S3, emit a human CSV + machine-grade OCSF JSON
prowler aws --services s3 -M csv json-ocsf -o ./output

Prowler prints each finding to the terminal as it goes and drops timestamped files under ./output/. Watch the exit code. Prowler returns 3 when any check FAILs, so a pipeline can gate on it. For evidence collection you usually do the opposite: run it non-blocking and archive every finding, pass and fail alike. The failures belong in the operating-effectiveness record every bit as much as the passes do.

terminal output (streamed per finding)
Using AWS credentials for account 123456789012 (us-east-1)
Executing 12 checks for service s3, please wait...
s3_bucket_default_encryption [High] Ensure S3 buckets have default encryption enabled
PASS us-east-1 acme-prod-logs Server-side encryption is enabled (SSE-KMS)
FAIL us-east-1 acme-legacy-uploads Server-side encryption is not configured
Overview:
PASS: 11 FAIL: 1 (91.6% compliant)
Findings written to ./output/prowler-output-123456789012-20260714090003.ocsf.json
$ echo $?
3

The OCSF file is the evidence you keep, not the terminal text. Each finding carries its status, the exact resource in scope, the time it was evaluated, and the control IDs it maps to. That embedded mapping is what lets one check stand as evidence for several requirements at once. The single S3-encryption finding below covers SOC 2 CC6.1, PCI DSS 3.5.1 (Payment Card Industry Data Security Standard, the rulebook for handling card data) and CIS 2.1.1 (Center for Internet Security, whose benchmarks are the widely used hardening baselines) in one shot.

the evidence artifact: one OCSF finding (trimmed)
{
"status_code": "FAIL",
"severity": "High",
"status_detail": "Server-side encryption is not configured for S3 bucket acme-legacy-uploads.",
"finding_info": {
"uid": "prowler-aws-s3_bucket_default_encryption-123456789012-us-east-1-acme-legacy-uploads",
"title": "Ensure S3 buckets have default encryption enabled",
"created_time": "2026-07-14T09:00:03Z"
},
"resources": [
{ "name": "acme-legacy-uploads", "type": "AwsS3Bucket", "region": "us-east-1" }
],
"unmapped": {
"check_id": "s3_bucket_default_encryption",
"compliance": { "SOC2": ["cc_6_1"], "PCI-4.0": ["3.5.1"], "CIS-3.0": ["2.1.1"] }
}
}

Step 2: seal it so nobody can edit it

Generated evidence is worth nothing if someone can rewrite it afterwards, so integrity comes next. WORM (write once read many) storage is the digital version of a bound ledger written in permanent ink. You can add a page. You cannot go back and change one. On AWS this is S3 Object Lock, and there is a catch worth knowing before you start: it has to be switched on when the bucket is created. It requires versioning, and you cannot retrofit it onto an existing bucket without going through support. Write the Prowler output with a COMPLIANCE-mode retention date and a SHA-256 checksum (Secure Hash Algorithm, 256-bit, a short fingerprint computed from the file's bytes, so any later edit produces a different fingerprint), then try to erase it.

seal the evidence in WORM storage
# Object Lock must be enabled at bucket creation; it auto-enables versioning
aws s3api create-bucket \
--bucket acme-audit-evidence --region us-east-1 \
--object-lock-enabled-for-bucket
# write the evidence write-once, retained 7 years, with an integrity checksum
aws s3api put-object \
--bucket acme-audit-evidence \
--key evidence/2026/07/14/prowler-123456789012.ocsf.json \
--body ./output/prowler-output-123456789012-20260714090003.ocsf.json \
--object-lock-mode COMPLIANCE \
--object-lock-retain-until-date 2033-07-14T00:00:00Z \
--checksum-algorithm SHA256
put-object response, then a delete that is refused
{
"ETag": "\"9b2cf5f0e1a4...\"",
"ServerSideEncryption": "AES256",
"VersionId": "3sL0nGvErSiOnIdExAmPlE",
"ChecksumSHA256": "K7fJ2r8Qw9cVb3mNpXe0dR4tYu6iOa1sD2fG3hJ4kL="
}
# now try to erase the evidence:
$ aws s3api delete-object --bucket acme-audit-evidence \
--key evidence/2026/07/14/prowler-123456789012.ocsf.json \
--version-id 3sL0nGvErSiOnIdExAmPlE
An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied

The delete is refused. In COMPLIANCE mode nobody can shorten the retention or remove the object before the date passes, not even the account root user, and that is precisely the property that makes the evidence defensible. The checksum layers tamper-evidence on top of it. An auditor can re-hash the stored object and confirm it is byte-for-byte the file that was written on the run date.

GOVERNANCE mode looks locked but will not survive an auditor
S3 Object Lock has two modes and only one of them satisfies an auditor. GOVERNANCE mode stops casual deletion, but any principal holding s3:BypassGovernanceRetention can shorten it or break it, and that permission normally sits with admins and root. A good auditor knows this and will treat GOVERNANCE-locked findings as alterable. Use COMPLIANCE mode for anything you plan to hand over as proof, because it removes the bypass path completely. The trade-off is genuine: a COMPLIANCE lock set with a retention date that is too long cannot be undone by anyone, so you pay to store that object for the whole term. Set retention deliberately to your evidence-retention requirement (commonly 7 years for SOC 2 and PCI programs) and never pad it "to be safe."

Step 3: turn the audit into a query

Once evidence is generated, mapped and sealed, the audit stops being a scramble and turns into a lookup. Because every finding carries control IDs, you can pull "every evaluation of the encryption control" straight out of the store instead of hunting for it through tickets and chat history. Cloud services keep the same history natively. AWS Config records a fresh compliance evaluation for a rule every time a matching resource changes, and you retrieve the full pass/fail timeline with one API (application programming interface) call. No engineer in the loop, no screenshot, no interpretation. Prowler's own --compliance flag (for example --compliance soc2_aws) does the equivalent job on the reporting side, emitting a per-framework CSV that lists each control ID beside the checks that evidence it, so the mapping stops being something you assert in a meeting and becomes a file you hand over.

query the control's evaluation history (AWS Config)
aws configservice get-compliance-details-by-config-rule \
--config-rule-name s3-bucket-server-side-encryption-enabled \
--compliance-types NON_COMPLIANT COMPLIANT
AWS Config response: a timestamped verdict per resource
{
"EvaluationResults": [
{
"EvaluationResultIdentifier": {
"EvaluationResultQualifier": {
"ConfigRuleName": "s3-bucket-server-side-encryption-enabled",
"ResourceType": "AWS::S3::Bucket",
"ResourceId": "acme-legacy-uploads"
},
"OrderingTimestamp": "2026-07-14T08:59:12.000Z"
},
"ComplianceType": "NON_COMPLIANT",
"ResultRecordedTime": "2026-07-14T09:00:41.000Z"
}
]
}

That one call tells you which bucket was non-compliant and the exact moment the verdict was recorded, an unbroken record the auditor can sample from any day in the period. Two habits keep it honest at scale. First, false positives and accepted risks get triaged, never deleted. A FAIL you have decided to live with is recorded as a documented exception, meaning a suppression carrying a written reason and a named owner, because quietly dropping findings makes the whole evidence set look edited. Second, OCSF is the glue once you go past a handful of accounts. Normalizing Prowler output, Config evaluations and admission decisions into one schema is what lets hundreds of accounts land in a single queryable, immutable store instead of a heap of reports that do not agree on what a finding even looks like.

The lifecycle of one piece of evidence
1scanner / rule fires
Prowler run or Config evaluation
2finding emitted (OCSF)
status + scope + timestamp
3mapped to control IDs
SOC2 CC6.1, PCI 3.5.1, CIS 2.1.1
4sealed in WORM store
S3 Object Lock, COMPLIANCE mode
5auditor queries it later
audit becomes a query, not a scramble
Each control run produces its own proof, tagged to the requirement it satisfies and locked so nobody can rewrite it. The path is the same whether the source is a scanner, a Config rule or an admission decision.
Quick check
01Your audit-evidence bucket uses S3 Object Lock in GOVERNANCE mode with a 7-year retention. Why might a SOC 2 auditor still refuse to call the stored findings tamper-evident?
Correct — GOVERNANCE mode is a soft lock. Only COMPLIANCE mode blocks removal for every principal, root included, until the retention date passes.
Incorrect — Every locked object version carries its own timestamps, and the put-object response even returns a VersionId. Dating is not the weak spot.
Incorrect — A Type II period runs six to twelve months, so a 7-year retention is far longer than the period, not shorter.
Incorrect — Object Lock modes behave the same regardless of storage class; the difference between the modes is who can bypass them.
02You already have a bucket holding last quarter's Prowler output and you want to make it WORM storage with S3 Object Lock. What does the lesson say you will run into?
Correct — That is why the evidence bucket is created up front with --object-lock-enabled-for-bucket, which also auto-enables versioning.
Incorrect — The lesson is explicit that Object Lock must be enabled at bucket creation, not added later.
Incorrect — That flag sets retention on one object inside a bucket that already has the feature enabled; it does not enable the feature on the bucket.
Incorrect — Backwards. Creating the bucket with Object Lock enabled auto-enables versioning rather than removing it.
03Your nightly evidence job runs prowler aws --services s3 -M csv json-ocsf -o ./output. The terminal shows 11 PASS and 1 FAIL, the run exits with code 3, and your pipeline marks the job failed and discards the output. What should you change?
Correct — Exit code 3 only means at least one check FAILed. Failures belong in the operating-effectiveness record, and an accepted risk gets a recorded suppression rather than a delete.
Incorrect — Silently dropping findings makes the whole evidence set look edited, which is exactly the impression you are trying to avoid with an auditor.
Incorrect — Prowler returns 3 specifically when a check FAILs. It is a result about your account, not a malfunction of the tool.
Incorrect — The OCSF file is the artifact you archive, because it carries the status, scope, timestamp and control mappings. The CSV is the convenience copy.

From one run to every day

Generating, mapping and sealing evidence on a single run is the atom of an audit program. The next lesson, Continuous compliance, turns that atom into an always-on loop: the same checks fire on a schedule and on every change, so evidence accrues daily instead of only when someone remembers to scan. Get there and "prove the encryption control operated all year" becomes a question you answer with a bucket listing.

Try this

Run s3_bucket_default_encryption [High] Ensure S3 buckets have default encryption enabled on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: gOVERNANCE mode looks locked but will not survive an auditor. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related