>_SecOpsLog_
CoursesAWS security engineeringOrg trails & immutable logging

Org trails & immutable logging

One trail to a locked account, tamper-evident.

Advanced30 min · lesson 14 of 15

When an attacker gets into an AWS account, the careful ones go quiet fast. First thing they do is kill the logging. If nothing wrote down what they touched, nobody can prove the break-in happened, and nobody can trace how they got in. So treat your audit trail (the running record of who did what) as the one system you have to assume they'll come for. Two rules fall out of that. It can't sit anywhere they can reach. And it can't be something they can quietly edit later. A good security-camera setup works the same way. The camera watches every door. The recorder lives in a locked room in a separate building, so nobody standing in the lobby can rip the tape out. And every tape gets a tamper-evident seal, so if it ever lands in court, you can show that nobody spliced it.

One trail, recording every account

AWS keeps that record through a service called CloudTrail. Every API call (an API call is just an instruction sent to AWS: create a user, open a firewall rule, read a secret) gets logged as a line saying who made it, from where, and when. The catch with the default setup is that each account keeps its own trail. Own the account and you own the record of your own break-in. An organization trail closes that gap. You define a single trail up in the management account (the account that sits at the top of your org and owns all the others), turn on --is-organization-trail, and it captures activity from every account in the organization, in every region, then drops all of it into one S3 bucket you control. S3 is Amazon's cloud file storage, basically a giant shared folder in the cloud. That bucket lives inside a dedicated log-archive account that runs no workloads and that almost nobody can sign into. The accounts being watched have no route to the place doing the watching. Management events, the calls that change how things are set up, are recorded by default. Data events, the individual reads and writes to things like files in S3 or Lambda functions, cost extra, so you turn those on where the risk is worth the money.

define the org trail and confirm it is delivering
$ aws cloudtrail create-trail --name org-trail \
    --s3-bucket-name acme-org-audit \
    --is-organization-trail --is-multi-region-trail \
    --enable-log-file-validation --kms-key-id alias/cloudtrail-cmk
{
    "Name": "org-trail",
    "S3BucketName": "acme-org-audit",
    "IncludeGlobalServiceEvents": true,
    "IsMultiRegionTrail": true,
    "TrailARN": "arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-trail",
    "LogFileValidationEnabled": true,
    "IsOrganizationTrail": true,
    "KmsKeyId": "arn:aws:kms:eu-west-1:111122223333:key/8f2b1c0a-4d3e-49aa-9c11-7e6b2f0d5a44"
}
 
$ aws cloudtrail start-logging --name org-trail
 
$ aws cloudtrail get-trail-status --name org-trail
{
    "IsLogging": true,
    "StartLoggingTime": "2026-07-16T09:12:03.882000+00:00",
    "LatestDeliveryTime": "2026-07-16T09:14:52.331000+00:00",
    "LatestDigestDeliveryTime": "2026-07-16T09:13:40.117000+00:00"
}

A couple of things have to line up before this works. CloudTrail and AWS Organizations need trusted access switched on between them, and you either create the trail from the management account or hand the job to a delegated administrator account, so you're not signing into the org's most powerful account for routine work. Once it's running, any account that joins the org later gets picked up on its own, with no extra setup per account. That's the whole reason to do it at the org level. Putting everything on one trail has a flip side, though. That single trail feeds detection for every account at once, so if it quietly stops delivering, you go blind across the org in one shot. The LatestDigestDeliveryTime field in the status output is your pulse check, and it's why the digest checks and the StopLogging alarms further down aren't optional extras.

A trail is only as trustworthy as the place its logs land. If an admin (or an attacker holding an admin's stolen login) can delete yesterday's logs, the record proves nothing. Stopping that is the job of S3 Object Lock. It puts every delivered log file under a retention rule that S3 enforces itself, so the file can't be overwritten or removed until the clock runs out. In Compliance mode, not even the account's root user (the all-powerful master login that can normally do anything) can take it early. Compare a filing cabinet that locks with a bank safe-deposit box. Anyone holding the cabinet key can still empty the drawer. The safe-deposit box the bank simply won't open before the agreed date, no matter who's asking or how senior they claim to be. Object Lock is the safe-deposit box. You switch it on when the bucket is created, set a default retention period (say seven years in Compliance mode, to match your evidence rules), then confirm it's on and prove it by trying to delete a log that's already been written.

lock the vault, then prove even you cannot delete a log
$ aws s3api create-bucket --bucket acme-org-audit --region eu-west-1 \
    --create-bucket-configuration LocationConstraint=eu-west-1 \
    --object-lock-enabled-for-bucket
{
    "Location": "http://acme-org-audit.s3.amazonaws.com/"
}
 
$ aws s3api put-object-lock-configuration --bucket acme-org-audit \
    --object-lock-configuration '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"COMPLIANCE","Years":7}}}'
 
$ aws s3api get-object-lock-configuration --bucket acme-org-audit
{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled",
        "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Years": 7 } }
    }
}
 
$ aws s3api delete-object --bucket acme-org-audit \
    --key AWSLogs/o-a1b2c3d4e5/111122223333/CloudTrail/eu-west-1/2026/07/16/111122223333_CloudTrail_eu-west-1_20260716T0910Z_9f3c.json.gz \
    --version-id 3sL4kqtJlcpXroDTDmJVBH40Nrjfkd6oCcH
An error occurred (AccessDenied) when calling the DeleteObject operation:
Access Denied because object protected by object lock.

Proving nobody edited the tape

Locking the logs away stops deletion. On its own, though, it doesn't prove the files sitting there are the exact bytes CloudTrail wrote. That's what log file validation is for, the option you switched on earlier with --enable-log-file-validation. Once an hour, CloudTrail writes a digest file: a signed list of every log file it delivered in that window, each one stamped with a cryptographic hash. A hash is a short fingerprint calculated from a file's contents, so change one character in the file and the fingerprint changes too. Every digest also names the digest before it, so the digests link up into a chain. Edit a log file and its fingerprint stops matching the digest. Delete a log file and the digest still lists it, so it shows up as missing. Delete a digest and the next link points at a hole where it used to be. CloudTrail signs each digest with a key that AWS holds and you never see, so an attacker can't forge a clean replacement that passes the check. That signature is what turns a pile of stored logs into evidence. You run validate-logs, it walks the chain, and it tells you in plain numbers whether anything stopped adding up.

validate the digest chain: a clean run
$ aws cloudtrail validate-logs \
    --trail-arn arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-trail \
    --start-time 2026-07-15T00:00:00Z --end-time 2026-07-16T00:00:00Z
Validating log files for trail arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-trail between 2026-07-15T00:00:00Z and 2026-07-16T00:00:00Z
 
Results requested for 2026-07-15T00:00:00Z to 2026-07-16T00:00:00Z
Results found for 2026-07-15T00:13:20Z to 2026-07-15T23:47:11Z:
 
24/24 digest files valid
576/576 log files valid

Now say someone opened one delivered log file, changed a single line to hide a call, and left everything else untouched. Run the exact same command again. The chain does the rest.

re-run after tampering: the edit is exposed
$ aws cloudtrail validate-logs \
    --trail-arn arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-trail \
    --start-time 2026-07-15T00:00:00Z --end-time 2026-07-16T00:00:00Z
Results found for 2026-07-15T00:13:20Z to 2026-07-15T23:47:11Z:
 
24/24 digest files valid
575/576 log files valid, 1/576 log files INVALID (hash value doesn't match)
 
Log file  s3://acme-org-audit/AWSLogs/o-a1b2c3d4e5/111122223333/CloudTrail/eu-west-1/2026/07/15/111122223333_CloudTrail_eu-west-1_20260715T1410Z_7b2e.json.gz  INVALID: hash value doesn't match

There's one more layer, and it's the loudest of the lot. A competent attacker tries to blind the trail before doing anything noisy, so an alert on the calls that touch logging (StopLogging, DeleteTrail, or a change to the bucket policy on the log sink) is one of the highest-value alarms you can wire up. In a real incident it's often the very first thing that fires, and it's exactly where the response you'll build in the next lesson starts.

The tamper-evident audit pipeline
capture
org CloudTrail
one trail, every account and region
management + data events
who did what, and to which objects
isolate
log-archive account
no workloads, near-zero human access
central S3 bucket
the source accounts can't reach it
lock
Object Lock (Compliance)
write-once, even root can't delete early
deny-delete bucket policy
a second lock on the vault door
prove + watch
log file validation
signed, chained hourly digests
validate-logs
run it, get a yes or no on tampering
alert on StopLogging / DeleteTrail
the first alarm in an incident
Isolation keeps the record out of reach and Object Lock keeps it from being deleted. The digest chain is how you prove it was never edited. Miss any one and it isn't evidence.
Quick check
01You enabled --enable-log-file-validation on the org trail. An attacker with write access to the log-archive bucket deletes yesterday's CloudTrail log files. What actually happens?
Incorrect — Validation doesn't block anything. It's detective, not preventive. Stopping deletion is Object Lock's job, not the digest chain's.
Correct — The digest still lists the deleted files, so a re-run flags them as INVALID or missing. To actually prevent the delete you need Object Lock in Compliance mode.
Incorrect — There's no automatic redelivery. Once a file is gone from your bucket it's gone, unless Object Lock or versioning was protecting it.
Incorrect — The opposite. Digests are signed with an AWS-held key and chained together, so a deletion surfaces as a mismatch or a gap, never a clean cover-up.
02In the organization CloudTrail as configured, which activity is recorded by default, and which costs extra to capture?
Incorrect — Data events carry an extra charge; making the trail organization-wide does not make them free.
Incorrect — This reverses the two — management events are the ones recorded by default.
Incorrect — CloudTrail can log data events such as S3 object reads and writes; they simply cost extra to turn on.
Correct — you get management events for free and turn on data events only where the risk justifies the added cost.
03Your log-archive S3 bucket was created months ago without Object Lock. You now need delivered CloudTrail logs to be undeletable even by the account's root user. What does the lesson say you must do?
Incorrect — You cannot turn Object Lock on for a bucket that was created without it through the API; that command only sets a retention rule on a bucket that already has Object Lock enabled.
Correct — Object Lock has to be enabled when the bucket is born, so the usual remedy is a fresh locked bucket with the trail redirected to it.
Incorrect — Object Lock depends on versioning but is not the same thing; versioning alone does not make delivered objects undeletable.
Incorrect — Governance mode lets sufficiently privileged users and root bypass the lock, so it would not stop root — that needs Compliance mode — and you still cannot enable Object Lock on a bucket that was created without it.
Object Lock only works if you turn it on when the bucket is born
You can't switch Object Lock on for a normal bucket after the fact. It has to be enabled when the bucket is created (turning on Object Lock also turns on versioning, which it depends on). Enabling it on a bucket that already exists means opening an AWS Support request, so in practice a log bucket built without it usually gets replaced by a fresh bucket with the trail re-pointed at it. And once you pick Compliance mode with a retention period, nobody (not your admins, not root, not AWS Support) can delete those objects early, and you pay to store every log for the full term. Pick the retention to match your legal and forensic needs before you flip it on, because you're committing to it.

Try this

Work through “Proving nobody edited the tape” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: object Lock only works if you turn it on when the bucket is born. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

SCP guardrails & landing zonesIncident response & forensics

Related

IAM mental modelAWS security for DevOps engineersMulti-account & org identity architectureAdvanced cloud securityResource hierarchy & org policyGCP securityEntra ID, Conditional Access & PIMAzure security