High availability & DR

Multi-AZ, multi-Region, RTO/RPO strategies.

A hospital doesn't stay open through a power cut on good intentions. There are diesel generators in the basement that pick up the load within seconds, and for something far worse there is a standing agreement with a second hospital across town. Resilience on AWS (Amazon Web Services) works on those same two levels. High availability (HA) is the generator: spare capacity *inside* one Region that soaks up a component failure automatically, in seconds, with nobody woken up. Disaster recovery (DR) is the other hospital: data already sitting somewhere else, plus a rehearsed plan for carrying on when an entire Region is unusable. People mix the two up constantly. The exam punishes that, and so do real outages.

Both ideas rest on one assumption, and it comes straight from the Reliability pillar of the Well-Architected Framework: *everything fails, all the time*. Disks die. An Availability Zone (AZ, one of the physically separate data center clusters that make up a Region) loses power. Whole Regions have had multi-hour bad days. So you don't design to stop failure happening. You design so failure is either swallowed invisibly (HA) or recovered from inside limits you promised in advance (DR). Those promises come down to two numbers.

RTO and RPO: the two numbers everything else hangs off

Recovery Time Objective (RTO) is how long the business can stand being down, the stopwatch running from the moment things break to the moment service is back. Recovery Point Objective (RPO) is how much data the business can stand losing, and you measure that one backwards: if the newest usable copy is 15 minutes old when disaster hits, up to 15 minutes of writes are gone. Both numbers are business decisions. Your job is to put a price on each. RPO is capped by how often data leaves the blast radius, so an hourly snapshot means an RPO of one hour, while synchronous replication (every write landing in two places before it counts as done) pushes it near zero. RTO is capped by how much of the recovery happens without a human. Someone restoring from backups by hand means hours. Pre-provisioned capacity behind automatic DNS (Domain Name System, the internet's address book) failover means minutes.

Exam tell. When a question states an RPO of minutes, backups on their own are out and you need continuous replication. When it states an RTO of minutes, rebuilding infrastructure from scratch is out and something must already be running in the recovery Region.

High availability: make Multi-AZ prove it

Inside a single Region the HA recipe barely changes: stateless compute in an Auto Scaling group spread over at least two AZs, a load balancer in front handing traffic to whichever ones are alive, and a Multi-AZ database behind. The database is the hard part, because it is the piece holding state. RDS (Relational Database Service) Multi-AZ keeps a synchronous standby in a second AZ. Every commit is written to both machines before your application is told it worked, so a failover loses nothing (RPO ≈ 0) and finishes in a typical 60 to 120 seconds. AWS does that by repointing the instance endpoint's DNS name at the standby, so applications reconnect to the same address and carry on. The standby answers no queries of its own. Compare it with read replicas, which copy data *asynchronously* and exist to spread read traffic or to seed a cross-Region DR copy. The exam hammers this distinction: synchronous standby means HA, asynchronous replica means scaling and DR.

Don't take the checkbox in the console at its word. Confirm the standby is really there, then force a failover inside a maintenance window and time it with an actual stopwatch:

# Is prod actually Multi-AZ, and where is the standby?
aws rds describe-db-instances \
  --db-instance-identifier prod-api-db \
  --query 'DBInstances[0].{MultiAZ:MultiAZ,Primary:AvailabilityZone,Standby:SecondaryAvailabilityZone}'
{
    "MultiAZ": true,
    "Primary": "eu-west-1a",
    "Standby": "eu-west-1b"
}
 
# Force a failover (maintenance window!) and start the stopwatch
aws rds reboot-db-instance \
  --db-instance-identifier prod-api-db \
  --force-failover \
  --query 'DBInstance.DBInstanceStatus'
"rebooting"
 
# ~90 seconds later the roles have swapped — same endpoint, new AZ
aws rds describe-db-instances --db-instance-identifier prod-api-db \
  --query 'DBInstances[0].AvailabilityZone'
"eu-west-1b"

Whatever number comes back, say 90 seconds, that is your real data-tier RTO for losing an AZ. It is a figure you can defend in an architecture review instead of a hope. (There is a newer deployment option called Multi-AZ DB cluster, which replicates semisynchronously to two *readable* standbys and usually fails over in under 35 seconds. Know the term for the exam.)

The four DR strategies, cheapest first

AWS names four DR strategies, and nearly every DR question on the exam is really asking you to match a stated RTO/RPO to one of them. Backup and restore: copy backups (snapshots, AWS Backup vaults) into the DR Region and run nothing there. It is the cheapest, with RTO and RPO measured in hours by AWS's own figures, and a large restore can run longer than that. Backup frequency sets the RPO. Pilot light: keep the data layer replicating all the time while the rest of the infrastructure is defined but switched off or scaled to zero, like the small flame in a boiler waiting for the burner to be called. On failover you promote the database and start the compute, which puts RTO in the tens of minutes. Warm standby: run a shrunken but fully working copy that could serve reduced traffic this second, then scale it up when you cut over. RTO in minutes. Multi-site active-active: full capacity running in both Regions with traffic split between them. RTO near zero, cost roughly double or more.

The trap is buying too much. Active-active for a workload whose owner shrugs at four hours of downtime burns money and, worse, burns engineering attention that something else needed. Pick the cheapest strategy that clears the stated objectives, and get those objectives in writing.

Get the data out of the Region first: S3 Cross-Region Replication

None of the four strategies does anything for you unless the data is already sitting in the second Region, so replication is step one every time. S3 (Simple Storage Service) Cross-Region Replication (CRR) copies new objects to a bucket in another Region in the background. Two rules catch people out. Versioning, where S3 keeps every previous version of an object instead of overwriting it, has to be switched on for *both* buckets. And replication is not retroactive, so anything already in the bucket before you wrote the rule needs S3 Batch Replication to move it. Most objects land within 15 minutes, but plain CRR promises nothing. If you need a promise, Replication Time Control (RTC) adds a paid SLA (service level agreement, a guarantee with money behind it) that 99.99% of objects replicate within 15 minutes.

# Prerequisite: versioning ON at both ends (CRR refuses without it)
aws s3api put-bucket-versioning --bucket acme-assets \
  --versioning-configuration Status=Enabled
aws s3api put-bucket-versioning --bucket acme-assets-dr \
  --versioning-configuration Status=Enabled --region us-west-2
 
# replication.json
{
  "Role": "arn:aws:iam::111122223333:role/s3-crr-replication",
  "Rules": [{
    "ID": "dr-copy-everything",
    "Status": "Enabled",
    "Priority": 1,
    "Filter": {},
    "DeleteMarkerReplication": { "Status": "Disabled" },
    "Destination": {
      "Bucket": "arn:aws:s3:::acme-assets-dr",
      "StorageClass": "STANDARD_IA"
    }
  }]
}
 
aws s3api put-bucket-replication --bucket acme-assets \
  --replication-configuration file://replication.json
 
# Per-object proof on the source bucket
aws s3api head-object --bucket acme-assets \
  --key invoices/2026-07.pdf --query ReplicationStatus
"COMPLETED"

ReplicationStatus is your per-object paper trail: PENDING, COMPLETED or FAILED on the source object, REPLICA on the copy at the far end. Look at the rule again and notice it deliberately does *not* replicate delete markers. If ransomware or a careless script wipes the source bucket, those deletions never reach your DR copy. One line of JSON, and it is doing the work of a security control.

Failing over: promote the database, then move the traffic

A Regional failover is two moves. Promote the data layer in the DR Region, then send the traffic there. For relational data, Aurora Global Database copies writes across Regions with lag that is usually under a second, which puts RPO in seconds. It also draws a hard line between two operations. A planned switchover is the one you use for drills: it brings the secondary fully in step with the primary before handing over, so nothing is lost. An unplanned failover promotes the secondary immediately and accepts whatever replication lag existed as lost data:

# Planned drill — switchover guarantees zero data loss (RPO = 0)
aws rds switchover-global-cluster \
  --global-cluster-identifier acme-global \
  --target-db-cluster-identifier \
    arn:aws:rds:us-west-2:111122223333:cluster:acme-dr \
  --query 'GlobalCluster.Status'
"switching-over"
 
# Real disaster — promote NOW, accept the lag as data loss
aws rds failover-global-cluster \
  --global-cluster-identifier acme-global \
  --target-db-cluster-identifier \
    arn:aws:rds:us-west-2:111122223333:cluster:acme-dr \
  --allow-data-loss

Traffic follows DNS. Route 53 failover routing hands out the primary Region's endpoint only while a health check is passing. Those checks run from fleets of checkers scattered around the world. Each checker calls the endpoint down after three failed probes in a row (the failure threshold set below), and once 18% or fewer of the checkers still see it as healthy, Route 53 starts answering with the secondary instead. No human in the loop. Keep the TTL (time to live, how long resolvers are allowed to cache an answer) low, 60 seconds here, so the stale answer stops circulating quickly:

# Health check: 3 consecutive failed HTTPS probes of /healthz per checker
aws route53 create-health-check \
  --caller-reference dr-primary-$(date +%s) \
  --health-check-config '{
    "Type": "HTTPS",
    "FullyQualifiedDomainName": "primary.acme.example",
    "ResourcePath": "/healthz",
    "RequestInterval": 30,
    "FailureThreshold": 3
  }' \
  --query 'HealthCheck.Id'
"9e54c2f1-8f3a-4c2e-b1d7-0a12c3d4e5f6"
 
# failover.json — PRIMARY answers while healthy, SECONDARY takes over
{
  "Changes": [
    { "Action": "UPSERT", "ResourceRecordSet": {
        "Name": "app.acme.example", "Type": "CNAME", "TTL": 60,
        "SetIdentifier": "primary", "Failover": "PRIMARY",
        "HealthCheckId": "9e54c2f1-8f3a-4c2e-b1d7-0a12c3d4e5f6",
        "ResourceRecords": [{ "Value": "alb-1948206512.eu-west-1.elb.amazonaws.com" }] } },
    { "Action": "UPSERT", "ResourceRecordSet": {
        "Name": "app.acme.example", "Type": "CNAME", "TTL": 60,
        "SetIdentifier": "dr", "Failover": "SECONDARY",
        "ResourceRecords": [{ "Value": "alb-dr-582910447.us-west-2.elb.amazonaws.com" }] } }
  ]
}
 
aws route53 change-resource-record-sets \
  --hosted-zone-id Z0123456789ABCDEFGHIJ \
  --change-batch file://failover.json \
  --query 'ChangeInfo.Status'
"PENDING"

Look back at what made each of those steps quick. The compute tier never really mattered. Instances could be rebuilt, resized or thrown away in another Region because nothing irreplaceable was living on them, and all the state sat in replicated services like S3 and Aurora. That property has a name, statelessness, and it is what lets an Auto Scaling group treat servers as disposable. It is also where the next lesson starts: Auto Scaling and stateless design.

RTO and RPO are business numbers wearing technical clothes. Backup and restore is cheap and slow. Pilot light and warm standby spend money to buy back minutes. Multi-Region active-active spends the most of all, and you still owe someone a rule for what happens when both Regions accept a conflicting write.

Multi-AZ inside one Region is HA, not DR. The moment an exam question says "Region impairment," it wants cross-Region data plus a DNS or traffic cutover you have actually rehearsed.

Try this

Two commands, covering the two most common HA and DR building blocks: check whether an RDS instance is genuinely Multi-AZ, then check a bucket's versioning and replication settings.

aws rds describe-db-instances --db-instance-identifier lab-pg \
  --query 'DBInstances[0].{MultiAZ:MultiAZ,AZ:AvailabilityZone,Secondary:SecondaryAvailabilityZone}' --output table
aws s3api get-bucket-versioning --bucket lab-dr-source
aws s3api get-bucket-replication --bucket lab-dr-source --query 'ReplicationConfiguration.Rules[].{ID:ID,Status:Status,Dest:Destination.Bucket}' --output table

---------------------------------
|      DescribeDBInstances      |
+------+----------+-------------+
|  AZ  | MultiAZ  | Secondary   |
+------+----------+-------------+
| use1-az1 | True | use1-az2  |
+------+----------+-------------+
{"Status": "Enabled"}
rule-1 | Enabled | arn:aws:s3:::lab-dr-replica

Takeaway

Remember it this way. HA absorbs the loss of an AZ inside a Region. DR survives the loss of the Region itself. Both are graded against RTO and RPO figures you chose before the outage, not during it.

Next: write down an RTO and an RPO for one store you would hate to lose, pick the cheapest strategy that clears them, and put a failover drill on the calendar.

Related