CloudWatch metrics, logs & alarms
Alarm on signals; drive automated actions.
A pilot flying through thick cloud stops looking out of the window and reads the instrument panel instead. Amazon CloudWatch is that panel for the systems you run. It gathers metrics (plain numbers measured over time, like how many milliseconds a request took), logs (the written record of what actually happened, line by line), and it links out to traces (the path one request takes across your services, which is the next lesson). Sitting on top of all that raw data are alarms: standing rules that watch one number and fire an action the moment it crosses a line you set. Collecting data is the easy part. The skill is picking the *right* signals to emit and wiring alarms that *act* on their own, so the system fixes itself before anyone's phone rings at 3am.
Measure what your users feel, not what the servers feel
A rental car's dashboard tells you the engine temperature and the fuel level. It tells you nothing about whether your passengers are comfortable. CloudWatch has the same blind spot out of the box. It picks up metrics from most AWS services for free: request counts on an ALB (Application Load Balancer, the thing that spreads incoming traffic across your servers) and Lambda error counts every minute, EC2 (Elastic Compute Cloud, plain virtual machines) CPU every five minutes under basic monitoring, with one-minute CPU available only if you pay for detailed monitoring. Those are engine readings. A box can sit at 12% CPU while every checkout running on it returns an error. The CloudWatch agent, a small program you install on the machine itself, fills in what the hypervisor cannot see from outside (memory in use, a disk filling up) and forwards your application's log files. The numbers that actually map to an SLO (service level objective, the promise you have made about how fast or how reliable the service is) live inside your application: the rate of HTTP 5xx responses, how long a checkout takes, how deep the work queue is. You publish those yourself as custom metrics, and there are two ways to do it. You can call the PutMetricData API once per data point, or you can write EMF (Embedded Metric Format) logs, which are single lines of structured JSON (JavaScript Object Notation, a plain-text way of writing nested data) that CloudWatch reads and turns into a metric on its own. The second way scales far better, because there is no extra API call sitting there waiting to be throttled or billed.
# Push an application-level custom metric (one API call per data point):aws cloudwatch put-metric-data \--namespace MyApp/Checkout \--metric-name Http5xxRate --unit Percent --value 0.4 \--dimensions Service=checkout,Env=prod# (returns nothing; exit code 0 on success)# Better at scale — emit EMF. Your app writes this one JSON log line and# CloudWatch extracts a metric from it for free, no PutMetricData to throttle:{ "_aws": {"Timestamp": 1752480000000,"CloudWatchMetrics": [{"Namespace": "MyApp/Checkout","Dimensions": [["Service", "Env"]],"Metrics": [{ "Name": "LatencyMs", "Unit": "Milliseconds" }]}] },"Service": "checkout", "Env": "prod", "LatencyMs": 428 }
Alarms: the statistic, the state machine, and the gaps
A smoke alarm that shrieks every time you make toast gets taped over inside a week. CloudWatch alarms fail the same way, and two settings decide whether yours is trusted or taped over. An alarm takes one metric, squashes each period of raw data down to a single number using a statistic, compares that number to a threshold, and waits to see the breach repeat across a number of evaluation periods before it does anything. The first setting is the statistic. Average is a great flattener: if 99 requests come back in 40ms and one takes 9 seconds, the average still looks lovely, and the customer who waited 9 seconds is invisible. Alarm on a percentile instead, using --extended-statistic p99, which reports the value only the slowest 1% of requests exceed. The second setting is M-of-N. --datapoints-to-alarm 3 --evaluation-periods 5 means fire only when 3 of the last 5 periods were bad, which shrugs off one ugly minute while still catching a real problem quickly. An alarm is always sitting in exactly one of three states: OK, ALARM, or INSUFFICIENT_DATA. Which one it lands in when a period has no data at all is decided entirely by --treat-missing-data.
# Alarm on p99 latency: 3-of-5 datapoints, and treat gaps as "not breaching"aws cloudwatch put-metric-alarm \--alarm-name checkout-p99-latency \--namespace MyApp/Checkout --metric-name LatencyMs \--extended-statistic p99 --period 60 \--threshold 800 --comparison-operator GreaterThanThreshold \--evaluation-periods 5 --datapoints-to-alarm 3 \--treat-missing-data notBreaching \--dimensions Name=Service,Value=checkout Name=Env,Value=prod \--alarm-actions arn:aws:sns:us-east-1:111122223333:oncallaws cloudwatch describe-alarms --alarm-names checkout-p99-latency \--query 'MetricAlarms[0].{State:StateValue,Reason:StateReason}'# {# "State": "OK",# "Reason": "Threshold Crossed: 3 out of the last 5 datapoints were# not greater than the threshold (800.0)"# }
Service or Env and you end up with a handful of metrics. Attach userId, requestId, or a raw URL path and *every distinct value* mints a brand new metric of its own. One deploy that adds a sessionId dimension can take you from 5 metrics to millions, which is a five-figure invoice, and the data is worthless on top of that, because no single one of those metrics has enough datapoints to alarm on. Keep dimensions small and predictable. Per-request identifiers belong in your logs, where you pay by the gigabyte instead of by the unique label.Search the logs without grepping a terabyte
An alarm tells you the building is on fire. It does not tell you which room. For that you need the logs, and opening raw log streams and scrolling stops working somewhere around the first gigabyte. CloudWatch Logs Insights is a small query language built for exactly this: point it at a log group, filter, pull named fields out of each line, and add things up across gigabytes in a few seconds. Queries run asynchronously, a bit like ordering a book from a library's back rooms. start-query hands you a ticket (a queryId) straight away, and you keep calling get-query-results until the status comes back Complete. You are billed per GB scanned, not per row returned, so a lazy time range is the expensive part. And if a number is something you want to alarm on all day long, do not re-scan the logs every minute to get it. Extract it once with a metric filter, which watches new log events as they land and turns every match into a real metric. Setting defaultValue=0 makes it publish a zero on quiet periods too, which walks you neatly around the sparse-metric trap in the diagram above.
# Async query across the last hour — returns a queryId immediately:aws logs start-query \--log-group-name /aws/lambda/checkout \--start-time $(date -d '1 hour ago' +%s) --end-time $(date +%s) \--query-string 'fields @timestamp, @message| filter @message like /ERROR/| stats count() as errors by bin(5m)'# { "queryId": "a1b2c3d4-5e6f-7890-abcd-ef1234567890" }aws logs get-query-results --query-id a1b2c3d4-5e6f-7890-abcd-ef1234567890# {# "status": "Complete",# "statistics": { "recordsScanned": 184320, "bytesScanned": 61403136 },# "results": [[ {"field":"bin(5m)","value":"2026-07-14 09:05:00"},# {"field":"errors","value":"37"} ]]# }# Alarm continuously without re-scanning: extract the signal as a metricaws logs put-metric-filter --log-group-name /aws/lambda/checkout \--filter-name error-count --filter-pattern 'ERROR' \--metric-transformations \metricName=ErrorCount,metricNamespace=MyApp/Checkout,metricValue=1,defaultValue=0
Wire the alarm to something that acts
A burglar alarm that only makes a noise is worth far less than one wired to a control room that sends someone round. The graph is not the payoff. The action hanging off it is. Whatever you put in --alarm-actions runs within seconds of the state change, with nobody awake: send a notification through SNS (Simple Notification Service, the AWS fan-out messaging service), add capacity to an Auto Scaling group, start an SSM (Systems Manager) Automation runbook that performs the fix you would otherwise have typed by hand, or trigger a CodeDeploy rollback to the last good version. Give that automation role only the specific actions its runbook needs, because an alarm that can restart one service is useful and an alarm that can do anything is a liability. To keep the pager meaningful, combine signals with composite alarms: wake a human only when latency *and* error rate are both bad at the same time, and attach an --actions-suppressor so a deploy-in-progress alarm silences its children while a rollout is still in flight. If you run many accounts, route alarm state into one of them so you can spot a shared dependency wobbling everywhere at once. Then keep pruning. Every low-value page teaches somebody to stop looking at the panel.
# Page on-call only when BOTH latency and 5xx are bad, and suppress# during an active deploy so a rollout doesn't wake anyone:aws cloudwatch put-composite-alarm \--alarm-name checkout-page-oncall \--alarm-rule "ALARM('checkout-p99-latency') AND ALARM('checkout-5xx')" \--actions-suppressor arn:aws:cloudwatch:us-east-1:111122223333:alarm:deploy-in-progress \--actions-suppressor-wait-period 120 \--alarm-actions arn:aws:sns:us-east-1:111122223333:pagerduty# (returns nothing; exit code 0 on success)
What it costs, what it caps, and how long it keeps
Three line items dominate the bill, the way a phone plan really comes down to calls, data, and roaming. Custom metrics run about $0.30 each per month, so cardinality (see the warning above) is where the money actually leaks. Log ingestion is $0.50/GB on the Standard class ($0.25 on Infrequent Access) plus $0.03/GB-month to keep it, and every Logs Insights query adds roughly $0.005 for each GB it *scans*, which is why narrow time windows and metric filters pay for themselves quickly. Alarms are $0.10 each per month, $0.30 for high-resolution ones, $0.50 for composite. The limits worth designing around: 5,000 alarms per region (you can ask for more), PutMetricData accepts at most 1,000 metrics and 1 MB in a single request, and one Logs Insights query reaches across up to 50 log groups and hands back 10,000 rows. Retention happens whether you asked for it or not, and it is tiered: high-resolution data survives 3 hours, 1-minute data 15 days, 5-minute data 63 days, 1-hour data 15 months. CloudWatch rolls older points up into coarser ones as they age, so do not go hunting for per-second detail from last month. It is already gone.
The three knobs that decide whether an alarm is honest are the statistic, the period, and the missing-data rule, and the last one has no universally correct setting. notBreaching keeps a sparse error metric quiet, and it will keep exactly as quiet if the process publishing that metric has died. breaching catches the dead publisher, and it will also page you at 2am because the staging service went dark overnight, working precisely as designed. Choose per alarm and per environment. If you pick notBreaching on something that matters, pair it with a separate heartbeat alarm whose only job is to notice the silence.
Logs Insights only feels like a superpower if your logs are structured. A line of free-form text forces you to invent a regular expression at 3am under pressure. A JSON line gives you named fields you can filter and group by straight away. Once the fields are queryable, stats count() by bin(5m) stops being a query and starts being a shape: you can see the exact five-minute bucket where errors went vertical, and it is usually the same five minutes as somebody's deploy.
Wiring an action is half the work. Rehearsing it is the other half. An Auto Scaling policy, an EC2 recovery action, or an SSM runbook that has never run in anger will pick your worst incident as the moment to reveal that its role is missing one permission. Force a breach in a lab account, watch the whole chain fire end to end, and read the runbook's own output afterwards. An untested action is decoration with a price tag.
Pick the small set of numbers a non-engineer at your company would recognize on sight: checkout latency, failed payments per minute, signups completed. Those are the ones worth alarming on and the only ones worth paging on. Nobody has ever been thanked for reporting that CPU held steady at 12% throughout the outage. Emit them from inside the application, where you know whether the request truly succeeded, and keep the list short enough that every metric has a named owner who would notice if it stopped moving.
Log class and retention decide most of your Logs bill, and both are set per log group, so choose them the day you create the group rather than a year into the invoices. Audit trails you have to keep for years belong on Infrequent Access, where storage is cheaper and you accept that you will rarely query them. A chatty debug stream should have a retention measured in days, because nobody reads week-old debug output. And before anyone sets a debug group to ten years "for compliance", go and ask compliance. The real answer is usually a much shorter number attached to a much smaller set of logs.
Treat a noisy alarm as a bug with a ticket, not as background weather. If it fires every Tuesday and the response is always to close it unread, either the threshold is wrong or the alarm should not exist. Deleting it is a legitimate fix. A pager that is quiet because you removed a bad rule is honest, and you know precisely what you are blind to. A pager everyone has muted looks like coverage and is not.
The classic ingestion shock comes from flipping an application to debug logging everywhere and then forgetting about it. Ten times the log volume at $0.50 a gigabyte adds up fast, and almost none of those lines will ever be read by a human. Sample instead. Log a small percentage of normal requests in full, and turn verbosity all the way up only for requests carrying a particular trace id, so you get complete detail on the ones you are chasing and nothing extra for the rest.
Give every dashboard an owner and one question it answers. "Is checkout failing right now?" is a question. "Is latency climbing?" is a question. "Is a dependency saturated?" is a question. A wall of CPU charts for twelve services answers none of them, and it is exactly how you scroll straight past the one red panel that mattered. When a dashboard stops being opened during incidents, delete it instead of letting it rot in the sidebar.
Try this
Three commands, about a minute of your time. Publish one custom metric, read the current state of an alarm that already exists, and fire a Logs Insights query at a lab log group so you can see the ticket it hands back.
aws cloudwatch put-metric-data --namespace Lab/App --metric-name CheckoutLatencyMs --value 240 --unit Millisecondsaws cloudwatch describe-alarms --alarm-names app-5xx --query 'MetricAlarms[0].{State:StateValue,Metric:MetricName,Threshold:Threshold}' --output tableaws logs start-query --log-group-name /aws/lambda/saa-echo \--start-time $(date -d '1 hour ago' +%s) --end-time $(date +%s) \--query-string 'fields @timestamp, @message | filter @message like /ERROR/ | limit 20'
-----------------------------| DescribeAlarms |+--------+---------+--------+| Metric | State |Thresh |+--------+---------+--------+| 5xx | OK | 5 |+--------+---------+--------+# start-query returns a queryId; get-query-results shows rowsqueryId: abcd-1234-...
Takeaway
If you change one thing after this lesson, change the missing-data rule on your sparsest alarm. That single setting is the difference between an alarm that speaks up when a metric goes quiet and one that sits in INSUFFICIENT_DATA right through an outage, looking perfectly calm.
Next, in a lab account: hang an action off one alarm, either an SSM document or an SNS notification, then force the metric past its threshold and watch the whole chain fire.
--statistic Average --threshold 800 and has never once fired, yet users keep reporting slow requests during traffic spikes. What is the most likely explanation?--extended-statistic p99 (or p95) so the alarm sees the latency your users actually sit through.Metrics tell you *that* checkout got slow. Logs tell you *what* the error said. Neither one tells you *where* those 400 milliseconds actually went in a chain of a dozen services. Following a single request through every service it touches, and picking out the one slow hop, is called distributed tracing, and AWS X-Ray is what gives you that next.