X-Ray & distributed tracing

Trace request flow across services.

A request moving through a microservice or serverless system behaves like a parcel in a courier network. It gets scanned at the drop-off point, routed through two or three sorting hubs, handed to a van, and finally left at a door. When it turns up a day late, no single hub's log can tell you where the delay happened. You need the tracking timeline that stitches every scan together in order. AWS X-Ray (Amazon Web Services' distributed tracing service) is that timeline for one request as it crosses API Gateway (the managed front door that receives application programming interface calls), Lambda (code that runs without servers you manage), DynamoDB (Amazon's key-value database), and your own downstream services. A CloudWatch metric tells you something is slow. A log line hands you one exception. A trace shows you which hop in the flow actually broke. This lesson stays command-first: you switch on instrumentation, drive the X-Ray API from the CLI (command-line interface) to find a failing hop, then tune sampling so tracing does not wreck your bill or your quota.

What a trace actually records

The tracking number on the parcel is the whole model. X-Ray treats one request as a trace, identified by a trace ID that the first instrumented service generates and every downstream service passes along in the 'X-Amzn-Trace-Id' HTTP (HyperText Transfer Protocol) header. The ID has a fixed shape, '1-{8-hex-epoch-seconds}-{24-hex-random}', for example '1-67a1c2f3-4b9d0e6a1f...', where the middle field encodes the second the trace started. Every service that touches the request writes a segment: a JSON (JavaScript Object Notation) document holding the service name, start and end time, HTTP status, and the 'error' / 'fault' / 'throttle' flags. A 'fault' is a 5xx, a server-side failure this service caused. An 'error' is a 4xx, a bad request. A 'throttle' is a 429, rate-limited. Work happening inside a segment gets its own subsegment with its own timing, so a DynamoDB call, an outbound HTTP request or one slow block of code each show up as a separate bar. Roll thousands of segments together and X-Ray draws the service map: a graph whose nodes are services and whose edges are calls, with latency percentiles and error rates on each edge. That map is the payoff. It turns 'the app is slow' into 'the payment service's outbound call is timing out on 4% of requests'.

Get the services emitting segments

X-Ray shows you nothing until your services emit segments. There is no switch on the X-Ray side that conjures data out of an uninstrumented app. You have two routes. The X-Ray SDK (software development kit), which is AWS-specific, or OpenTelemetry through the AWS Distro for OpenTelemetry, known as ADOT, the vendor-neutral standard AWS now points people toward so your instrumentation keeps working if you ever send traces to a different backend. Managed services make it close to free: Lambda writes segments the moment you flip on active tracing, and API Gateway writes them once you enable tracing on the stage. On EC2 (Elastic Compute Cloud) or ECS (Elastic Container Service) you run a sidecar beside your app, either the older X-Ray daemon listening on UDP (User Datagram Protocol) port 2000 or an ADOT collector, which batches segments and ships them to the X-Ray API. Whichever route you pick, the IAM (Identity and Access Management) execution role your code runs under needs 'xray:PutTraceSegments' and 'xray:PutTelemetryRecords'. Both come bundled in the 'AWSXRayDaemonWriteAccess' managed policy. Miss them and your segments are rejected in silence, with nothing in the application logs to explain the empty service map.

# Turn on active tracing for a Lambda function
aws lambda update-function-configuration \
  --function-name checkout \
  --tracing-config Mode=Active
 
# --- response (trimmed) ---
# {
#   "FunctionName": "checkout",
#   "TracingConfig": { "Mode": "Active" },   # Active | PassThrough
#   "LastUpdateStatus": "InProgress"
# }
 
# Give the role write permission if it lacks it
aws iam attach-role-policy \
  --role-name checkout-role \
  --policy-arn arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess

Read the map, then hunt the failing hop

With segments flowing, you can run the whole investigation from a terminal. 'get-service-graph' returns the aggregate map for a time window. 'get-trace-summaries' searches individual traces using a filter expression, a small query language with terms like 'fault = true', 'responsetime > 2', 'service("payment-svc")' or 'annotation.orderId = "..."'. 'batch-get-traces' pulls the full segment tree for a single trace so you can read the timeline hop by hop. The order never changes: start at the map to spot the red node, filter the summaries down to the traces that faulted, then open one of those traces and see which subsegment ate the latency budget. Watch the arguments, because they are not the same. 'get-service-graph' and 'get-trace-summaries' take epoch-second '--start-time' and '--end-time' bounds. 'batch-get-traces' instead takes the '--trace-ids' you copied out of the summaries.

# 1) Aggregate service map for a 1-hour window (epoch seconds)
aws xray get-service-graph \
  --start-time 1783987200 --end-time 1783990800 \
  --query 'Services[].{svc:Name, edges:length(Edges), faults:SummaryStatistics.FaultStatistics.TotalCount}'
 
# [
#   { "svc": "checkout",    "edges": 2, "faults": 0   },
#   { "svc": "DynamoDB",    "edges": 0, "faults": 0   },
#   { "svc": "payment-svc", "edges": 0, "faults": 214 }   <- the red node
# ]
 
# 2) Find the faulting traces with a filter expression
aws xray get-trace-summaries \
  --start-time 1783987200 --end-time 1783990800 \
  --filter-expression 'fault = true AND service("payment-svc")' \
  --query 'TraceSummaries[].{id:Id, sec:ResponseTime, status:Http.HttpStatus}'
 
# [
#   { "id": "1-67a1c2f3-4b9d0e6a1f...", "sec": 3.42, "status": 500 },
#   { "id": "1-67a1c2f8-9c1a2b3d4e...", "sec": 3.39, "status": 500 }
# ]

# Pull the full segment tree for one faulting trace
aws xray batch-get-traces \
  --trace-ids 1-67a1c2f3-4b9d0e6a1f... \
  --query 'Traces[0].Segments[].Document' --output text
 
# Each Document is a JSON segment. Rendered as a timeline, the console draws:
#   API Gateway            12 ms   ok
#   checkout (Lambda)      45 ms   ok
#     |- DynamoDB GetItem   8 ms   ok
#     |- payment-svc call  380 ms  FAULT   "error": true, cause: connection timeout
#
#   => 380 ms on ONE downstream subsegment = the whole latency + error budget.
#      No single service's own metrics could have located this.

Sampling: what you keep, and what it costs

Airport security cannot pat down every passenger, so it pulls a fixed few plus a random percentage of the rest. X-Ray works the same way, with one twist. It uses head-based sampling, which means the keep-or-drop decision is made at the very first service, as the trace starts, before anybody knows whether the request will fail. The built-in rule records the first request each second (the reservoir) plus 5% of everything above that, enough to see patterns without recording every call. You change that with sampling rules, 25 per Region by default and adjustable on request. Each rule carries a priority, a reservoir size and a fixed rate, and can match on service, URL path or HTTP method, so tracing 100% of '/checkout' while leaving health checks at 1% is a two-rule job. Sampling is also the dial on your bill and your quota. Recorded traces cost $5.00 per million, with the first 100,000 each month free. Retrieving or scanning them through the CLI or console costs $0.50 per million, first 1,000,000 free. Switching on X-Ray Insights adds $1.00 per million analyzed. The ingestion ceiling is the part people forget: X-Ray accepts at most 2,600 segments per second per Region, so cranking sampling to 100% under real load gets you throttled, which drops the very traces you were trying to keep. Retention is a fixed 30 days, and a single segment document cannot exceed 64 KB (kilobytes).

# See the active sampling rules (the built-in 'Default' rule is always present)
aws xray get-sampling-rules \
  --query 'SamplingRuleRecords[].SamplingRule.{name:RuleName, rate:FixedRate, reservoir:ReservoirSize, prio:Priority}'
 
# [
#   { "name": "Default", "rate": 0.05, "reservoir": 1, "prio": 10000 }
# ]
 
# Add a high-priority rule: trace 100% of /checkout, keep a 5/sec reservoir
aws xray create-sampling-rule --sampling-rule '{
  "RuleName": "trace-checkout", "Priority": 100,
  "FixedRate": 1.0, "ReservoirSize": 5,
  "ServiceName": "*", "ServiceType": "*", "Host": "*",
  "HTTPMethod": "*", "URLPath": "/checkout*",
  "ResourceARN": "*", "Version": 1
}'
 
# {
#   "SamplingRuleRecord": {
#     "SamplingRule": { "RuleName": "trace-checkout", "FixedRate": 1.0, "Priority": 100 }
#   }
# }

Annotations, metadata, and tying the three signals together

Custom data on a segment comes in two kinds, and picking the wrong one costs you later. Annotations are indexed key-value pairs, up to 50 per trace, and X-Ray makes them searchable inside filter expressions, so attach the business keys you will actually query by: 'orderId', 'tenant', 'customerTier'. Metadata is free-form JSON that rides along with the trace but is never indexed, so you can read it once you have found the trace and you can never search on it. For the case where a support ticket names one exact request and sampling threw that trace away, CloudWatch Transaction Search (turned on through Application Signals) indexes 100% of spans into CloudWatch Logs, decoupled from the trace sampling rate, so the order ID still resolves. X-Ray also lives inside the CloudWatch console now. ServiceLens ties each service-map node to its metrics and its logs, so you can move from a latency spike on the graph, to the trace behind it, to the exact log line, in a few clicks. Metrics tell you something broke, logs tell you what the error said, traces tell you where it happened, and here all three sit in one view.

Sampling rules protect both your bill and your ingestion quota. Full 100% tracing in production is rarely worth what it costs. Raise the rate on the route you are chasing while an incident is live, then put it back down once the incident closes.

Annotations get indexed, metadata does not. High-cardinality values, a raw customer ID being the classic one, belong in metadata. Push them into annotations and you have invented a new cost center for search you will barely use.

A service map built on partial instrumentation lies to you quietly. A hop with no SDK on it does not show up as broken, it does not show up at all, and an absent node reads exactly like a healthy one. Instrument the services you keep blaming during incidents.

Try this

Pull a few recent trace summaries, then open one trace and read its duration breakdown. You need X-Ray data already flowing in the lab account for this to return anything.

aws xray get-trace-summaries --start-time $(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --query 'TraceSummaries[:3].{Id:Id,Duration:Duration,HasError:HasError}' --output table
aws xray batch-get-traces --trace-ids 1-687ea1a2-abcdef012345678901234567 \
  --query 'Traces[0].{Id:Id,Duration:Duration,Segments:Segments[].Document}' --output json | head -c 400

---------------------------------------------
|           GetTraceSummaries               |
+----------------------+----------+---------+
|          Id          | Duration | HasError|
+----------------------+----------+---------+
| 1-687ea1a2-abcd...   | 1.84     | True    |
+----------------------+----------+---------+
# Segment documents show which subsegment owned the latency

Takeaway

Remember: a trace stitches the hops together so you stop guessing which service was slow. Sample on purpose, and instrument the path you get paged about.

Next: add X-Ray (or OpenTelemetry feeding X-Ray) to one Lambda and API Gateway path, then check that the service map draws the DynamoDB or outbound HTTP hop behind it.

Tracing gives you the guided walk down the failing path: red node on the map, filter to the faulting traces, open one, read the subsegment that burned the budget. What it will not do is act on what it found. In EventBridge & orchestration you wire these signals into event-driven workflows, so a fault X-Ray surfaced can kick off remediation with nobody watching the console.

Related