Systems Manager

Session Manager, Parameter Store, Patch Manager.

Every server in your estate has a quiet clerk sitting inside it. The clerk never opens the front door to the street. It phones head office over an encrypted line, asks whether there is any work for it, does the work, and files a report. That clerk is the SSM Agent (the Systems Manager agent, a small program that runs on the machine), head office is AWS Systems Manager (Amazon Web Services' fleet operations service), and the machines it looks after are called managed nodes. The direction of that phone call is the whole trick. The node reaches out; nothing reaches in. It is why Systems Manager ends up being the thing a professional-level DevOps engineer touches a dozen times before lunch: shell access with no keys, one command across a whole fleet, config and secrets in one place, patching on a schedule. Every one of those actions is scoped by IAM (Identity and Access Management, the AWS permission system) and written to a log.

The agent model: why nothing on your instance is listening

The agent already ships on current AMIs (Amazon Machine Images, the disk templates your instances boot from): Amazon Linux 2 and 2023, recent Ubuntu, Windows Server. It runs as an ordinary background process and calls out to the Systems Manager service over TCP (Transmission Control Protocol) port 443, the same encrypted port your browser uses. The instance is a client, never a server. No inbound rule. No open port 22. No bastion. Three things have to be true for that outbound call to land. The node needs an instance profile (an IAM role bolted onto an EC2 (Elastic Compute Cloud) virtual machine) carrying the managed policy AmazonSSMManagedInstanceCore. It needs a network path to three service endpoints: ssm (the control API, or application programming interface), ssmmessages (the Session Manager channel), and ec2messages (the Run Command channel). And the agent has to actually be running. In a public subnet, the path is an internet gateway or a NAT gateway (Network Address Translation, the box that lets private machines call out without being reachable themselves). In a locked-down private subnet with no NAT at all, you create VPC interface endpoints (private doorways into an AWS service that live inside your own network) for those three services. Miss any one of the three and the node never appears in your fleet list. Nothing errors. It is absent, and that is all the feedback you get. That silent absence is the most common Systems Manager support ticket there is, so the first command worth memorising is the one that tells you who actually phoned in.

# Which nodes has the SSM Agent successfully registered? (private-subnet sanity check)
aws ssm describe-instance-information \
  --query "InstanceInformationList[].{Agent:AgentVersion,Id:InstanceId,Ping:PingStatus,Platform:PlatformName}" \
  --output table
 
# -------------------------------------------------------------
# |                 DescribeInstanceInformation               |
# +-------------+-----------+----------+------------------------+
# |   Agent     |    Id     |   Ping   |        Platform        |
# +-------------+-----------+----------+------------------------+
# | 3.3.1611.0  | i-0abc12  | Online   | Amazon Linux           |
# | 3.3.1611.0  | i-0def34  | Online   | Ubuntu                 |
# +-------------+-----------+----------+------------------------+
#
# A node missing here has no route to ssm/ssmmessages/ec2messages,
# is missing the AmazonSSMManagedInstanceCore instance profile,
# or the agent isn't running. No error is raised — it is just absent.

Session Manager: a shell you can audit, with nothing open

A bastion host is a doorman you have to keep paying, keep patching, and keep hoping nobody picks the lock on. SSH (Secure Shell, the classic remote-login protocol) key pairs and inbound port 22 rules come with the same bill. Session Manager retires all of it. start-session opens a two-way shell that rides the same outbound 443 channel the agent already uses, so there is still nothing exposed. IAM decides who may open a session and on which machines. Put a condition on ssm:resourceTag/Env: dev in the policy and a developer reaches the dev boxes and nothing else. Sessions can be streamed to CloudWatch Logs or S3 (Simple Storage Service, the AWS object store), keystroke-logged, encrypted with a KMS (Key Management Service) key, and pinned to a specific operating system user through the SSM-SessionManagerRunShell preferences document. Port forwarding lets you tunnel through a node to a private RDS (Relational Database Service) database, so that database never needs a public endpoint of its own. Session Manager costs nothing; you pay for the log storage. The only thing you install on your laptop is the Session Manager plugin for the AWS CLI (command line interface).

# Keyless, IAM-authorized, fully logged shell — no SSH key, no open port 22.
aws ssm start-session --target i-0abc12
# Starting session with SessionId: alice-0a1b2c3d4e5f6g7h8
# sh-5.2$ whoami
# ssm-user
# sh-5.2$ exit
 
# Reach a private RDS instance without exposing it — tunnel through the node.
aws ssm start-session --target i-0abc12 \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["db.abc.us-east-1.rds.amazonaws.com"],"portNumber":["5432"],"localPortNumber":["15432"]}'
# Starting session with SessionId: alice-0f9e8d7c6b5a4
# Port 15432 opened for sessionId alice-0f9e8d7c6b5a4.
# Waiting for connections...   # now: psql -h 127.0.0.1 -p 15432 ...

Parameter Store: a config tree with encryption built in

Parameter Store is a filing cabinet for configuration, with drawers inside drawers. Names read like file paths: /prod/db/password. Values come in three types. String and StringList are plain text. SecureString is encrypted with KMS and decrypts only when a role that is allowed to read it asks with --with-decryption. Parameters are versioned, and your applications or pipelines fetch them at run time using their own IAM identity, so nothing sensitive gets baked into code or burned into an AMI. There are two tiers. Standard is free, caps a value at 4 KB, and allows 10,000 parameters per account per Region. Advanced costs $0.05 per parameter per month, lifts those ceilings to 8 KB and 100,000 parameters, and turns on parameter policies such as automatic expiry. The line between this and Secrets Manager is rotation: Parameter Store has none built in, while Secrets Manager rotates credentials for you at $0.40 per secret per month. Default read throughput is 40 transactions per second, a number that bites much harder than it looks, as the warning below explains.

# Store a secret encrypted with KMS (SecureString). Nothing lands in code or the AMI.
aws ssm put-parameter --name /prod/db/password --type SecureString \
  --value 'S0meR3alSecret' --key-id alias/aws/ssm
# {
#     "Version": 1,
#     "Tier": "Standard"
# }
 
# Pull an app's whole config subtree in one call, decrypting SecureStrings via the role.
aws ssm get-parameters-by-path --path /prod/ --recursive --with-decryption \
  --query "Parameters[].{Name:Name,Value:Value}" --output table
# ---------------------------------------------------
# |               GetParametersByPath               |
# +----------------------+--------------------------+
# |         Name         |          Value           |
# +----------------------+--------------------------+
# |  /prod/db/host       |  db.internal.example.com |
# |  /prod/db/password   |  S0meR3alSecret          |
# |  /prod/feature/newui |  true                    |
# +----------------------+--------------------------+

Do it now, and keep it that way

Two features split the work between a one-off order and a standing rule. Run Command is the one-off. You pick a document (AWS-RunShellScript, AWS-RunPowerShellScript, or one you wrote yourself) and a target set by tag, resource group, or instance ID, and it runs once everywhere that matches. Two safety valves stop that from becoming an outage. Rate control is the term: --max-concurrency 10% rolls the change through the fleet in slices, so you never restart nginx on 500 nodes in the same second, and --max-errors 5% halts the rollout the moment failures cross the line. A blast-radius brake, with nothing extra to install. State Manager is the standing rule. You bind a document to a target on a cron or rate schedule, and the configuration converges, then converges again on the next tick, actively fighting drift rather than only reporting it. Every invocation leaves a record you can push to S3 or CloudWatch when an auditor asks.

# Roll a restart across every web node, 10% at a time; stop if >5% fail.
CMD=$(aws ssm send-command \
  --document-name AWS-RunShellScript \
  --targets Key=tag:Role,Values=web \
  --parameters 'commands=["systemctl restart nginx"]' \
  --max-concurrency 10% --max-errors 5% \
  --query Command.CommandId --output text)
echo "$CMD"
# 8f7e6d5c-4b3a-2109-8765-43210fedcba9
 
aws ssm list-command-invocations --command-id "$CMD" \
  --query "CommandInvocations[].{Node:InstanceId,Status:Status}" --output table
# -------------------------------
# |   ListCommandInvocations    |
# +-----------+-----------------+
# |   Node    |     Status      |
# +-----------+-----------------+
# |  i-0abc12 |  Success        |
# |  i-0def34 |  Success        |
# +-----------+-----------------+

Patch Manager across hundreds of nodes

Patching hundreds of machines should be a policy with a calendar attached, not somebody's Saturday. Patch Manager works from a patch baseline, which is the rulebook for what counts as approved. AWS ships a sensible default per operating system, or you write your own with auto-approval rules: approve Security and Critical updates seven days after release, say, so a bad batch has time to surface before it reaches you. The Patch Group tag maps each node to its baseline. A maintenance window decides when the scan and install happen, so the work lands in a slot you chose instead of the middle of peak traffic. Afterwards, compliance is a number you can query rather than a feeling: describe-instance-patch-states returns installed, missing, and failed counts per node, ready to roll up into AWS Config or Security Hub. Patch Manager adds no charge of its own beyond the compute time each run burns.

# Fleet patch compliance in one call — the number that goes on the dashboard.
aws ssm describe-instance-patch-states --instance-ids i-0abc12 \
  --query "InstancePatchStates[].{Node:InstanceId,Installed:InstalledCount,Missing:MissingCount,Failed:FailedCount,Baseline:BaselineId}"
# [
#     {
#         "Node": "i-0abc12",
#         "Installed": 214,
#         "Missing": 3,
#         "Failed": 0,
#         "Baseline": "pb-0123456789abcdef0"
#     }
# ]
# Missing > 0 against a Security/Critical baseline is a compliance finding.

Session Manager takes the bastion and the open SSH port off your network diagram. IAM says who may start a session; CloudTrail (the AWS log of every API call) and session logging say what they did once they were inside. If your jump box still allows 0.0.0.0/0 on port 22, you are guarding a door nobody needs to walk through any more.

Parameter Store and Secrets Manager both hold configuration, and only Secrets Manager rotates it for you. Standard parameters are kind to a free-tier budget; Advanced ones cost money and hold bigger values. Give your parameters a path hierarchy and tags on day one, or by month three you will be squinting at a flat list of four thousand names.

Baselines and maintenance windows turn "we really should patch" into a schedule with a compliance report attached. Skip enough windows and you are back to one tired person doing it by hand at midnight.

Try this

List the nodes that have checked in, write and read back a parameter, then fire one harmless command at a node. Use send-command with a read-only uname rather than an interactive session; it is easier to practise on and there is nothing to break.

aws ssm describe-instance-information \
  --query 'InstanceInformationList[].{Id:InstanceId,Ping:PingStatus,Agent:AgentVersion}' --output table
aws ssm put-parameter --name /lab/app/log_level --type String --value INFO --overwrite
aws ssm get-parameter --name /lab/app/log_level --query 'Parameter.{Name:Name,Value:Value}' --output table
aws ssm send-command --document-name AWS-RunShellScript --instance-ids i-0abc123 \
  --parameters 'commands=["uname -a"]' --query 'Command.CommandId' --output text

i-0abc123 | Online | 3.3.551.0
-----------------------------
|        GetParameter       |
+----------------+----------+
|      Name      |  Value   |
+----------------+----------+
| /lab/app/log_level | INFO |
+----------------+----------+
command-id: 12345678-abcd-...

Takeaway

Carry one thing out of here: the direction of the connection. The agent calls out, so inbound SSH never has to exist at all, and Session Manager, Parameter Store, and Patch Manager become the surface you work through every day.

Next: turn off inbound SSH on a lab instance, check that Systems Manager still reaches it, then switch on session logging to S3 or CloudWatch and watch your own keystrokes land there.

Run Command fires one document once. State Manager keeps one document applied. Neither can express a *procedure*: cordon a node, drain its connections, patch it, confirm it came back healthy, and only then put it back in rotation, with an approval gate part way through and a rollback if a step fails. Ordered steps, branches, their own inputs, and error handling are the job of the SSM Automation runbook, where all of these primitives get composed into a playbook you can run at three in the morning. That is the next lesson.

Related