AWS Systems Manager: Beyond Parameter Store
Most engineers know AWS Systems Manager as the place where Parameter Store lives. That is an accurate but deeply incomplete picture. SSM is a fleet management platform with eight distinct capabilities, several of which eliminate entire categories of operational burden that teams carry by default. The bastion host. The SSH key rotation process. The ad-hoc patching script that runs as a one-off cron job. The configuration drift that accumulates on instances between deployments.
This post covers each major SSM capability in depth: what it does, how it works, and what the practical operational impact is. Parameter Store gets a section at the end — it is useful, but it is not the interesting part.
Session Manager: The Bastion Killer
The traditional path to a private EC2 instance runs through a bastion host: a small instance in a public subnet with port 22 open, SSH keys distributed to the team, access logs to a shell history file, and the operational burden of keeping that instance patched, monitored, and available. The bastion is also a persistent attack surface. A misconfigured security group, a leaked key, or an unpatched vulnerability in the bastion is the blast radius for every private instance behind it.
Session Manager eliminates this entirely. SSM Agent — installed by default on Amazon Linux 2, Amazon Linux 2023, and recent Ubuntu and Windows Server AMIs — establishes an outbound HTTPS connection to the Systems Manager endpoint over TLS 1.2 using SigV4 signing. No inbound ports. No SSH keys. No bastion. Access is controlled entirely by IAM.
Developer → (HTTPS outbound) → SSM Service → (WebSocket) → SSM Agent on instance
The connection is initiated by the instance, not the developer. From a network perspective, the instance needs outbound HTTPS access to three endpoints — nothing needs to reach in.
What the instance needs
One IAM instance profile with the AmazonSSMManagedInstanceCore managed policy attached. This policy grants the minimum permissions for SSM Agent to register with the service and receive commands:
|
|
That is it. No SSH daemon configuration, no key pair, no security group inbound rules.
Starting a session
|
|
Port forwarding is the feature that eliminates the most bastion-replacement friction. Any TCP service on the instance or reachable from it — RDS, Elasticache, internal APIs — can be tunneled through Session Manager without opening additional ports.
For teams that cannot give up native SSH semantics (existing tooling, SCP file transfers, rsync), SSH over Session Manager is available by adding a ProxyCommand to ~/.ssh/config:
Host i-* mi-*
ProxyCommand sh -c "aws ssm start-session \
--target %h \
--document-name AWS-StartSSHSession \
--parameters 'portNumber=%p'"
After this, ssh ec2-user@i-0123456789abcdef0 works normally, with all traffic tunneled through Session Manager. The SSH key is still used for the SSH layer, but the network path is Session Manager — no public IP or open port 22 required.
Session logging
Every session can be logged to CloudWatch Logs or S3. Configure this in the Session Manager preferences in the console, or via the SSM-SessionManagerRunShell document. With CloudWatch logging enabled, the full session input and output — every command typed, every line of output — is stored in a log group. KMS encryption of session content is available and recommended for sensitive environments.
CloudTrail independently records the session lifecycle events: StartSession, TerminateSession, and ResumeSession, with the IAM principal, instance ID, and timestamp. This gives you two audit layers: who started the session (CloudTrail) and what they did in it (CloudWatch).
VPC endpoints for private subnets
Instances with no internet access — no IGW, no NAT gateway — need Interface VPC endpoints to reach the SSM service. Three endpoints are required:
| Endpoint | Purpose |
|---|---|
com.amazonaws.<region>.ssm |
SSM API calls |
com.amazonaws.<region>.ssmmessages |
Session Manager data channel |
com.amazonaws.<region>.ec2messages |
EC2 message delivery |
Each endpoint needs a security group that allows inbound HTTPS (port 443) from the instance security group. The instances need outbound HTTPS to the endpoint security group. No other ports.
|
|
With --private-dns-enabled, the standard ssm.us-east-1.amazonaws.com hostname resolves to the private endpoint IP — no code changes needed. Without it, you need to override the endpoint URL in SDK configuration.
Session Manager vs EC2 Instance Connect
| Session Manager | EC2 Instance Connect | |
|---|---|---|
| Requires public IP | No | Yes (for CLI) |
| Inbound port 22 open | No | Yes |
| SSH key management | None (IAM only) | Ephemeral key, pushed at connect time |
| Windows support | Yes | No |
| On-premises instances | Yes (hybrid activation) | No |
| Session content logging | CloudWatch Logs / S3 | CloudTrail only |
| Port forwarding | Yes | No |
EC2 Instance Connect solves the key distribution problem (it pushes a 60-second ephemeral key to the instance at connect time) but still requires port 22 open and a public IP. Session Manager eliminates both requirements and extends to Windows and on-premises.
Run Command: Operations at Fleet Scale
Run Command executes shell scripts, PowerShell commands, or SSM document steps across a set of managed instances. It replaces the pattern of SSH-ing into each box individually, or of maintaining a separate Ansible control node for ad-hoc fleet operations.
SSM Agent polls for commands continuously. When a command arrives, the agent executes it and returns status and output. Commands go through the following states: Pending (queued, not yet delivered) → InProgress (running) → terminal state (Success, Failed, TimedOut, Cancelled, DeliveryTimedOut).
Targeting instances
Commands can target instances by ID, by tag, or by resource group:
|
|
Multiple tag conditions in a single --targets list are ANDed together. To target instances matching any of several tags, make separate send-command calls.
Rate controls
The two parameters that make fleet operations safe:
--max-concurrency: How many instances to run on simultaneously. Accepts an absolute number (10) or a percentage of the target set (10%). Use percentages for auto-scaling groups where the total count varies. The default is 50, which is too high for most production fleet operations — start at 10% and increase based on your confidence in the command.
--max-errors: Stop sending to new instances after this many failures. Accepts absolute number or percentage. The default is 0, meaning the first failure stops the entire operation. For patching or restarts, 5% is a reasonable threshold — enough to absorb a handful of expected failures without letting a widespread problem run to completion.
|
|
10% concurrency means at most 10% of the fleet is patching simultaneously. 2% max errors means the operation halts if more than 2% of instances report failure — catching a problem that affects a small number of instances before it affects all of them.
Output storage
Command output is truncated to 2500 bytes in the console and API response. For commands that produce significant output — package installation, log analysis, audit scripts — direct output to S3:
|
|
Output is retained for 30 days before automatic deletion.
Common managed documents
| Document | Purpose |
|---|---|
AWS-RunShellScript |
Linux shell commands (inline or script) |
AWS-RunPowerShellScript |
PowerShell on Windows or Linux |
AWS-ConfigureAWSPackage |
Install/update software packages (SSM Distributor) |
AWS-UpdateSSMAgent |
Update the SSM Agent binary itself |
AWS-RunPatchBaseline |
Scan or install patches per the baseline |
AWS-GatherSoftwareInventory |
Collect software inventory |
Patch Manager: Compliance-Driven Fleet Patching
Ad-hoc patching — SSHing into boxes and running yum update — does not scale, does not provide compliance tracking, and provides no guardrails against patching too many instances simultaneously. Patch Manager is the SSM subsystem that replaces it.
Patch baselines
A patch baseline defines which patches are approved for installation. Every OS type has a default baseline (AWS-AmazonLinux2DefaultPatchBaseline, AWS-WindowsServerDefaultPatchBaseline, etc.), but production environments should use custom baselines with explicit auto-approval rules.
A custom baseline specifies:
- Approval rules: Auto-approve patches of a given severity after N days. For example, approve Critical patches after 0 days (immediately) and Important patches after 7 days.
- Approved patches: An explicit whitelist, useful for specific CVE patches you want to fast-track.
- Rejected patches: Patches known to cause problems in your environment.
|
|
Patch groups
Patch groups associate instances with baselines using the Patch Group EC2 tag (case-sensitive). An instance tagged Patch Group=production is automatically associated with whichever baseline you register for the production group:
|
|
This lets you maintain different patch policies for different tiers — a tighter auto-approval window for non-production, a human-reviewed baseline for databases, a separate baseline for Windows instances — without any per-instance configuration.
The recommended workflow
Running Operation=Install directly on production instances without assessment first is risky. The recommended pattern is scan-then-install:
Step 1: Scan — Run AWS-RunPatchBaseline with Operation=Scan across the fleet. This identifies missing patches without installing anything. No reboots, no downtime.
|
|
Step 2: Assess — Review the Systems Manager Compliance dashboard. Each instance shows patch compliance status: Compliant, Non-Compliant, or InsufficientData. Non-Compliant instances list the missing patches with their severity and classification.
Step 3: Schedule installation via a Maintenance Window during a low-traffic period. The window defines a time slot, a duration, and a cutoff time (stop starting new tasks N hours before the window closes).
|
|
RebootOption=RebootIfNeeded reboots only if the installed patches require it. NoReboot installs patches and marks them InstalledPendingReboot — the instance will show as non-compliant until the next reboot. For stateful services, NoReboot followed by a scheduled reboot during the next window gives more control.
Automation: Runbooks for Everything Else
SSM Automation documents are YAML or JSON runbooks that chain AWS API calls, shell commands, waits, and approval gates into repeatable operational procedures. They replace the operational scripts that live in someone’s ~/scripts/ directory and are never documented or version-controlled.
Document structure
Schema version 0.3 is current. Each step specifies an action type and its inputs:
|
|
Step outputs are referenced by subsequent steps using {{ StepName.OutputName }} syntax. The aws:waitForAwsResourceProperty action polls an API until a value matches a desired state — the correct way to wait for asynchronous operations like AMI creation or instance state transitions.
Action types
| Action | Purpose |
|---|---|
aws:executeAwsApi |
Call any AWS API |
aws:runCommand |
Run SSM Run Command on instances |
aws:waitForAwsResourceProperty |
Poll API until a value matches |
aws:branch |
Conditional step routing |
aws:approve |
Pause for human sign-off |
aws:sleep |
Wait N seconds |
aws:executeScript |
Run Python or PowerShell inline |
aws:invokeAwsService |
Simplified AWS service calls |
Human-in-the-loop approvals
The aws:approve action pauses execution and sends an SNS notification. A human approves or rejects the execution through the SSM console or CLI. Useful for runbooks that perform irreversible actions — production deployments, AMI deregistrations, database schema changes:
|
|
EventBridge integration
Any EventBridge event can trigger an Automation document. Common patterns:
Auto-remediate Security Hub findings: When Security Hub generates a HIGH severity finding for an EC2 instance, EventBridge triggers a runbook that isolates the instance (modifies its security group to block all inbound traffic) and creates a forensic snapshot.
EC2 state change reactions: When an instance terminates unexpectedly, trigger a runbook that captures CloudWatch logs, posts a notification, and optionally launches a replacement.
Scheduled AMI rotation: EventBridge Scheduler triggers a runbook weekly that creates a new AMI from a reference instance, runs validation checks, and deregisters AMIs older than 30 days.
|
|
State Manager: Preventing Configuration Drift
Run Command is imperative and one-shot: send a command, it runs once, done. State Manager is declarative and continuous: define the desired state, associate it with instances, and SSM enforces it on a schedule. If an instance drifts from the desired state between enforcement runs, it shows as non-compliant in the Compliance dashboard, and the next scheduled run corrects it.
Associations
An association links a document to a set of instances with a schedule and parameters:
|
|
The SSH hardening association runs every 6 hours. If an operator manually enables password authentication on an instance (accident, debugging, forgotten cleanup), State Manager reverts it within 6 hours. The association does not prevent the change — it corrects it.
Association vs Run Command
| State Manager Association | Run Command | |
|---|---|---|
| Execution model | Scheduled, recurring | One-time |
| Idempotency requirement | Required (runs repeatedly) | Not required |
| Drift correction | Automatic | Manual |
| History | Association execution history | Command history |
| Use case | “Keep this configuration true” | “Do this thing once” |
State Manager replaces the noisier parts of a configuration management system for AWS instances. For enforcing a small number of critical configuration invariants — monitoring agent running, security baseline applied, specific files in place — it is sufficient. For complex multi-step application configuration, Ansible or a proper CM system is still appropriate.
Inventory: Fleet-Wide Asset Discovery
SSM Inventory collects configuration data from managed instances and stores it in Systems Manager. It is a passive collection mechanism: you define what to collect via a State Manager association, and the data accumulates centrally.
Enabling inventory collection:
|
|
What it collects by default: installed applications (with version and publisher), network configuration, Windows updates (on Windows), running services, instance metadata, and the SSM Agent version.
Custom inventory
Custom inventory lets you collect application-specific metadata that AWS does not know about. Place a JSON file on the instance at /var/lib/amazon/ssm/inventory/custom/:
|
|
SSM Agent picks this up on the next collection cycle and makes it queryable through the Inventory console and CLI alongside all standard inventory data.
Querying at scale with Athena
For fleets larger than a few hundred instances, the console query interface becomes limiting. Resource Data Sync exports all inventory data to S3 in a structured format that Athena can query:
|
|
Once synced, AWS Glue can crawl the S3 prefix to build a table schema, and Athena can query across the entire fleet:
|
|
This pattern gives you cross-account, cross-region fleet visibility without a dedicated asset management tool.
Parameter Store
Parameter Store stores configuration values and secrets in a hierarchical namespace. The path structure mirrors how configuration is actually organized:
/app/production/database/host
/app/production/database/port
/app/production/database/password (SecureString, KMS encrypted)
/app/staging/database/host
/app/staging/database/password
GetParametersByPath retrieves all parameters under a prefix, which lets an application load its entire environment configuration in a single API call:
|
|
Standard vs Advanced tier
| Standard | Advanced | |
|---|---|---|
| Max value size | 4 KB | 8 KB |
| Throughput | 40 req/sec (burst 100) | 100 req/sec (burst 1000) |
| Parameter policies | No | Yes (expiration, notification) |
| Cost | Free (up to 10,000 params) | $0.05/parameter/month |
Advanced tier adds parameter policies — TTL-based expiration that notifies via EventBridge when a parameter approaches expiry. Useful for enforcing secret rotation deadlines.
Parameter Store vs Secrets Manager
The two services overlap and the choice between them is straightforward: Secrets Manager is for secrets that need automatic rotation; Parameter Store is for everything else.
| Parameter Store | Secrets Manager | |
|---|---|---|
| Automatic rotation | No (manual via Lambda) | Yes (built-in for RDS, Redshift, etc.) |
| Cross-region replication | No | Yes |
| Max secret size | 8 KB (Advanced) | 64 KB |
| Cost | Free (Standard) | $0.40/secret/month |
| RDS credential rotation | Manual Lambda required | Native, built-in |
Database passwords for RDS belong in Secrets Manager. API keys, feature flags, environment-specific config values, and anything that does not rotate automatically belongs in Parameter Store.
The No-Bastion-Host Architecture
Putting the above together, the full architecture for a production VPC that requires zero inbound access to instances:
┌─────────────────────────────────────────────────────────────────┐
│ VPC (private subnets only) │
│ │
│ ┌─────────────────────────┐ ┌──────────────────────────┐ │
│ │ Interface VPC Endpoints│ │ EC2 Instances │ │
│ │ - ssm │ │ - No public IP │ │
│ │ - ssmmessages │◄───│ - No inbound SG rules │ │
│ │ - ec2messages │ │ - SSM Agent (outbound) │ │
│ │ (sg: allow 443 inbound │ │ - Instance profile: │ │
│ │ from instance SG) │ │ AmazonSSMManagedCore │ │
│ └─────────────────────────┘ └──────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
│ HTTPS outbound
▼
SSM Service (Region)
│
│ Session / Command / Automation
▼
Developer / CI/CD / EventBridge
(IAM-authenticated, CloudTrail-logged)
Security group rules for instances in this architecture:
Inbound: (none)
Outbound: TCP 443 → endpoint security group (VPC endpoints)
TCP 443 → 0.0.0.0/0 (optional, for S3/other AWS services via gateway endpoint)
Security group rules for the VPC endpoint ENIs:
Inbound: TCP 443 → instance security group
Outbound: (none required — AWS manages endpoint responses)
Network ACLs must allow outbound TCP 443 from instance subnets and inbound ephemeral ports (1024–65535) back. If your NACLs are default allow, nothing needs to change.
The audit trail for this architecture:
- CloudTrail: Every
StartSession,SendCommand,StartAutomationExecutioncall — who, when, which instance, which document. - CloudWatch Logs: Full session content (commands typed and output) for every Session Manager session.
- S3: Run Command output, Automation execution logs, Patch Manager results.
- Systems Manager Compliance: Patch state, State Manager association compliance, drift history.
This is strictly more auditable than a bastion host architecture. A bastion captures network access but not what the user did after connecting. Session Manager captures both.
IAM Reference
Minimum instance profile permissions — attach AmazonSSMManagedInstanceCore to the EC2 role. No additional permissions needed for Session Manager and Run Command.
Operator policy — restrict to specific documents and tagged instances:
|
|
Tag-based access control — restrict sessions to instances tagged with the operator’s username:
|
|
With this policy, each operator can only open sessions on instances explicitly tagged with their IAM username. No group-level access, no shared account.
Fleet Manager
Fleet Manager is the console UI built on top of the SSM APIs. It provides a browser-based file system browser, process viewer, user and group management, and — on Windows — a registry editor. It is useful for ad-hoc investigation without installing additional tooling or opening additional network access.
The file system browser supports viewing, creating, editing, and deleting files up to a configurable size limit. The tail feature streams new lines from a log file in near-real-time — a reasonable substitute for tail -f when you need to check an application log without a full session.
For Windows environments, the registry editor eliminates the most common reason for RDP access: making a configuration change that requires a registry key edit. Combined with Session Manager’s port forwarding (for RDP tunneling when the GUI is genuinely necessary), Fleet Manager covers most Windows management needs from the AWS console without any inbound network access.
Comments