LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

AWS Systems Manager: Beyond Parameter Store

awssystems-managerdevopssecuritycloudinfrastructure-as-codeplatform-engineeringoperations

Most engineers know AWS Systems Manager as the place where Parameter Store lives. That is an accurate but deeply incomplete picture. SSM is a fleet management platform with eight distinct capabilities, several of which eliminate entire categories of operational burden that teams carry by default. The bastion host. The SSH key rotation process. The ad-hoc patching script that runs as a one-off cron job. The configuration drift that accumulates on instances between deployments.

This post covers each major SSM capability in depth: what it does, how it works, and what the practical operational impact is. Parameter Store gets a section at the end — it is useful, but it is not the interesting part.


Session Manager: The Bastion Killer

The traditional path to a private EC2 instance runs through a bastion host: a small instance in a public subnet with port 22 open, SSH keys distributed to the team, access logs to a shell history file, and the operational burden of keeping that instance patched, monitored, and available. The bastion is also a persistent attack surface. A misconfigured security group, a leaked key, or an unpatched vulnerability in the bastion is the blast radius for every private instance behind it.

Session Manager eliminates this entirely. SSM Agent — installed by default on Amazon Linux 2, Amazon Linux 2023, and recent Ubuntu and Windows Server AMIs — establishes an outbound HTTPS connection to the Systems Manager endpoint over TLS 1.2 using SigV4 signing. No inbound ports. No SSH keys. No bastion. Access is controlled entirely by IAM.

Developer → (HTTPS outbound) → SSM Service → (WebSocket) → SSM Agent on instance

The connection is initiated by the instance, not the developer. From a network perspective, the instance needs outbound HTTPS access to three endpoints — nothing needs to reach in.

What the instance needs

One IAM instance profile with the AmazonSSMManagedInstanceCore managed policy attached. This policy grants the minimum permissions for SSM Agent to register with the service and receive commands:

1
2
3
4
5
6
7
8
9
{
  "Action": [
    "ssm:UpdateInstanceInformation",
    "ssmmessages:AcknowledgeMessage",
    "ssmmessages:GetEndpoint",
    "ssmmessages:GetMessages",
    "ec2messages:GetMessages"
  ]
}

That is it. No SSH daemon configuration, no key pair, no security group inbound rules.

Starting a session

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Interactive shell
aws ssm start-session --target i-0123456789abcdef0

# Port forwarding — tunnel localhost:15432 to the instance's port 5432
aws ssm start-session \
  --target i-0123456789abcdef0 \
  --document-name AWS-StartPortForwardingSession \
  --parameters 'portNumber=5432,localPortNumber=15432'

# After that, psql works normally against the private RDS endpoint via the tunnel
psql -h localhost -p 15432 -U admin -d mydb

Port forwarding is the feature that eliminates the most bastion-replacement friction. Any TCP service on the instance or reachable from it — RDS, Elasticache, internal APIs — can be tunneled through Session Manager without opening additional ports.

For teams that cannot give up native SSH semantics (existing tooling, SCP file transfers, rsync), SSH over Session Manager is available by adding a ProxyCommand to ~/.ssh/config:

Host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session \
      --target %h \
      --document-name AWS-StartSSHSession \
      --parameters 'portNumber=%p'"

After this, ssh ec2-user@i-0123456789abcdef0 works normally, with all traffic tunneled through Session Manager. The SSH key is still used for the SSH layer, but the network path is Session Manager — no public IP or open port 22 required.

Session logging

Every session can be logged to CloudWatch Logs or S3. Configure this in the Session Manager preferences in the console, or via the SSM-SessionManagerRunShell document. With CloudWatch logging enabled, the full session input and output — every command typed, every line of output — is stored in a log group. KMS encryption of session content is available and recommended for sensitive environments.

CloudTrail independently records the session lifecycle events: StartSession, TerminateSession, and ResumeSession, with the IAM principal, instance ID, and timestamp. This gives you two audit layers: who started the session (CloudTrail) and what they did in it (CloudWatch).

VPC endpoints for private subnets

Instances with no internet access — no IGW, no NAT gateway — need Interface VPC endpoints to reach the SSM service. Three endpoints are required:

Endpoint Purpose
com.amazonaws.<region>.ssm SSM API calls
com.amazonaws.<region>.ssmmessages Session Manager data channel
com.amazonaws.<region>.ec2messages EC2 message delivery

Each endpoint needs a security group that allows inbound HTTPS (port 443) from the instance security group. The instances need outbound HTTPS to the endpoint security group. No other ports.

1
2
3
4
5
6
7
8
# Create the three endpoints (repeat for ssmmessages and ec2messages)
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-12345678 \
  --service-name com.amazonaws.us-east-1.ssm \
  --vpc-endpoint-type Interface \
  --subnet-ids subnet-aaa subnet-bbb \
  --security-group-ids sg-endpoints \
  --private-dns-enabled

With --private-dns-enabled, the standard ssm.us-east-1.amazonaws.com hostname resolves to the private endpoint IP — no code changes needed. Without it, you need to override the endpoint URL in SDK configuration.

Session Manager vs EC2 Instance Connect

Session Manager EC2 Instance Connect
Requires public IP No Yes (for CLI)
Inbound port 22 open No Yes
SSH key management None (IAM only) Ephemeral key, pushed at connect time
Windows support Yes No
On-premises instances Yes (hybrid activation) No
Session content logging CloudWatch Logs / S3 CloudTrail only
Port forwarding Yes No

EC2 Instance Connect solves the key distribution problem (it pushes a 60-second ephemeral key to the instance at connect time) but still requires port 22 open and a public IP. Session Manager eliminates both requirements and extends to Windows and on-premises.


Run Command: Operations at Fleet Scale

Run Command executes shell scripts, PowerShell commands, or SSM document steps across a set of managed instances. It replaces the pattern of SSH-ing into each box individually, or of maintaining a separate Ansible control node for ad-hoc fleet operations.

SSM Agent polls for commands continuously. When a command arrives, the agent executes it and returns status and output. Commands go through the following states: Pending (queued, not yet delivered) → InProgress (running) → terminal state (Success, Failed, TimedOut, Cancelled, DeliveryTimedOut).

Targeting instances

Commands can target instances by ID, by tag, or by resource group:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# By tag — all production web servers
aws ssm send-command \
  --document-name AWS-RunShellScript \
  --targets 'Key=tag:Role,Values=webserver' 'Key=tag:Environment,Values=production' \
  --parameters 'commands=["systemctl status nginx"]'

# By specific IDs
aws ssm send-command \
  --document-name AWS-RunShellScript \
  --targets 'Key=InstanceIds,Values=i-abc,i-def,i-ghi' \
  --parameters 'commands=["df -h"]'

Multiple tag conditions in a single --targets list are ANDed together. To target instances matching any of several tags, make separate send-command calls.

Rate controls

The two parameters that make fleet operations safe:

--max-concurrency: How many instances to run on simultaneously. Accepts an absolute number (10) or a percentage of the target set (10%). Use percentages for auto-scaling groups where the total count varies. The default is 50, which is too high for most production fleet operations — start at 10% and increase based on your confidence in the command.

--max-errors: Stop sending to new instances after this many failures. Accepts absolute number or percentage. The default is 0, meaning the first failure stops the entire operation. For patching or restarts, 5% is a reasonable threshold — enough to absorb a handful of expected failures without letting a widespread problem run to completion.

1
2
3
4
5
6
7
8
9
# Rolling security patch across the fleet
aws ssm send-command \
  --document-name AWS-RunPatchBaseline \
  --targets 'Key=tag:Environment,Values=production' \
  --parameters 'Operation=Install,RebootOption=RebootIfNeeded' \
  --max-concurrency 10% \
  --max-errors 2% \
  --output-s3-bucket-name ops-logs \
  --output-s3-key-prefix patch-runs/

10% concurrency means at most 10% of the fleet is patching simultaneously. 2% max errors means the operation halts if more than 2% of instances report failure — catching a problem that affects a small number of instances before it affects all of them.

Output storage

Command output is truncated to 2500 bytes in the console and API response. For commands that produce significant output — package installation, log analysis, audit scripts — direct output to S3:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
aws ssm send-command \
  --document-name AWS-RunShellScript \
  --targets 'Key=tag:Environment,Values=production' \
  --parameters 'commands=["ss -tlnp"]' \
  --output-s3-bucket-name ops-logs \
  --output-s3-key-prefix network-audit/

# Retrieve output for a specific invocation
aws ssm get-command-invocation \
  --command-id abc123 \
  --instance-id i-0123456789abcdef0 \
  --query 'StandardOutputContent'

Output is retained for 30 days before automatic deletion.

Common managed documents

Document Purpose
AWS-RunShellScript Linux shell commands (inline or script)
AWS-RunPowerShellScript PowerShell on Windows or Linux
AWS-ConfigureAWSPackage Install/update software packages (SSM Distributor)
AWS-UpdateSSMAgent Update the SSM Agent binary itself
AWS-RunPatchBaseline Scan or install patches per the baseline
AWS-GatherSoftwareInventory Collect software inventory

Patch Manager: Compliance-Driven Fleet Patching

Ad-hoc patching — SSHing into boxes and running yum update — does not scale, does not provide compliance tracking, and provides no guardrails against patching too many instances simultaneously. Patch Manager is the SSM subsystem that replaces it.

Patch baselines

A patch baseline defines which patches are approved for installation. Every OS type has a default baseline (AWS-AmazonLinux2DefaultPatchBaseline, AWS-WindowsServerDefaultPatchBaseline, etc.), but production environments should use custom baselines with explicit auto-approval rules.

A custom baseline specifies:

  • Approval rules: Auto-approve patches of a given severity after N days. For example, approve Critical patches after 0 days (immediately) and Important patches after 7 days.
  • Approved patches: An explicit whitelist, useful for specific CVE patches you want to fast-track.
  • Rejected patches: Patches known to cause problems in your environment.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
aws ssm create-patch-baseline \
  --name "production-linux-baseline" \
  --operating-system "AMAZON_LINUX_2" \
  --approval-rules '[{
    "PatchFilterGroup": {
      "PatchFilters": [
        {"Key":"CLASSIFICATION","Values":["Security","Bugfix"]},
        {"Key":"SEVERITY","Values":["Critical","Important"]}
      ]
    },
    "AutoApproveAfterDays": 7,
    "ComplianceLevel": "CRITICAL"
  }]' \
  --rejected-patches "kernel-*" \
  --description "Production servers — auto-approve security patches after 7 days"

Patch groups

Patch groups associate instances with baselines using the Patch Group EC2 tag (case-sensitive). An instance tagged Patch Group=production is automatically associated with whichever baseline you register for the production group:

1
2
3
aws ssm register-patch-baseline-for-patch-group \
  --baseline-id pb-0123456789abcdef0 \
  --patch-group production

This lets you maintain different patch policies for different tiers — a tighter auto-approval window for non-production, a human-reviewed baseline for databases, a separate baseline for Windows instances — without any per-instance configuration.

Running Operation=Install directly on production instances without assessment first is risky. The recommended pattern is scan-then-install:

Step 1: Scan — Run AWS-RunPatchBaseline with Operation=Scan across the fleet. This identifies missing patches without installing anything. No reboots, no downtime.

1
2
3
4
aws ssm send-command \
  --document-name AWS-RunPatchBaseline \
  --targets 'Key=tag:Patch Group,Values=production' \
  --parameters 'Operation=Scan'

Step 2: Assess — Review the Systems Manager Compliance dashboard. Each instance shows patch compliance status: Compliant, Non-Compliant, or InsufficientData. Non-Compliant instances list the missing patches with their severity and classification.

Step 3: Schedule installation via a Maintenance Window during a low-traffic period. The window defines a time slot, a duration, and a cutoff time (stop starting new tasks N hours before the window closes).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# Create a Sunday 2 AM maintenance window, 4 hours, cutoff 1 hour before end
aws ssm create-maintenance-window \
  --name "Sunday-Production-Patching" \
  --schedule "cron(0 2 ? * SUN *)" \
  --duration 4 \
  --cutoff 1 \
  --allow-unassociated-targets false

# Register the patch task
aws ssm register-task-with-maintenance-window \
  --window-id mw-0123456789abcdef0 \
  --task-type RUN_COMMAND \
  --task-arn AWS-RunPatchBaseline \
  --service-role-arn arn:aws:iam::123456789012:role/SSM-MaintenanceWindowRole \
  --targets 'Key=tag:Patch Group,Values=production' \
  --task-invocation-parameters '{
    "aws:runCommand": {
      "Parameters": {
        "Operation": ["Install"],
        "RebootOption": ["RebootIfNeeded"]
      }
    }
  }' \
  --max-concurrency 20% \
  --max-errors 5%

RebootOption=RebootIfNeeded reboots only if the installed patches require it. NoReboot installs patches and marks them InstalledPendingReboot — the instance will show as non-compliant until the next reboot. For stateful services, NoReboot followed by a scheduled reboot during the next window gives more control.


Automation: Runbooks for Everything Else

SSM Automation documents are YAML or JSON runbooks that chain AWS API calls, shell commands, waits, and approval gates into repeatable operational procedures. They replace the operational scripts that live in someone’s ~/scripts/ directory and are never documented or version-controlled.

Document structure

Schema version 0.3 is current. Each step specifies an action type and its inputs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
schemaVersion: '0.3'
description: 'Create AMI from instance, wait for completion, share to staging account'
assumeRole: '{{ AutomationAssumeRole }}'
parameters:
  InstanceId:
    type: String
    description: Source instance ID
  StagingAccountId:
    type: String
    description: Account ID to share the AMI with
  AutomationAssumeRole:
    type: String

mainSteps:
  - name: StopInstance
    action: aws:executeAwsApi
    inputs:
      Service: ec2
      Api: StopInstances
      InstanceIds: ['{{ InstanceId }}']

  - name: WaitForStop
    action: aws:waitForAwsResourceProperty
    inputs:
      Service: ec2
      Api: DescribeInstances
      InstanceIds: ['{{ InstanceId }}']
      PropertySelector: '$.Reservations[0].Instances[0].State.Name'
      DesiredValues: ['stopped']

  - name: CreateImage
    action: aws:executeAwsApi
    inputs:
      Service: ec2
      Api: CreateImage
      InstanceId: '{{ InstanceId }}'
      Name: 'golden-ami-{{ global:DATE_TIME }}'
      NoReboot: false
    outputs:
      - Name: AmiId
        Selector: $.ImageId
        Type: String

  - name: WaitForAmi
    action: aws:waitForAwsResourceProperty
    inputs:
      Service: ec2
      Api: DescribeImages
      ImageIds: ['{{ CreateImage.AmiId }}']
      PropertySelector: '$.Images[0].State'
      DesiredValues: ['available']

  - name: ShareAmi
    action: aws:executeAwsApi
    inputs:
      Service: ec2
      Api: ModifyImageAttribute
      ImageId: '{{ CreateImage.AmiId }}'
      LaunchPermission:
        Add: [{ UserId: '{{ StagingAccountId }}' }]

  - name: StartInstance
    action: aws:executeAwsApi
    inputs:
      Service: ec2
      Api: StartInstances
      InstanceIds: ['{{ InstanceId }}']

Step outputs are referenced by subsequent steps using {{ StepName.OutputName }} syntax. The aws:waitForAwsResourceProperty action polls an API until a value matches a desired state — the correct way to wait for asynchronous operations like AMI creation or instance state transitions.

Action types

Action Purpose
aws:executeAwsApi Call any AWS API
aws:runCommand Run SSM Run Command on instances
aws:waitForAwsResourceProperty Poll API until a value matches
aws:branch Conditional step routing
aws:approve Pause for human sign-off
aws:sleep Wait N seconds
aws:executeScript Run Python or PowerShell inline
aws:invokeAwsService Simplified AWS service calls

Human-in-the-loop approvals

The aws:approve action pauses execution and sends an SNS notification. A human approves or rejects the execution through the SSM console or CLI. Useful for runbooks that perform irreversible actions — production deployments, AMI deregistrations, database schema changes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
  - name: ApproveDeployment
    action: aws:approve
    inputs:
      Approvers:
        - arn:aws:iam::123456789012:role/ReleaseManager
        - arn:aws:iam::123456789012:user/oncall-engineer
      MinRequiredApprovals: 1
      Message: 'Approve production deployment of {{ ImageId }}?'
      NotificationArn: arn:aws:sns:us-east-1:123456789012:deployment-approvals
    timeoutSeconds: 3600   # Auto-reject after 1 hour

EventBridge integration

Any EventBridge event can trigger an Automation document. Common patterns:

Auto-remediate Security Hub findings: When Security Hub generates a HIGH severity finding for an EC2 instance, EventBridge triggers a runbook that isolates the instance (modifies its security group to block all inbound traffic) and creates a forensic snapshot.

EC2 state change reactions: When an instance terminates unexpectedly, trigger a runbook that captures CloudWatch logs, posts a notification, and optionally launches a replacement.

Scheduled AMI rotation: EventBridge Scheduler triggers a runbook weekly that creates a new AMI from a reference instance, runs validation checks, and deregisters AMIs older than 30 days.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
{
  "Name": "SecurityHub-HighFinding-Remediation",
  "EventPattern": {
    "source": ["aws.securityhub"],
    "detail-type": ["Security Hub Findings - Imported"],
    "detail": {
      "findings": {
        "Severity": { "Label": ["HIGH", "CRITICAL"] },
        "Resources": { "Type": ["AwsEc2Instance"] }
      }
    }
  },
  "Targets": [{
    "Arn": "arn:aws:ssm:us-east-1:123456789012:automation-definition/IsolateInstance",
    "RoleArn": "arn:aws:iam::123456789012:role/EventBridge-SSM-Role"
  }]
}

State Manager: Preventing Configuration Drift

Run Command is imperative and one-shot: send a command, it runs once, done. State Manager is declarative and continuous: define the desired state, associate it with instances, and SSM enforces it on a schedule. If an instance drifts from the desired state between enforcement runs, it shows as non-compliant in the Compliance dashboard, and the next scheduled run corrects it.

Associations

An association links a document to a set of instances with a schedule and parameters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# Ensure CloudWatch Agent is installed and running on all production instances
aws ssm create-association \
  --name AWS-ConfigureAWSPackage \
  --parameters '{"action":["Install"],"name":["AmazonCloudWatchAgent"],"version":["latest"]}' \
  --targets 'Key=tag:Environment,Values=production' \
  --schedule-expression "rate(1 day)" \
  --compliance-severity MEDIUM

# Enforce SSH hardening configuration
aws ssm create-association \
  --name AWS-RunShellScript \
  --parameters '{
    "commands": [
      "sed -i \"s/^#*PermitRootLogin.*/PermitRootLogin no/\" /etc/ssh/sshd_config",
      "sed -i \"s/^#*PasswordAuthentication.*/PasswordAuthentication no/\" /etc/ssh/sshd_config",
      "systemctl reload sshd"
    ]
  }' \
  --targets 'Key=tag:Compliance,Values=cis-level1' \
  --schedule-expression "rate(6 hours)"

The SSH hardening association runs every 6 hours. If an operator manually enables password authentication on an instance (accident, debugging, forgotten cleanup), State Manager reverts it within 6 hours. The association does not prevent the change — it corrects it.

Association vs Run Command

State Manager Association Run Command
Execution model Scheduled, recurring One-time
Idempotency requirement Required (runs repeatedly) Not required
Drift correction Automatic Manual
History Association execution history Command history
Use case “Keep this configuration true” “Do this thing once”

State Manager replaces the noisier parts of a configuration management system for AWS instances. For enforcing a small number of critical configuration invariants — monitoring agent running, security baseline applied, specific files in place — it is sufficient. For complex multi-step application configuration, Ansible or a proper CM system is still appropriate.


Inventory: Fleet-Wide Asset Discovery

SSM Inventory collects configuration data from managed instances and stores it in Systems Manager. It is a passive collection mechanism: you define what to collect via a State Manager association, and the data accumulates centrally.

Enabling inventory collection:

1
2
3
4
aws ssm create-association \
  --name AWS-GatherSoftwareInventory \
  --targets 'Key=tag:Environment,Values=production' \
  --schedule-expression "rate(1 hour)"

What it collects by default: installed applications (with version and publisher), network configuration, Windows updates (on Windows), running services, instance metadata, and the SSM Agent version.

Custom inventory

Custom inventory lets you collect application-specific metadata that AWS does not know about. Place a JSON file on the instance at /var/lib/amazon/ssm/inventory/custom/:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "SchemaVersion": "1.0",
  "TypeName": "Custom:AppVersion",
  "Content": [
    {
      "AppName": "my-api",
      "Version": "2.4.1",
      "DeployedAt": "2026-05-20T14:30:00Z",
      "CommitSHA": "abc123def456"
    }
  ]
}

SSM Agent picks this up on the next collection cycle and makes it queryable through the Inventory console and CLI alongside all standard inventory data.

Querying at scale with Athena

For fleets larger than a few hundred instances, the console query interface becomes limiting. Resource Data Sync exports all inventory data to S3 in a structured format that Athena can query:

1
2
3
aws ssm create-resource-data-sync \
  --sync-name central-inventory \
  --s3-destination BucketName=inventory-bucket,Prefix=ssm-data,Region=us-east-1

Once synced, AWS Glue can crawl the S3 prefix to build a table schema, and Athena can query across the entire fleet:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
-- Find all instances running nginx older than version 1.24
SELECT instanceid, name, version, architecture
FROM inventory_application
WHERE name = 'nginx'
  AND version < '1.24'
ORDER BY instanceid;

-- Find instances missing a required package
SELECT i.instanceid
FROM inventory_instanceinformation i
LEFT JOIN inventory_application a
  ON i.instanceid = a.instanceid AND a.name = 'amazon-cloudwatch-agent'
WHERE a.name IS NULL;

This pattern gives you cross-account, cross-region fleet visibility without a dedicated asset management tool.


Parameter Store

Parameter Store stores configuration values and secrets in a hierarchical namespace. The path structure mirrors how configuration is actually organized:

/app/production/database/host
/app/production/database/port
/app/production/database/password    (SecureString, KMS encrypted)
/app/staging/database/host
/app/staging/database/password

GetParametersByPath retrieves all parameters under a prefix, which lets an application load its entire environment configuration in a single API call:

1
2
3
4
aws ssm get-parameters-by-path \
  --path /app/production \
  --recursive \
  --with-decryption

Standard vs Advanced tier

Standard Advanced
Max value size 4 KB 8 KB
Throughput 40 req/sec (burst 100) 100 req/sec (burst 1000)
Parameter policies No Yes (expiration, notification)
Cost Free (up to 10,000 params) $0.05/parameter/month

Advanced tier adds parameter policies — TTL-based expiration that notifies via EventBridge when a parameter approaches expiry. Useful for enforcing secret rotation deadlines.

Parameter Store vs Secrets Manager

The two services overlap and the choice between them is straightforward: Secrets Manager is for secrets that need automatic rotation; Parameter Store is for everything else.

Parameter Store Secrets Manager
Automatic rotation No (manual via Lambda) Yes (built-in for RDS, Redshift, etc.)
Cross-region replication No Yes
Max secret size 8 KB (Advanced) 64 KB
Cost Free (Standard) $0.40/secret/month
RDS credential rotation Manual Lambda required Native, built-in

Database passwords for RDS belong in Secrets Manager. API keys, feature flags, environment-specific config values, and anything that does not rotate automatically belongs in Parameter Store.


The No-Bastion-Host Architecture

Putting the above together, the full architecture for a production VPC that requires zero inbound access to instances:

┌─────────────────────────────────────────────────────────────────┐
│  VPC (private subnets only)                                     │
│                                                                 │
│  ┌─────────────────────────┐    ┌──────────────────────────┐   │
│  │  Interface VPC Endpoints│    │  EC2 Instances           │   │
│  │  - ssm                  │    │  - No public IP          │   │
│  │  - ssmmessages          │◄───│  - No inbound SG rules   │   │
│  │  - ec2messages          │    │  - SSM Agent (outbound)  │   │
│  │  (sg: allow 443 inbound │    │  - Instance profile:     │   │
│  │   from instance SG)     │    │    AmazonSSMManagedCore  │   │
│  └─────────────────────────┘    └──────────────────────────┘   │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
                           │ HTTPS outbound
                           ▼
                   SSM Service (Region)
                           │
                           │ Session / Command / Automation
                           ▼
                   Developer / CI/CD / EventBridge
                   (IAM-authenticated, CloudTrail-logged)

Security group rules for instances in this architecture:

Inbound:  (none)
Outbound: TCP 443 → endpoint security group (VPC endpoints)
          TCP 443 → 0.0.0.0/0  (optional, for S3/other AWS services via gateway endpoint)

Security group rules for the VPC endpoint ENIs:

Inbound:  TCP 443 → instance security group
Outbound: (none required — AWS manages endpoint responses)

Network ACLs must allow outbound TCP 443 from instance subnets and inbound ephemeral ports (1024–65535) back. If your NACLs are default allow, nothing needs to change.

The audit trail for this architecture:

  • CloudTrail: Every StartSession, SendCommand, StartAutomationExecution call — who, when, which instance, which document.
  • CloudWatch Logs: Full session content (commands typed and output) for every Session Manager session.
  • S3: Run Command output, Automation execution logs, Patch Manager results.
  • Systems Manager Compliance: Patch state, State Manager association compliance, drift history.

This is strictly more auditable than a bastion host architecture. A bastion captures network access but not what the user did after connecting. Session Manager captures both.


IAM Reference

Minimum instance profile permissions — attach AmazonSSMManagedInstanceCore to the EC2 role. No additional permissions needed for Session Manager and Run Command.

Operator policy — restrict to specific documents and tagged instances:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:StartSession",
        "ssm:SendCommand",
        "ssm:GetCommandInvocation",
        "ssm:ListCommandInvocations",
        "ssm:DescribeInstanceInformation",
        "ssm:TerminateSession"
      ],
      "Resource": [
        "arn:aws:ec2:*:123456789012:instance/*",
        "arn:aws:ssm:*:123456789012:document/AWS-RunShellScript",
        "arn:aws:ssm:*:123456789012:document/AWS-StartInteractiveCommand",
        "arn:aws:ssm:*::document/AWS-StartSSHSession"
      ],
      "Condition": {
        "StringEquals": {
          "ec2:ResourceTag/Environment": "production"
        }
      }
    }
  ]
}

Tag-based access control — restrict sessions to instances tagged with the operator’s username:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
{
  "Effect": "Allow",
  "Action": "ssm:StartSession",
  "Resource": "arn:aws:ec2:*:123456789012:instance/*",
  "Condition": {
    "StringEquals": {
      "ec2:ResourceTag/AllowedUser": "${aws:username}"
    }
  }
}

With this policy, each operator can only open sessions on instances explicitly tagged with their IAM username. No group-level access, no shared account.


Fleet Manager

Fleet Manager is the console UI built on top of the SSM APIs. It provides a browser-based file system browser, process viewer, user and group management, and — on Windows — a registry editor. It is useful for ad-hoc investigation without installing additional tooling or opening additional network access.

The file system browser supports viewing, creating, editing, and deleting files up to a configurable size limit. The tail feature streams new lines from a log file in near-real-time — a reasonable substitute for tail -f when you need to check an application log without a full session.

For Windows environments, the registry editor eliminates the most common reason for RDP access: making a configuration change that requires a registry key edit. Combined with Session Manager’s port forwarding (for RDP tunneling when the GUI is genuinely necessary), Fleet Manager covers most Windows management needs from the AWS console without any inbound network access.

Comments