Incident Response Automation: Self-Healing Infrastructure with AWS, Ansible, and Observable Feedback Loops

May 23, 2026 37 min read Security

awsincident-responseautomationsecurityssmeventbridgelambdaansibleguarddutysecurity-hubsredevops

Contents

Incidents do not wait for business hours, and humans do not respond at machine speed. The gap between detection and containment — measured in minutes during a compromise — is where real damage happens. Automated incident response closes that gap, but it introduces its own failure mode: automation that makes things worse faster. This post covers the full stack: the runbook authoring model, the event routing layer, Lambda remediation patterns, Ansible for ad-hoc response, and the observability you need to trust any of it.

1. AWS Systems Manager Automation Runbooks

SSM Automation is the orchestration backbone for AWS incident response. Runbooks are YAML or JSON documents at schemaVersion: "0.3" (the only version that supports the full Automation action set). Each document defines parameters, an assumeRole (the IAM role the runbook executes as), and mainSteps, where each step maps to one action type.

1.1 Document Skeleton

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


---
description: "EC2 Incident Response — Isolate and Snapshot"
schemaVersion: "0.3"
assumeRole: "{{ AutomationAssumeRole }}"
parameters:
  AutomationAssumeRole:
    type: String
    description: "IAM role ARN for SSM Automation"
  InstanceId:
    type: String
    description: "EC2 instance ID to isolate"
  IsolationSecurityGroupId:
    type: String
    description: "Pre-created isolation SG with no ingress, SSM-only egress"
  SnsTopic:
    type: String
    description: "ARN of SNS topic for notifications"
mainSteps: []   # see below

The assumeRole is critical and is separate from the EventBridge execution role. The automation role needs permissions for every API call in the runbook. The EventBridge execution role needs only ssm:StartAutomationExecution and iam:PassRole.

1.2 Core Action Types

`aws:executeAwsApi`

Calls any AWS API synchronously and can chain outputs to later steps. The Selector field is a JMESPath expression against the API response.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


- name: getInstanceDetails
  action: aws:executeAwsApi
  onFailure: Abort
  inputs:
    Service: ec2
    Api: DescribeInstances
    InstanceIds:
      - "{{ InstanceId }}"
  outputs:
    - Name: VpcId
      Selector: "$.Reservations[0].Instances[0].VpcId"
      Type: String
    - Name: PrimaryENI
      Selector: "$.Reservations[0].Instances[0].NetworkInterfaces[0].NetworkInterfaceId"
      Type: String
    - Name: RootVolumeId
      Selector: "$.Reservations[0].Instances[0].BlockDeviceMappings[0].Ebs.VolumeId"
      Type: String

`aws:waitForAwsResourceProperty`

Polls an API until a property reaches a desired value. Use this after any state-changing call — stopping an instance, detaching a volume, creating a snapshot.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


- name: waitForInstanceStopped
  action: aws:waitForAwsResourceProperty
  timeoutSeconds: 300
  inputs:
    Service: ec2
    Api: DescribeInstances
    InstanceIds:
      - "{{ InstanceId }}"
    PropertySelector: "$.Reservations[0].Instances[0].State.Name"
    DesiredValues:
      - stopped

Gotcha: timeoutSeconds counts from when the step starts, not from the API call. If polling takes longer than the timeout, the step fails and onFailure applies. Always set this generously for large EBS volumes and slow instance transitions.

`aws:branch`

Conditional routing based on step outputs or parameter values. Supports StringEquals, StringNotEquals, Contains, StartsWith, NumericEquals, BooleanEquals, and their negations.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


- name: checkFindingType
  action: aws:branch
  inputs:
    Choices:
      - NextStep: isolateInstance
        Variable: "{{ FindingType }}"
        Contains: "UnauthorizedAccess"
      - NextStep: snapshotAndTerminate
        Variable: "{{ FindingType }}"
        Contains: "CryptoCurrency"
    Default: notifyAndPause

`aws:approve`

Pauses automation and sends an SNS notification. Approvers listed by IAM user name, user ARN, or role ARN respond via the SSM console or API. Default timeout is 7 days; max is 30 days. Use this before destructive actions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


- name: requireApprovalBeforeTermination
  action: aws:approve
  timeoutSeconds: 3600    # 1 hour for urgent incidents
  onFailure: Abort
  inputs:
    NotificationArn: "arn:aws:sns:us-east-1:123456789012:Automation-Approvals"
    Message: "Instance {{ InstanceId }} flagged by GuardDuty finding {{ FindingId }}. Approve termination or reject to keep for forensics."
    MinRequiredApprovals: 1
    Approvers:
      - "arn:aws:iam::123456789012:role/SecurityIncidentResponder"
      - "arn:aws:iam::123456789012:user/oncall-engineer"

Note: aws:approve does not support multi-account and multi-region automations. For cross-account runbooks, use a Step Functions state machine with a human approval task instead.

`aws:sleep`

A simple delay. Useful for letting IAM policy propagation settle before making an authenticated call, or for rate-limiting retries.

1
2
3
4


- name: waitForPolicyPropagation
  action: aws:sleep
  inputs:
    Duration: PT30S    # ISO 8601 duration — 30 seconds

`aws:runCommand`

Runs an SSM document (typically AWS-RunShellScript or AWS-RunPowerShellScript) on one or more managed instances. The instance must have SSM Agent running and an instance profile with AmazonSSMManagedInstanceCore. Does not require SSH or open inbound ports.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


- name: collectForensicData
  action: aws:runCommand
  timeoutSeconds: 600
  inputs:
    DocumentName: AWS-RunShellScript
    InstanceIds:
      - "{{ InstanceId }}"
    Parameters:
      commands:
        - |
          #!/bin/bash
          TIMESTAMP=$(date +%Y%m%d-%H%M%S)
          OUTDIR="/tmp/forensics-${TIMESTAMP}"
          mkdir -p "$OUTDIR"
          ps auxww > "$OUTDIR/ps.txt"
          ss -tulpn > "$OUTDIR/netstat.txt"
          last -n 50 > "$OUTDIR/last.txt"
          cat /etc/crontab /etc/cron.d/* 2>/dev/null > "$OUTDIR/crons.txt"
          find /tmp /var/tmp -maxdepth 2 -newer /proc/1 2>/dev/null > "$OUTDIR/recent_tmp.txt"
          tar czf "/tmp/forensics-${TIMESTAMP}.tar.gz" "$OUTDIR"
          aws s3 cp "/tmp/forensics-${TIMESTAMP}.tar.gz" \
            "s3://incident-forensics-bucket/{{ InstanceId }}/${TIMESTAMP}/" \
            --region us-east-1

`aws:invokeLambdaFunction`

Invokes a Lambda function synchronously and can capture its output for use in later steps.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


- name: enrichFindingWithThreatIntel
  action: aws:invokeLambdaFunction
  inputs:
    FunctionName: "arn:aws:lambda:us-east-1:123456789012:function:ThreatIntelEnrichment"
    InvocationType: RequestResponse
    Payload: |
      {
        "instanceId": "{{ InstanceId }}",
        "findingType": "{{ FindingType }}",
        "accountId": "{{ global:ACCOUNT_ID }}"
      }
  outputs:
    - Name: ThreatScore
      Selector: "$.Payload.threatScore"
      Type: Integer
    - Name: RecommendedAction
      Selector: "$.Payload.recommendedAction"
      Type: String

`aws:executeScript`

Embeds Python (3.8, 3.11) or PowerShell directly in the runbook. Useful for logic too complex for aws:branch but too small to warrant a Lambda function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


- name: calculateBlastRadius
  action: aws:executeScript
  timeoutSeconds: 60
  inputs:
    Runtime: python3.11
    Handler: calculate_blast_radius
    InputPayload:
      instanceId: "{{ InstanceId }}"
      region: "{{ global:REGION }}"
    Script: |
      import boto3

      def calculate_blast_radius(events, context):
          ec2 = boto3.client('ec2', region_name=events['region'])
          instance_id = events['instanceId']
          
          # Find all instances sharing the same IAM role
          instance = ec2.describe_instances(
              InstanceIds=[instance_id]
          )['Reservations'][0]['Instances'][0]
          
          role_name = None
          if 'IamInstanceProfile' in instance:
              profile_arn = instance['IamInstanceProfile']['Arn']
              role_name = profile_arn.split('/')[-1]
          
          affected_count = 1
          if role_name:
              all_instances = ec2.describe_instances(
                  Filters=[{'Name': 'iam-instance-profile.arn', 
                            'Values': [instance['IamInstanceProfile']['Arn']]}]
              )
              affected_count = sum(
                  len(r['Instances']) for r in all_instances['Reservations']
              )
          
          return {
              'affectedInstanceCount': affected_count,
              'sharedRoleName': role_name or 'none',
              'requiresApproval': affected_count > 5
          }
  outputs:
    - Name: AffectedInstanceCount
      Selector: "$.Payload.affectedInstanceCount"
      Type: Integer
    - Name: RequiresApproval
      Selector: "$.Payload.requiresApproval"
      Type: Boolean

1.3 Step Output Chaining

Outputs from step stepName are referenced downstream as {{ stepName.OutputName }}. SSM Automation also provides global variables:

{{ global:ACCOUNT_ID }} — current AWS account ID
{{ global:DATE }} — execution date (UTC, YYYY-MM-DD)
{{ global:DATE_TIME }} — execution datetime (UTC, YYYY-MM-DDTHH:mm:ssZ)
{{ global:REGION }} — execution region
{{ automation:EXECUTION_ID }} — the automation execution ID (useful for tagging resources)

1.4 Complete EC2 Isolation Runbook

This runbook represents the isolation phase of a real incident response workflow: snapshot, isolate, notify.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114


---
description: "EC2 Incident Response: Snapshot, Isolate, Notify"
schemaVersion: "0.3"
assumeRole: "{{ AutomationAssumeRole }}"
parameters:
  AutomationAssumeRole:
    type: String
  InstanceId:
    type: String
  IsolationSgId:
    type: String
    description: "Pre-created SG: no ingress, SSM VPC endpoint egress only"
  SnsTopicArn:
    type: String
  FindingId:
    type: String
    default: "manual"
mainSteps:
  - name: getInstanceDetails
    action: aws:executeAwsApi
    onFailure: Abort
    inputs:
      Service: ec2
      Api: DescribeInstances
      InstanceIds: ["{{ InstanceId }}"]
    outputs:
      - Name: PrimaryEniId
        Selector: "$.Reservations[0].Instances[0].NetworkInterfaces[0].NetworkInterfaceId"
        Type: String
      - Name: RootVolumeId
        Selector: "$.Reservations[0].Instances[0].BlockDeviceMappings[0].Ebs.VolumeId"
        Type: String

  - name: tagInstanceAsCompromised
    action: aws:executeAwsApi
    onFailure: Continue   # tagging failure is non-critical
    inputs:
      Service: ec2
      Api: CreateTags
      Resources: ["{{ InstanceId }}"]
      Tags:
        - Key: "incident:status"
          Value: "COMPROMISED"
        - Key: "incident:finding-id"
          Value: "{{ FindingId }}"
        - Key: "incident:isolation-time"
          Value: "{{ global:DATE_TIME }}"
        - Key: "incident:automation-id"
          Value: "{{ automation:EXECUTION_ID }}"
    nextStep: createForensicSnapshot

  - name: createForensicSnapshot
    action: aws:executeAwsApi
    onFailure: Continue   # snapshot failure must not block isolation
    inputs:
      Service: ec2
      Api: CreateSnapshot
      VolumeId: "{{ getInstanceDetails.RootVolumeId }}"
      Description: "Incident forensic snapshot - {{ InstanceId }} - {{ FindingId }}"
      TagSpecifications:
        - ResourceType: snapshot
          Tags:
            - Key: "incident:instance-id"
              Value: "{{ InstanceId }}"
    outputs:
      - Name: SnapshotId
        Selector: "$.SnapshotId"
        Type: String
    nextStep: isolateInstance

  - name: isolateInstance
    action: aws:executeAwsApi
    onFailure: Abort
    isCritical: true
    inputs:
      Service: ec2
      Api: ModifyNetworkInterfaceAttribute
      NetworkInterfaceId: "{{ getInstanceDetails.PrimaryEniId }}"
      Groups: ["{{ IsolationSgId }}"]
    nextStep: verifyIsolation

  - name: verifyIsolation
    action: aws:executeAwsApi
    onFailure: Abort
    inputs:
      Service: ec2
      Api: DescribeNetworkInterfaces
      NetworkInterfaceIds: ["{{ getInstanceDetails.PrimaryEniId }}"]
    outputs:
      - Name: CurrentSgs
        Selector: "$.NetworkInterfaces[0].Groups[0].GroupId"
        Type: String
    nextStep: sendNotification

  - name: sendNotification
    action: aws:executeAwsApi
    onFailure: Continue
    inputs:
      Service: sns
      Api: Publish
      TopicArn: "{{ SnsTopicArn }}"
      Subject: "[INCIDENT] EC2 Instance Isolated: {{ InstanceId }}"
      Message: |
        Instance {{ InstanceId }} has been automatically isolated.

        Finding ID: {{ FindingId }}
        Snapshot: {{ createForensicSnapshot.SnapshotId }}
        Isolation ENI: {{ getInstanceDetails.PrimaryEniId }}
        Applied SG: {{ IsolationSgId }}
        Execution: {{ automation:EXECUTION_ID }}

        Session Manager access remains available via VPC endpoints.
        CloudTrail audit trail: search for execution ID above.
    isEnd: true

1.5 Triggering Runbooks from the CLI

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# Start manually
aws ssm start-automation-execution \
  --document-name "EC2-IncidentResponse-Isolate" \
  --parameters \
    "InstanceId=i-0abc123def456789a,\
     IsolationSgId=sg-0quarantine1234,\
     SnsTopicArn=arn:aws:sns:us-east-1:123456789012:incident-alerts,\
     AutomationAssumeRole=arn:aws:iam::123456789012:role/SSMAutomationRole"

# Check execution status
aws ssm get-automation-execution \
  --automation-execution-id "abc12345-1234-1234-1234-abc123456789"

# List recent executions
aws ssm describe-automation-executions \
  --filters "Key=DocumentNamePrefix,Values=EC2-Incident" \
  --max-items 20

2. EventBridge: The Event Routing Layer

EventBridge is the glue between detection services and remediation actions. Every AWS security service publishes to the default event bus in near-real time. EventBridge evaluates rules against incoming events and routes matches to targets.

2.1 GuardDuty Finding Event Pattern

GuardDuty publishes all findings as detail-type: "GuardDuty Finding" with source: "aws.guardduty". The detail field is the full GuardDuty finding JSON.

Pattern for high-severity EC2 findings:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


{
  "source": ["aws.guardduty"],
  "detail-type": ["GuardDuty Finding"],
  "detail": {
    "severity": [
      7, 7.0, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8, 7.9,
      8, 8.0, 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7, 8.8, 8.9,
      9, 9.0, 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7, 9.8, 9.9,
      10, 10.0
    ],
    "resource": {
      "resourceType": ["Instance"]
    }
  }
}

Why enumerate severity values instead of using a numeric range? EventBridge event patterns do not support numeric range comparisons in the basic pattern syntax for all fields. GuardDuty severities are floats (7.0 through 7.9 for high), and you must list them explicitly, or use EventBridge content filtering with { "numeric": [">", 6.9] } syntax in supported contexts.

Pattern with numeric comparison (preferred approach):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


{
  "source": ["aws.guardduty"],
  "detail-type": ["GuardDuty Finding"],
  "detail": {
    "severity": [{ "numeric": [">=", 7] }],
    "resource": {
      "resourceType": ["Instance"]
    }
  }
}

Pattern for a specific finding type:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


{
  "source": ["aws.guardduty"],
  "detail-type": ["GuardDuty Finding"],
  "detail": {
    "type": [
      { "prefix": "UnauthorizedAccess:EC2" },
      { "prefix": "Backdoor:EC2" },
      { "prefix": "CryptoCurrency:EC2" },
      "Execution:EC2/MaliciousFile"
    ]
  }
}

2.2 Security Hub Finding Event Pattern

Security Hub uses detail-type: "Security Hub Findings - Imported" and wraps findings in an array under detail.findings. The finding format is ASFF (Amazon Security Finding Format).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


{
  "source": ["aws.securityhub"],
  "detail-type": ["Security Hub Findings - Imported"],
  "detail": {
    "findings": {
      "Compliance": {
        "Status": ["FAILED"]
      },
      "Severity": {
        "Label": ["HIGH", "CRITICAL"]
      },
      "RecordState": ["ACTIVE"],
      "WorkflowState": ["NEW"]
    }
  }
}

Pattern for CIS benchmark security group violations (controls 4.1 and 4.2):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


{
  "source": ["aws.securityhub"],
  "detail-type": ["Security Hub Findings - Imported"],
  "detail": {
    "findings": {
      "GeneratorId": [
        { "prefix": "arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark/v/1.2.0/rule/4.1" },
        { "prefix": "arn:aws:securityhub:::ruleset/cis-aws-foundations-benchmark/v/1.2.0/rule/4.2" }
      ],
      "Compliance": {
        "Status": ["FAILED"]
      }
    }
  }
}

2.3 AWS Config Non-Compliance Pattern

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


{
  "source": ["aws.config"],
  "detail-type": ["Config Rules Compliance Change"],
  "detail": {
    "messageType": ["ComplianceChangeNotification"],
    "newEvaluationResult": {
      "complianceType": ["NON_COMPLIANT"]
    },
    "configRuleName": [
      "restricted-ssh",
      "restricted-common-ports",
      "vpc-sg-open-only-to-authorized-ports"
    ]
  }
}

2.4 Input Transformer: Passing Event Data to SSM Automation

When the EventBridge target is SSM Automation, use an Input Transformer to extract fields from the event and pass them as runbook parameters.

For a GuardDuty finding targeting an EC2 instance:

Input Path (extracts values from event JSON):

1
2
3
4
5
6
7


{
  "instance": "$.detail.resource.instanceDetails.instanceId",
  "finding": "$.detail.id",
  "type": "$.detail.type",
  "account": "$.account",
  "region": "$.region"
}

Input Template (maps to SSM parameter names — note the array wrapping for StringList params):

{"InstanceId":["<instance>"],"FindingId":["<finding>"],"FindingType":["<type>"]}

CLI example to create the rule with an SSM Automation target:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


aws events put-targets \
  --rule "guardduty-high-severity-ec2" \
  --targets '[
    {
      "Id": "SSMAutomationTarget",
      "Arn": "arn:aws:ssm:us-east-1:123456789012:automation-definition/EC2-IncidentResponse-Isolate:$DEFAULT",
      "RoleArn": "arn:aws:iam::123456789012:role/EventBridgeSSMRole",
      "InputTransformer": {
        "InputPathsMap": {
          "instance": "$.detail.resource.instanceDetails.instanceId",
          "finding": "$.detail.id",
          "type": "$.detail.type"
        },
        "InputTemplate": "{\"InstanceId\":[\"<instance>\"],\"FindingId\":[\"<finding>\"],\"FindingType\":[\"<type>\"],\"IsolationSgId\":[\"sg-0quarantine1234\"],\"SnsTopicArn\":[\"arn:aws:sns:us-east-1:123456789012:incident-alerts\"],\"AutomationAssumeRole\":[\"arn:aws:iam::123456789012:role/SSMAutomationRole\"]}"
      },
      "DeadLetterConfig": {
        "Arn": "arn:aws:sqs:us-east-1:123456789012:incident-response-dlq"
      },
      "RetryPolicy": {
        "MaximumRetryAttempts": 3,
        "MaximumEventAgeInSeconds": 900
      }
    }
  ]'

Gotcha: The InputTemplate for SSM Automation wraps each parameter value in a JSON array (["<value>"]). This is because SSM parameters are internally typed as StringList when passed from EventBridge. Forgetting the array brackets is the single most common cause of “invalid parameter” errors in this integration.

2.5 Dead-Letter Queues for Failed Invocations

Every EventBridge rule target should have a DLQ. Without one, failed invocations — due to throttling, permission errors, or target unavailability — are silently dropped.

SQS resource policy required on the DLQ:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowEventBridgeToDLQ",
    "Effect": "Allow",
    "Principal": {"Service": "events.amazonaws.com"},
    "Action": "sqs:SendMessage",
    "Resource": "arn:aws:sqs:us-east-1:123456789012:incident-response-dlq",
    "Condition": {
      "ArnEquals": {
        "aws:SourceArn": "arn:aws:events:us-east-1:123456789012:rule/guardduty-high-severity-ec2"
      }
    }
  }]
}

EventBridge adds message attributes to DLQ messages: RULE_ARN, TARGET_ARN, ERROR_CODE (e.g., NO_PERMISSIONS, THROTTLING, TIMEOUT), ERROR_MESSAGE, RETRY_ATTEMPTS, and EXHAUSTED_RETRY_CONDITION. Use these to build a CloudWatch alarm on InvocationsSentToDLQ — this is your automation’s own health check.

3. Lambda Auto-Remediation Patterns

Lambda is the Swiss army knife of incident response: stateless, fast to deploy, and able to call any AWS API. The key discipline is keeping each function narrowly scoped to one finding type and one remediation action.

3.1 EC2 Quarantine: The Complete Lambda Pattern

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137


import boto3
import json
import logging
import os
from datetime import datetime, timezone

logger = logging.getLogger()
logger.setLevel(logging.INFO)

ec2 = boto3.client('ec2')
sns = boto3.client('sns')

ISOLATION_SG = os.environ['ISOLATION_SG_ID']
SNS_TOPIC = os.environ['SNS_TOPIC_ARN']
FORENSICS_BUCKET = os.environ['FORENSICS_BUCKET']

def lambda_handler(event, context):
    logger.info("Event: %s", json.dumps(event))

    # GuardDuty finding arrives under event['detail']
    detail = event.get('detail', {})
    finding_id = detail.get('id', 'unknown')
    finding_type = detail.get('type', 'unknown')
    severity = detail.get('severity', 0)

    resource = detail.get('resource', {})
    instance_details = resource.get('instanceDetails', {})
    instance_id = instance_details.get('instanceId')

    if not instance_id:
        logger.error("No instanceId in finding %s — cannot remediate", finding_id)
        raise ValueError(f"Missing instanceId in finding {finding_id}")

    logger.info("Isolating instance %s for finding %s (type: %s, severity: %s)",
                instance_id, finding_id, finding_type, severity)

    results = {
        'instanceId': instance_id,
        'findingId': finding_id,
        'findingType': finding_type,
        'timestamp': datetime.now(timezone.utc).isoformat(),
        'actions': []
    }

    # 1. Tag instance before anything else — creates audit trail even if later steps fail
    try:
        ec2.create_tags(
            Resources=[instance_id],
            Tags=[
                {'Key': 'incident:status', 'Value': 'QUARANTINED'},
                {'Key': 'incident:finding-id', 'Value': finding_id},
                {'Key': 'incident:finding-type', 'Value': finding_type},
                {'Key': 'incident:quarantine-time', 'Value': results['timestamp']},
                {'Key': 'incident:lambda-request-id', 'Value': context.aws_request_id}
            ]
        )
        results['actions'].append('tagged')
    except Exception as e:
        logger.warning("Failed to tag instance %s: %s", instance_id, e)
        # non-fatal — continue to isolation

    # 2. Snapshot all volumes
    snapshot_ids = []
    try:
        instance_data = ec2.describe_instances(
            InstanceIds=[instance_id]
        )['Reservations'][0]['Instances'][0]

        for bdm in instance_data.get('BlockDeviceMappings', []):
            vol_id = bdm['Ebs']['VolumeId']
            snap = ec2.create_snapshot(
                VolumeId=vol_id,
                Description=f"Incident {finding_id} — {instance_id} — {bdm['DeviceName']}",
                TagSpecifications=[{
                    'ResourceType': 'snapshot',
                    'Tags': [
                        {'Key': 'incident:instance-id', 'Value': instance_id},
                        {'Key': 'incident:finding-id', 'Value': finding_id}
                    ]
                }]
            )
            snapshot_ids.append(snap['SnapshotId'])
            logger.info("Created snapshot %s for volume %s", snap['SnapshotId'], vol_id)

        results['actions'].append(f"snapshots:{','.join(snapshot_ids)}")
    except Exception as e:
        logger.error("Snapshot failed for %s: %s", instance_id, e)
        # non-fatal — isolation is more important than snapshots

    # 3. Replace all ENI security groups with isolation SG
    # This is the critical containment step — must succeed
    try:
        instance_data = ec2.describe_instances(
            InstanceIds=[instance_id]
        )['Reservations'][0]['Instances'][0]

        for eni in instance_data.get('NetworkInterfaces', []):
            eni_id = eni['NetworkInterfaceId']
            original_sgs = [sg['GroupId'] for sg in eni['Groups']]
            logger.info("Replacing SGs %s on %s with isolation SG", original_sgs, eni_id)

            ec2.modify_network_interface_attribute(
                NetworkInterfaceId=eni_id,
                Groups=[ISOLATION_SG]
            )
            # Tag the ENI with original SGs so responders can restore if needed
            ec2.create_tags(
                Resources=[eni_id],
                Tags=[{
                    'Key': 'incident:original-sgs',
                    'Value': ','.join(original_sgs)
                }]
            )

        results['actions'].append('isolated')
    except Exception as e:
        logger.error("CRITICAL: Failed to isolate %s: %s", instance_id, e)
        raise   # re-raise — isolation failure means remediation failed

    # 4. Notify
    message = json.dumps({
        'summary': f'EC2 instance {instance_id} automatically isolated',
        'details': results
    }, indent=2)

    try:
        sns.publish(
            TopicArn=SNS_TOPIC,
            Subject=f'[INCIDENT] Auto-isolated {instance_id} | {finding_type}',
            Message=message
        )
        results['actions'].append('notified')
    except Exception as e:
        logger.warning("SNS notification failed: %s", e)

    logger.info("Remediation complete: %s", json.dumps(results))
    return results

3.2 IAM Role Isolation: Revoking Active Sessions

When GuardDuty or Security Hub flags IAM credentials as compromised, the immediate goal is to invalidate all active sessions without a race condition. AWS IAM’s AWSRevokeOlderSessions mechanism does this via a time-based deny condition.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100


import boto3
import json
import logging
from datetime import datetime, timezone, timedelta

logger = logging.getLogger()
logger.setLevel(logging.INFO)

iam = boto3.client('iam')

def revoke_iam_role_sessions(role_name: str, finding_id: str) -> dict:
    """
    Attaches a deny-all policy conditioned on TokenIssueTime to revoke all
    existing sessions for a role. New sessions (assumed after this call +30s)
    are unaffected.
    
    Returns a dict with the policy ARN and revocation timestamp.
    """
    # Use "now minus a small buffer" to catch any sessions created moments ago
    # AWS recommends using current time; the propagation buffer (~30s) is handled internally
    revocation_time = datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ')

    revoke_policy = {
        "Version": "2012-10-17",
        "Statement": [{
            "Sid": "AWSRevokeOlderSessions",
            "Effect": "Deny",
            "Action": "*",
            "Resource": "*",
            "Condition": {
                "DateLessThan": {
                    "aws:TokenIssueTime": revocation_time
                }
            }
        }]
    }

    # Put as an inline policy (inline cannot be detached without PutRolePolicy permission)
    policy_name = f"AWSRevokeOlderSessions"
    iam.put_role_policy(
        RoleName=role_name,
        PolicyName=policy_name,
        PolicyDocument=json.dumps(revoke_policy)
    )
    logger.info("Attached session revocation policy to role %s at %s", role_name, revocation_time)

    return {
        'roleName': role_name,
        'revocationTime': revocation_time,
        'policyName': policy_name
    }


def quarantine_iam_user(username: str, finding_id: str) -> dict:
    """
    Full IAM user quarantine:
    1. Attach deny-all managed policy
    2. Disable all access keys
    3. Delete console password (if login profile exists)
    """
    results = {'username': username, 'actions': []}

    # Attach inline deny-all (prefer inline over managed so SCP can protect it)
    deny_all = {
        "Version": "2012-10-17",
        "Statement": [{
            "Sid": "IncidentResponseQuarantine",
            "Effect": "Deny",
            "Action": "*",
            "Resource": "*"
        }]
    }
    iam.put_user_policy(
        UserName=username,
        PolicyName='IncidentResponseQuarantine',
        PolicyDocument=json.dumps(deny_all)
    )
    results['actions'].append('attached-deny-all')

    # Disable all access keys
    paginator = iam.get_paginator('list_access_keys')
    for page in paginator.paginate(UserName=username):
        for key in page['AccessKeyMetadata']:
            if key['Status'] == 'Active':
                iam.update_access_key(
                    UserName=username,
                    AccessKeyId=key['AccessKeyId'],
                    Status='Inactive'
                )
                logger.info("Disabled access key %s for %s", key['AccessKeyId'], username)
                results['actions'].append(f"disabled-key:{key['AccessKeyId']}")

    # Disable console access
    try:
        iam.delete_login_profile(UserName=username)
        results['actions'].append('deleted-console-password')
    except iam.exceptions.NoSuchEntityException:
        pass   # user had no console password

    return results

Eventual consistency gotcha: IAM policy propagation takes up to a few seconds. An attacker with iam:PutRolePolicy can race the defender and remove the inline policy. The countermeasure is an SCP (Service Control Policy) at the AWS Organizations level that denies iam:DeleteRolePolicy and iam:DetachRolePolicy on the quarantine policy ARN, with an exemption only for your incident response role. This makes the quarantine policy irremovable by the compromised identity even if it holds administrator privileges within the account.

3.3 Security Group Remediation: Removing Overly Permissive Rules

This Lambda function is triggered by Security Hub or Config findings for CIS controls 4.1 (no SSH from 0.0.0.0/0) and 4.2 (no RDP from 0.0.0.0/0):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66


import boto3
import json
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

ec2 = boto3.client('ec2')

# Ports that must not be open to the world
RESTRICTED_PORTS = {22, 3389, 23, 21, 1433, 3306, 5432, 6379, 27017}
WORLD_CIDRS = ['0.0.0.0/0', '::/0']

def lambda_handler(event, context):
    # Parse Security Hub ASFF finding
    findings = event.get('detail', {}).get('findings', [])

    for finding in findings:
        resources = finding.get('Resources', [])
        for resource in resources:
            if resource['Type'] == 'AwsEc2SecurityGroup':
                sg_id = resource['Id'].split('/')[-1]
                remediate_security_group(sg_id, finding['Id'])


def remediate_security_group(sg_id: str, finding_id: str):
    sg_data = ec2.describe_security_groups(GroupIds=[sg_id])['SecurityGroups'][0]
    removed = []

    for rule in sg_data.get('IpPermissions', []):
        port = rule.get('FromPort')
        if port is None:
            continue

        # Check for world-open rules on restricted ports
        world_ipv4 = any(
            r['CidrIp'] in WORLD_CIDRS
            for r in rule.get('IpRanges', [])
        )
        world_ipv6 = any(
            r['CidrIpv6'] in WORLD_CIDRS
            for r in rule.get('Ipv6Ranges', [])
        )

        if (world_ipv4 or world_ipv6) and port in RESTRICTED_PORTS:
            logger.info("Removing rule: port %s world-open on %s (finding: %s)",
                       port, sg_id, finding_id)
            try:
                ec2.revoke_security_group_ingress(
                    GroupId=sg_id,
                    IpPermissions=[rule]
                )
                removed.append(f"port:{port}")
            except Exception as e:
                logger.error("Failed to revoke rule on %s: %s", sg_id, e)

    if removed:
        ec2.create_tags(
            Resources=[sg_id],
            Tags=[{
                'Key': 'incident:auto-remediated',
                'Value': f"removed:{','.join(removed)}"
            }]
        )

    return removed

3.4 Full Flow: GuardDuty Finding to Remediation with Audit Trail

The complete flow for an UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration finding:

GuardDuty detects unusual API calls from EC2 instance credentials
  |
  └─> EventBridge rule (source: aws.guardduty, type prefix: UnauthorizedAccess)
        |
        └─> Target 1: Lambda (IAM-Isolation)
        |     Extracts instanceId, roleArn from finding
        |     Calls revoke_iam_role_sessions() for the instance role
        |     Disables all access keys issued by the instance profile
        |     Writes finding enrichment back to Security Hub via BatchImportFindings
        |
        └─> Target 2: SSM Automation (EC2-IncidentResponse-Isolate)
        |     Snapshots volumes
        |     Replaces security groups with isolation SG
        |     Tags instance with incident metadata
        |
        └─> Target 3: SNS (immediate human notification)
              Sends to PagerDuty/OpsGenie via webhook subscription
              Triggers on-call rotation escalation

All API calls appear in CloudTrail under the Lambda execution role and SSM
automation role — providing a complete audit trail tied back to the
GuardDuty finding ID.

4. Ansible for Incident Response

Ansible’s strength in incident response is not orchestration (Lambda and SSM handle that better at scale) — it is the ability to run complex, multi-step, stateful procedures across many hosts simultaneously using existing playbook infrastructure. Combined with the SSM connection plugin, you can do this without SSH, without opening ports, and from a controller node that has no network path to the target hosts.

4.1 Ansible with SSM Agent as Transport

Install the amazon.aws collection and the SSM Session Manager plugin:

1
2
3
4
5


ansible-galaxy collection install amazon.aws
# Install Session Manager plugin (Linux x86_64)
curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/ubuntu_64bit/session-manager-plugin.deb" \
  -o session-manager-plugin.deb
sudo dpkg -i session-manager-plugin.deb

ansible.cfg:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


[defaults]
inventory = ./inventory/
remote_user = ec2-user
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_facts

[ssh_connection]
# SSM plugin does not use SSH — but pipelining can be enabled for performance
pipelining = false

Dynamic inventory file (inventory/aws_ec2.yaml):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


plugin: amazon.aws.aws_ec2
regions:
  - us-east-1
  - us-west-2
filters:
  instance-state-name: running
  tag:Environment: production
keyed_groups:
  - key: tags.Role
    prefix: role
  - key: tags['incident:status']
    prefix: incident
compose:
  ansible_host: instance_id
hostnames:
  - tag:Name
  - instance_id
# SSM transport settings applied to all hosts
vars:
  ansible_connection: amazon.aws.aws_ssm
  ansible_aws_ssm_bucket_name: "ansible-ssm-transfers-{{ account_id }}"
  ansible_aws_ssm_region: "{{ placement.region }}"
  ansible_python_interpreter: /usr/bin/python3

The ansible_aws_ssm_bucket_name is required by the plugin — it uses S3 as an intermediate for file transfers. Set up lifecycle policies on this bucket to auto-delete files after 1 day.

4.2 Emergency Forensic Collection Playbook

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94


---
# forensic-collect.yaml
# Usage: ansible-playbook forensic-collect.yaml -l "tag_incident_status_COMPROMISED" --forks 10
- name: Collect Forensic Data from Compromised Hosts
  hosts: all
  gather_facts: false    # skip slow fact gathering for urgent response
  become: true
  vars:
    forensics_bucket: "incident-forensics-{{ lookup('env', 'AWS_ACCOUNT_ID') }}"
    timestamp: "{{ lookup('pipe', 'date +%Y%m%d-%H%M%S') }}"

  tasks:
    - name: Create forensics staging directory
      ansible.builtin.file:
        path: "/tmp/forensics-{{ timestamp }}"
        state: directory
        mode: '0700'
      register: forensics_dir

    - name: Capture running processes
      ansible.builtin.command: ps auxwwef
      register: ps_output
      changed_when: false

    - name: Capture network connections
      ansible.builtin.command: ss -tulpnW
      register: netstat_output
      changed_when: false

    - name: Capture listening processes with lsof
      ansible.builtin.command: lsof -i -P -n
      register: lsof_output
      changed_when: false
      ignore_errors: true

    - name: Capture recent auth logs
      ansible.builtin.command: journalctl -u ssh --since "2 hours ago" --no-pager
      register: auth_logs
      changed_when: false
      ignore_errors: true

    - name: Find recently modified files in /tmp and /var/tmp
      ansible.builtin.find:
        paths:
          - /tmp
          - /var/tmp
          - /dev/shm
        age: -3600    # modified in last hour
        recurse: true
      register: recent_files

    - name: Collect cron jobs
      ansible.builtin.shell: |
        crontab -l 2>/dev/null || true
        cat /etc/crontab /etc/cron.d/* 2>/dev/null || true
        ls -la /var/spool/cron/crontabs/ 2>/dev/null || true
      register: cron_data
      changed_when: false

    - name: Capture bash history for all users
      ansible.builtin.shell: |
        for f in /home/*/.bash_history /root/.bash_history; do
          [ -f "$f" ] && echo "=== $f ===" && cat "$f"
        done
      register: bash_history
      changed_when: false

    - name: Write forensic report
      ansible.builtin.copy:
        dest: "{{ forensics_dir.path }}/report.json"
        content: "{{ {
          'host': inventory_hostname,
          'timestamp': timestamp,
          'processes': ps_output.stdout_lines,
          'network': netstat_output.stdout_lines,
          'lsof': lsof_output.stdout_lines | default([]),
          'recent_files': recent_files.files | map(attribute='path') | list,
          'crons': cron_data.stdout_lines,
          'bash_history': bash_history.stdout_lines
        } | to_nice_json }}"

    - name: Archive forensic data
      ansible.builtin.archive:
        path: "{{ forensics_dir.path }}"
        dest: "/tmp/forensics-{{ inventory_hostname }}-{{ timestamp }}.tar.gz"
        format: gz

    - name: Upload to S3
      amazon.aws.s3_object:
        bucket: "{{ forensics_bucket }}"
        object: "{{ inventory_hostname }}/{{ timestamp }}/forensics.tar.gz"
        src: "/tmp/forensics-{{ inventory_hostname }}-{{ timestamp }}.tar.gz"
        mode: put
        region: us-east-1

4.3 Emergency Credential Rotation Playbook

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


---
# rotate-credentials.yaml
# Usage: ansible-playbook rotate-credentials.yaml -l webservers --serial 5
- name: Emergency Credential Rotation
  hosts: all
  become: true
  serial: 5    # rotate 5 hosts at a time to limit blast radius
  max_fail_percentage: 10   # abort if >10% of hosts fail

  tasks:
    - name: Rotate application database password
      ansible.builtin.shell: |
        NEW_PASS=$(openssl rand -base64 32 | tr -d '=+/')
        # Update local app config
        sed -i "s/^DB_PASSWORD=.*/DB_PASSWORD=${NEW_PASS}/" /etc/app/config.env
        # Update AWS Secrets Manager
        aws secretsmanager put-secret-value \
          --secret-id "{{ app_db_secret_arn }}" \
          --secret-string "{\"password\":\"${NEW_PASS}\"}" \
          --region "{{ aws_region }}"
        echo "Password rotated"
      register: rotation_result
      no_log: true   # never log credentials

    - name: Reload application service
      ansible.builtin.systemd:
        name: "{{ app_service_name }}"
        state: reloaded

    - name: Verify application health after rotation
      ansible.builtin.uri:
        url: "http://localhost:{{ app_port }}/health"
        status_code: 200
      register: health_check
      retries: 5
      delay: 3
      until: health_check.status == 200

    - name: Revoke SSH authorized keys for shared service account
      ansible.builtin.file:
        path: "/home/{{ item }}/.ssh/authorized_keys"
        state: absent
      loop: "{{ service_accounts_to_revoke | default([]) }}"

4.4 Parallel Execution and Limits

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# Run against all compromised hosts in parallel (forks=50)
ansible-playbook forensic-collect.yaml \
  -i inventory/aws_ec2.yaml \
  -l "incident_COMPROMISED" \
  --forks 50

# Limit to a specific role tag and step through one at a time
ansible-playbook rotate-credentials.yaml \
  -i inventory/aws_ec2.yaml \
  -l "role_webserver:&us-east-1" \
  --serial 1 \
  --step   # pause at each task for manual confirmation

# Emergency patch with aggressive parallelism, stop on first failure in batch
ansible-playbook emergency-patch.yaml \
  -i inventory/aws_ec2.yaml \
  -l production \
  --forks 100 \
  --serial "20%"   # roll through 20% of hosts per batch
  --max-fail-percentage 5

5. PagerDuty and OpsGenie as Automation Entry Points

5.1 PagerDuty as EventBridge Event Source

PagerDuty is a native AWS EventBridge partner event source. In your AWS account, PagerDuty appears as a Partner Event Source in the EventBridge console. You associate it with a custom event bus and then write rules against that bus.

1
2
3
4


# Associate a PagerDuty partner event source with a custom event bus
aws events create-event-bus \
  --name pagerduty-incidents \
  --event-source-name "aws.partner/pagerduty.com/P1234567/your-service-id"

PagerDuty events arrive on this bus with detail-type values like "PagerDuty Webhook v3". The detail contains incident metadata: event.data.incident.status, event.data.incident.urgency, event.data.service.name.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


{
  "source": ["aws.partner/pagerduty.com/P1234567"],
  "detail-type": ["PagerDuty Webhook v3"],
  "detail": {
    "event": {
      "event_type": ["incident.triggered"],
      "data": {
        "incident": {
          "urgency": ["high"],
          "status": ["triggered"]
        }
      }
    }
  }
}

5.2 OpsGenie Outbound Webhook to EventBridge

OpsGenie does not have a native EventBridge integration, but EventBridge API Destinations provide a generic inbound webhook endpoint. Create an API Destination pointing at a Lambda URL or API Gateway endpoint, then post from OpsGenie’s integration settings.

Alternatively: configure OpsGenie to fire a webhook to API Gateway, which puts the event on a custom EventBridge bus via PutEvents.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# API Gateway Lambda handler to bridge OpsGenie webhooks to EventBridge
import boto3
import json

events = boto3.client('events')

def lambda_handler(event, context):
    body = json.loads(event.get('body', '{}'))
    
    # OpsGenie alert payload
    alert = body.get('alert', {})
    
    events.put_events(Entries=[{
        'EventBusName': 'arn:aws:events:us-east-1:123456789012:event-bus/ops-alerts',
        'Source': 'opsgenie.alert',
        'DetailType': f"OpsGenie Alert - {alert.get('priority', 'P3')}",
        'Detail': json.dumps({
            'alertId': alert.get('alertId'),
            'message': alert.get('message'),
            'priority': alert.get('priority'),
            'tags': alert.get('tags', []),
            'entity': alert.get('entity'),
            'source': alert.get('source'),
            'action': body.get('action')  # 'Create', 'Acknowledge', 'Close'
        })
    }])
    
    return {'statusCode': 200, 'body': 'ok'}

6. Self-Healing Infrastructure Patterns

6.1 The Health-Check → Alert → Runbook → Verify Loop

True self-healing requires a feedback loop that checks whether the runbook actually fixed the problem. Without it, your automation silently succeeds at the wrong thing.

┌─────────────────────────────────────────────────────────────┐
│                    Self-Healing Loop                        │
│                                                             │
│  CloudWatch Alarm (metric threshold breach)                 │
│       │                                                     │
│       ▼                                                     │
│  EventBridge rule → SSM Automation runbook                  │
│       │                                                     │
│       ├── Step 1: Record pre-remediation metrics            │
│       ├── Step 2: Execute remediation action                │
│       ├── Step 3: aws:sleep (wait for convergence)          │
│       ├── Step 4: aws:waitForAwsResourceProperty (verify)   │
│       ├── Step 5: Branch on verification result             │
│       │       ├── SUCCESS → SNS "resolved" notification     │
│       │       │            → CloudWatch metric (remediated) │
│       │       └── FAILURE → SNS "escalate" notification     │
│       │                    → SSM OpsItem (human review)     │
│       └── All steps: CloudTrail audit trail                 │
└─────────────────────────────────────────────────────────────┘

Runbook fragment for post-remediation verification:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


  - name: verifyRemediation
    action: aws:executeAwsApi
    maxAttempts: 5
    timeoutSeconds: 300
    inputs:
      Service: cloudwatch
      Api: GetMetricStatistics
      Namespace: AWS/ApplicationELB
      MetricName: HTTPCode_Target_5XX_Count
      Dimensions:
        - Name: LoadBalancer
          Value: "{{ LoadBalancerArn }}"
      StartTime: "{{ global:DATE_TIME }}"
      EndTime: "{{ global:DATE_TIME }}"
      Period: 60
      Statistics:
        - Sum
    outputs:
      - Name: ErrorCount
        Selector: "$.Datapoints[0].Sum"
        Type: Integer

  - name: branchOnVerification
    action: aws:branch
    inputs:
      Choices:
        - NextStep: escalateToHuman
          Variable: "{{ verifyRemediation.ErrorCount }}"
          NumericGreaterOrEquals: 10
      Default: recordSuccess

  - name: recordSuccess
    action: aws:executeAwsApi
    inputs:
      Service: cloudwatch
      Api: PutMetricData
      Namespace: "IncidentResponse/Automation"
      MetricData:
        - MetricName: RemediationSucceeded
          Value: 1
          Unit: Count
          Dimensions:
            - Name: RunbookName
              Value: "{{ global:DOCUMENT_NAME }}"
    isEnd: true

  - name: escalateToHuman
    action: aws:executeAwsApi
    inputs:
      Service: ssm
      Api: CreateOpsItem
      Title: "Auto-remediation failed: {{ global:DOCUMENT_NAME }}"
      OperationalData:
        /aws/automations:
          Value: "{{ automation:EXECUTION_ID }}"
          Type: SearchableString
      Severity: "2"
      Source: "automation"
    isEnd: true

6.2 Circuit Breakers in Runbooks

A circuit breaker prevents automation from thrashing. If your runbook restarted a service three times and the service keeps dying, the fourth restart is not helpful.

Pattern: check restart count before acting

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


  - name: checkRestartCount
    action: aws:executeScript
    inputs:
      Runtime: python3.11
      Handler: check_restart_count
      InputPayload:
        instanceId: "{{ InstanceId }}"
        metricNamespace: "IncidentResponse/Automation"
      Script: |
        import boto3
        from datetime import datetime, timezone, timedelta

        def check_restart_count(events, context):
            cw = boto3.client('cloudwatch')
            response = cw.get_metric_statistics(
                Namespace=events['metricNamespace'],
                MetricName='ServiceRestart',
                Dimensions=[{'Name': 'InstanceId', 'Value': events['instanceId']}],
                StartTime=datetime.now(timezone.utc) - timedelta(hours=1),
                EndTime=datetime.now(timezone.utc),
                Period=3600,
                Statistics=['Sum']
            )
            count = int(response['Datapoints'][0]['Sum']) if response['Datapoints'] else 0
            return {
                'restartCount': count,
                'circuitOpen': count >= 3   # open after 3 restarts in 1 hour
            }
    outputs:
      - Name: CircuitOpen
        Selector: "$.Payload.circuitOpen"
        Type: Boolean
      - Name: RestartCount
        Selector: "$.Payload.restartCount"
        Type: Integer

  - name: branchOnCircuitBreaker
    action: aws:branch
    inputs:
      Choices:
        - NextStep: escalateCircuitOpen
          Variable: "{{ checkRestartCount.CircuitOpen }}"
          BooleanEquals: true
      Default: executeRestart

6.3 Idempotent Remediation

Every remediation action must be safe to run multiple times. EventBridge can deliver the same event more than once (at-least-once delivery), and concurrent GuardDuty findings can trigger multiple executions.

Patterns:

Check current state before acting: DescribeInstances before ModifyNetworkInterfaceAttribute. If the isolation SG is already applied, skip and record.
Use conditional API calls: S3 PutObject with --if-none-match or DynamoDB conditional writes to record remediation state.
Tag resources with execution ID and check for tag before acting.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


def is_already_isolated(instance_id: str, isolation_sg_id: str) -> bool:
    """Idempotency check — returns True if instance is already isolated."""
    ec2 = boto3.client('ec2')
    instance = ec2.describe_instances(
        InstanceIds=[instance_id]
    )['Reservations'][0]['Instances'][0]

    for eni in instance.get('NetworkInterfaces', []):
        current_sgs = {sg['GroupId'] for sg in eni['Groups']}
        if current_sgs == {isolation_sg_id}:
            return True
    return False

6.4 Chaos Engineering and Self-Healing Validation

Self-healing infrastructure is only credible if you regularly test it. Use AWS Fault Injection Simulator (FIS) to inject failures and verify that your runbooks fire.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


{
  "description": "Validate EC2 self-healing: terminate instance, verify ASG replaces it",
  "targets": {
    "compromisedInstances": {
      "resourceType": "aws:ec2:instance",
      "resourceTags": {"Environment": "staging", "ChaosTest": "true"},
      "selectionMode": "COUNT(1)"
    }
  },
  "actions": {
    "terminateInstance": {
      "actionId": "aws:ec2:terminate-instances",
      "targets": {"Instances": "compromisedInstances"}
    }
  },
  "stopConditions": [{
    "source": "aws:cloudwatch:alarm",
    "value": "arn:aws:cloudwatch:us-east-1:123456789012:alarm/ChaosTestAbort"
  }],
  "roleArn": "arn:aws:iam::123456789012:role/FISExecutionRole"
}

The verification checklist after a chaos test:

Did the EventBridge rule trigger within 60 seconds?
Did the runbook reach the isolateInstance step?
Did CloudWatch receive the RemediationSucceeded metric?
Is there a CloudTrail event for every API call in the runbook?
Was the SNS notification delivered?

7. Practical End-to-End: GuardDuty Unusual Outbound Traffic

Here is the complete flow for the scenario: an EC2 instance is making unusual outbound connections, GuardDuty fires a Trojan:EC2/BlackholeTraffic or UnauthorizedAccess:EC2/TorIPCaller finding.

Step 1: Create the Isolation Security Group (one-time setup)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# Create isolation SG
ISOLATION_SG=$(aws ec2 create-security-group \
  --group-name "incident-isolation-sg" \
  --description "Incident isolation: no ingress, SSM-only egress" \
  --vpc-id vpc-0abc123def456789 \
  --query 'GroupId' --output text)

# Remove default allow-all egress
aws ec2 revoke-security-group-egress \
  --group-id "$ISOLATION_SG" \
  --protocol all \
  --cidr 0.0.0.0/0

# Allow HTTPS egress only to SSM VPC endpoint SG (for Session Manager access)
SSM_ENDPOINT_SG=$(aws ec2 describe-vpc-endpoints \
  --filters "Name=service-name,Values=com.amazonaws.us-east-1.ssm" \
  --query 'VpcEndpoints[0].Groups[0].GroupId' --output text)

aws ec2 authorize-security-group-egress \
  --group-id "$ISOLATION_SG" \
  --protocol tcp \
  --port 443 \
  --source-group "$SSM_ENDPOINT_SG"

echo "Isolation SG: $ISOLATION_SG"

Step 2: Deploy the SSM Automation Runbook

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


aws ssm create-document \
  --name "EC2-IncidentResponse-Isolate-v2" \
  --document-type Automation \
  --document-format YAML \
  --content file://ec2-isolate-runbook.yaml

# Pin a specific version for production use
aws ssm update-document \
  --name "EC2-IncidentResponse-Isolate-v2" \
  --document-version "\$LATEST" \
  --document-format YAML \
  --content file://ec2-isolate-runbook.yaml

Why version pinning matters: AWS-managed runbooks (AWS-*) are updated by AWS. Custom runbooks referenced by their name default to $DEFAULT. In production, always reference the specific version number (e.g., EC2-IncidentResponse-Isolate-v2:3) in your EventBridge target, and use a change management process to promote new versions.

Step 3: Create EventBridge Rule

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61


# Create the detection rule
aws events put-rule \
  --name "guardduty-ec2-network-threat" \
  --event-bus-name default \
  --event-pattern '{
    "source": ["aws.guardduty"],
    "detail-type": ["GuardDuty Finding"],
    "detail": {
      "severity": [{"numeric": [">=", 7]}],
      "resource": {
        "resourceType": ["Instance"]
      },
      "type": [
        {"prefix": "Trojan:EC2"},
        {"prefix": "UnauthorizedAccess:EC2"},
        {"prefix": "Backdoor:EC2"},
        {"prefix": "CryptoCurrency:EC2"}
      ]
    }
  }' \
  --state ENABLED

# Add SSM Automation target with input transformer
aws events put-targets \
  --rule "guardduty-ec2-network-threat" \
  --targets '[
    {
      "Id": "IsolationRunbook",
      "Arn": "arn:aws:ssm:us-east-1:123456789012:automation-definition/EC2-IncidentResponse-Isolate-v2:3",
      "RoleArn": "arn:aws:iam::123456789012:role/EventBridgeSSMRole",
      "InputTransformer": {
        "InputPathsMap": {
          "instance": "$.detail.resource.instanceDetails.instanceId",
          "finding": "$.detail.id",
          "type": "$.detail.type"
        },
        "InputTemplate": "{\"InstanceId\":[\"<instance>\"],\"FindingId\":[\"<finding>\"],\"FindingType\":[\"<type>\"],\"IsolationSgId\":[\"sg-0isolation123\"],\"SnsTopicArn\":[\"arn:aws:sns:us-east-1:123456789012:incident-alerts\"],\"AutomationAssumeRole\":[\"arn:aws:iam::123456789012:role/SSMAutomationRole\"]}"
      },
      "DeadLetterConfig": {
        "Arn": "arn:aws:sqs:us-east-1:123456789012:incident-response-dlq"
      },
      "RetryPolicy": {
        "MaximumRetryAttempts": 3,
        "MaximumEventAgeInSeconds": 900
      }
    },
    {
      "Id": "ImmediateAlert",
      "Arn": "arn:aws:sns:us-east-1:123456789012:incident-alerts-immediate",
      "InputTransformer": {
        "InputPathsMap": {
          "finding": "$.detail.id",
          "type": "$.detail.type",
          "severity": "$.detail.severity",
          "instance": "$.detail.resource.instanceDetails.instanceId",
          "region": "$.region"
        },
        "InputTemplate": "\"GuardDuty finding <type> (severity <severity>) on instance <instance> in <region>. Finding ID: <finding>. Automated isolation initiated.\""
      }
    }
  ]'

Step 4: Verify the Audit Trail

After an incident fires, verify coverage in CloudTrail:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Find all API calls made by the automation role during the incident
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=SSMAutomationRole \
  --start-time "2026-05-23T10:00:00Z" \
  --end-time "2026-05-23T11:00:00Z" \
  --query 'Events[*].{Time:EventTime,Event:EventName,Resource:Resources[0].ResourceName}' \
  --output table

# Find the automation execution in SSM
aws ssm describe-automation-executions \
  --filters "Key=ExecutionStatus,Values=Success,Failed" \
  --max-items 10 \
  --query 'AutomationExecutionMetadataList[*].{ID:AutomationExecutionId,Status:AutomationExecutionStatus,Start:ExecutionStartTime}'

8. Pitfalls and Production Hardening

8.1 Automation That Makes Things Worse

The most dangerous failure mode is a runbook that is correct for one scenario but runs destructively in another. Common examples:

A “restart unhealthy instances” runbook that fires when the health check endpoint itself is broken — restarting every instance simultaneously causes a full outage.
A “revoke open security group rules” runbook that removes a rule that was intentionally opened for a maintenance window.
A “terminate and replace” runbook on instances that are not in an ASG — the instance is gone, the service is down, and there is nothing to replace it.

Mitigations:

Scope runbooks to instances with specific tags (Environment=production AND AutoRemediation=enabled).
Add an aws:executeScript step at the start that checks preconditions — is the ASG healthy? Is there a maintenance window tag? Is there a minimum healthy threshold?
Never target * in resource selectors.

8.2 Blast Radius Controls

1
2
3
4
5
6
7


# EventBridge rule with rate-limiting via SQS as intermediate target
# Rather than invoking Lambda directly, route through SQS with a concurrency-limited consumer
# This provides a natural rate limit on remediation rate
- name: SQSBuffer
  Arn: "arn:aws:sqs:us-east-1:123456789012:remediation-queue"
  # Lambda consumer of this queue: ReservedConcurrentExecutions: 5
  # Ensures max 5 simultaneous remediations

Use Lambda reserved concurrency as a blast radius control. Set ReservedConcurrentExecutions on your remediation Lambda to a value that limits how many instances can be simultaneously acted on.

For SSM Automation, the MaxConcurrency parameter on StartAutomationExecution (for multi-target executions) and the rate controls on aws:runCommand steps limit parallel execution:

1
2
3
4
5
6
7
8
9


  - name: patchInstances
    action: aws:runCommand
    inputs:
      DocumentName: AWS-RunPatchBaseline
      Targets:
        - Key: tag:Environment
          Values: [production]
      MaxConcurrency: "10%"   # max 10% of targets simultaneously
      MaxErrors: "5%"         # stop if >5% error rate

8.3 Approval Gates for Destructive Actions

Any runbook that terminates instances, modifies IAM policies broadly, or modifies network infrastructure (route tables, VPC peering) should require manual approval in production. The aws:approve action with a tight timeout (1 hour for urgent incidents) forces human review while keeping total response time reasonable.

The anti-pattern is requiring approval for every step — this defeats the purpose of automation. Reserve approval gates for:

Instance termination (as opposed to stop/snapshot)
IAM policy changes that affect more than one resource
Any action in the management VPC or on infrastructure shared across services
Runbook step counts above a blast-radius threshold (e.g., AffectedInstanceCount > 5)

8.4 Runbook Version Pinning

Referencing $DEFAULT in EventBridge targets means a runbook update automatically applies to all future incident executions. This is dangerous: a bug in a new runbook version could cause mass remediation failures during an incident.

Practice: Reference specific version numbers in EventBridge targets. Maintain a staging environment where new versions are tested with FIS chaos injection before promotion. Use a separate SSM document per environment (EC2-Isolate-staging-v3, EC2-Isolate-prod-v7).

8.5 Eventual Consistency and IAM Propagation

IAM policy changes take up to several seconds to propagate across all AWS regions and services. After attaching a deny policy to a compromised role, add an aws:sleep for 30 seconds before making any API calls that verify the isolation. Verifying immediately after PutRolePolicy will show success from the same region’s IAM endpoint, but the compromised credentials may still work in other regions.

8.6 CloudTrail Coverage for the Automation Itself

Every API call made by the SSM automation role, the Lambda execution role, and the EventBridge execution role appears in CloudTrail under the respective principal. This means every automated remediation action has a complete, tamper-evident audit trail — provided CloudTrail is enabled with log file validation and S3 Object Lock.

Tag your automation execution roles clearly (incident-response:true) so CloudTrail queries can filter to all automated incident actions in a given time window.

Observability for the Automation

Emit these CloudWatch metrics from every remediation path:

Metric	Namespace	Dimensions	Alarm
`RemediationTriggered`	`IncidentResponse/Automation`	RunbookName, FindingType	No
`RemediationSucceeded`	`IncidentResponse/Automation`	RunbookName, FindingType	No
`RemediationFailed`	`IncidentResponse/Automation`	RunbookName, FindingType	Yes — page on any failure
`RemediationDurationSeconds`	`IncidentResponse/Automation`	RunbookName	Yes — alarm if p99 > 5 min
`CircuitBreakerOpen`	`IncidentResponse/Automation`	InstanceId, RunbookName	Yes — human escalation
`DLQMessagesVisible`	`AWS/SQS`	QueueName (incident-response-dlq)	Yes — events being dropped

Use SSM OpsCenter to surface runbooks that terminated in Failed or TimedOut state. Connect OpsCenter to your ticketing system so failed automation executions automatically create incidents for human review.

Sources: