Remediating misconfiguration

Lesson 3 of 3 in Troubleshooting Detection.

Finding the misconfiguration is the easy half. The exam’s real question is where the fix belongs, and there are four honest answers — arranged, as always, from latest to earliest:

Fix it by hand. Appropriate exactly once, while you work out which of the three below it should be. A fix applied by a human and recorded nowhere will be undone by the next deployment, and nobody will know.

Fix it reactively. A Config rule goes non-compliant and its remediation action runs an SSM Automation runbook; or an EventBridge rule catches the finding and invokes a Lambda or a Step Functions runbook. Latency is seconds to minutes, and the window between the misconfiguration and the fix is real exposure.

Fix it convergently. A State Manager association re-applies the desired state on a schedule whether or not anything changed. This catches the drift nothing emits an event for — a configuration file edited inside an instance, an agent removed by a package update — and it bounds the lifetime of drift rather than reacting to it.

Make it impossible. A service control policy, a resource control policy, a permission boundary, a Firewall Manager policy, or a pipeline check that fails the deployment. The exposure window is zero because the misconfiguration never exists. This is Domain 6 territory, and when a question emphasises that a particular state must never occur rather than must be corrected, it is asking for prevention and every remediation option is a distractor.

Mature estates run all four, with the ladder as a design preference: prevent what you can, converge on what you cannot, react to the rest, and keep the manual path for genuine novelty.

Work one through end to end, because the pieces only make sense assembled.

The requirement. Every S3 bucket in the organization must have default encryption enabled. It must be corrected automatically within minutes, the correction must be visible, and a bucket in the regulated OU must never reach the unencrypted state at all.

The detection. The managed Config rule s3-bucket-server-side-encryption-enabled, shipped inside an organization conformance pack so every current and future account gets it from one deployment. The pack needs the configuration recorder on in every account and Region — enforced through Control Tower or a StackSet — because the rule evaluates configuration items and nothing else.

The correction. The rule’s remediation action runs the AWS-EnableS3BucketEncryption Automation document, executed automatically, with the bucket name passed from the non-compliant resource and a retry count for transient failures. The automation assumes a role scoped to s3:PutEncryptionConfiguration on buckets in that account — not s3:*, and not every resource.

The proof. The fix is itself a configuration change, so a fresh configuration item flows back, the rule re-evaluates, and the resource flips to compliant with no human in the loop. Meanwhile the remediation’s own outcome is what you alarm on: an EventBridge rule on remediation failures, because a remediation that silently stops working leaves you with a dashboard that says compliant and a reality that is not.

The prevention. For the regulated OU, none of the above is sufficient, because the requirement said never. That is an SCP denying s3:CreateBucket without the encryption configuration, or a pipeline check that fails the template — and the Config rule stays in place as evidence that the prevention is holding.

# AWS Config rule + automatic remediation (abridged CloudFormation)
EncryptionRule:
  Type: AWS::Config::ConfigRule
  Properties:
    ConfigRuleName: s3-bucket-server-side-encryption-enabled
    Source: { Owner: AWS, SourceIdentifier: S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED }
    Scope: { ComplianceResourceTypes: [ "AWS::S3::Bucket" ] }

EncryptionRemediation:
  Type: AWS::Config::RemediationConfiguration
  Properties:
    ConfigRuleName: !Ref EncryptionRule
    TargetType: SSM_DOCUMENT
    TargetId: AWS-EnableS3BucketEncryption   # AWS-owned Automation runbook
    Automatic: true                           # false ⇒ an operator must execute it
    MaximumAutomaticAttempts: 3               # transient failures only
    RetryAttemptSeconds: 60
    Parameters:
      AutomationAssumeRole:
        StaticValue: { Values: [ "arn:aws:iam::111122223333:role/ConfigRemediationS3" ] }
      BucketName:
        ResourceValue: { Value: RESOURCE_ID } # the non-compliant bucket
      SSEAlgorithm:
        StaticValue: { Values: [ "AES256" ] }

Four properties separate a remediation you can leave running from one that becomes its own incident.

Least privilege on the automation role. This is the one that matters most, and it is easy to get catastrophically wrong. A remediation role holding s3:* or iam:* so that “the runbooks all work” is a privilege-escalation path wearing a housekeeping costume: anyone who can trigger the automation, or alter the document it runs, inherits that power. One narrow role per remediation, scoped to the resource type and the single action.

Idempotence. The fix will run more than once — a retry, a duplicate event, a second rule covering the same resource. Enabling encryption that is already enabled must be a no-op, not an error and certainly not a second bucket.

Bounded blast radius. Cap retries, and scope the rule to the resource types and accounts you intend. A remediation that fires across an entire organization on a rule you misparameterised is an outage you built yourself, at machine speed.

Observability of the fix. Alarm on remediation failures and on the remediation rate. A sudden spike means something upstream is creating non-compliant resources faster than you are fixing them — which is the signal that the control belongs one rung up the ladder, in prevention.

The final judgement call the exam likes: remediation is the right answer when the misconfiguration is recoverable and the fix is mechanical. When the misconfiguration is “this instance has been compromised”, the answer is not remediation but containment and forensics — snapshot, isolate, preserve — and that is Domain 2’s subject, not this one.

Key terms: AWS Config rule, Systems Manager Automation document (runbook), Systems Manager State Manager, Conformance pack, Amazon EventBridge

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.