Skip to content

Commit 5847e71

Browse files
NithinChandranR-AWSbfreiberg
NithinChandranR-AWS
and
bfreiberg
authored
cloudwatch-log-alarm-lambda-remediation-cdk: Log-based alarm with auto-remediation (#3228)
* feat(cloudwatch-log-alarm-lambda-remediation-cdk): Log Alarm auto-remediation Deploy self-healing architecture using AWS::CloudWatch::LogAlarm (new CFN resource, July 2026). Logs Insights query counts ERROR messages every 5 minutes, triggers alarm when threshold breached, Amazon SNS fans out to AWS Lambda which runs remediation via AWS Systems Manager RunCommand on tagged Amazon EC2 instances. Eliminates the 3-resource MetricFilter workaround — single LogAlarm resource replaces MetricFilter + Metric + Standard Alarm. 5 services composed: Amazon CloudWatch Logs, Log Alarm, Amazon SNS, AWS Lambda, AWS Systems Manager. * fix: apply bfreiberg review suggestions (README, icons, non-pinned CDK) * Add final pattern file * Update example-pattern.json --------- Co-authored-by: Ben <9841563+bfreiberg@users.noreply.github.com>
1 parent f8d9d38 commit 5847e71

10 files changed

Lines changed: 638 additions & 0 deletions

File tree

‎cloudwatch-log-alarm-lambda-remediation-cdk/README.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# Amazon CloudWatch Log Alarm with AWS Lambda Auto-Remediation
2+
3+
This pattern deploys a self-healing architecture using the Amazon CloudWatch Log Alarm resource (`AWS::CloudWatch::LogAlarm`) to monitor application logs with CloudWatch Logs Insights queries and automatically trigger remediation through AWS Lambda and AWS Systems Manager when error thresholds are breached.
4+
5+
Learn more about this pattern at Serverless Land Patterns: https://serverlessland.com/patterns/cloudwatch-log-alarm-lambda-remediation-cdk
6+
7+
## Architecture
8+
9+
```
10+
┌─────────────────────┐ ┌──────────────────────────┐ ┌─────────────┐ ┌──────────────┐ ┌──────────────────┐
11+
│ Amazon CloudWatch │────▶│ Amazon CloudWatch │────▶│ Amazon SNS │────▶│ AWS Lambda │────▶│ AWS Systems │
12+
│ Logs │ │ Log Alarm │ │ │ │ (Remediation)│ │ Manager │
13+
│ (Application Logs) │ │ (Logs Insights Query) │ │ │ │ │ │ (RunCommand) │
14+
└─────────────────────┘ └──────────────────────────┘ └─────────────┘ └──────────────┘ └──────────────────┘
15+
```
16+
17+
**How it works:**
18+
19+
1. Application logs are written to Amazon CloudWatch Logs
20+
2. The Amazon CloudWatch Log Alarm runs a CloudWatch Logs Insights query every 5 minutes to count ERROR messages
21+
3. When error count ≥ 5 in a single evaluation window, the alarm transitions to ALARM state
22+
4. The alarm publishes to Amazon SNS, which invokes the AWS Lambda remediation function
23+
5. AWS Lambda sends a command via AWS Systems Manager to restart the application service on tagged Amazon EC2 instances
24+
25+
The `AWS::CloudWatch::LogAlarm` resource eliminates the need for metric filters. Previously, monitoring log content required creating a CloudWatch Metric Filter, waiting for metric data points, then creating a standard alarm on that metric. Log Alarms run Logs Insights queries directly on schedule and evaluate results against thresholds.
26+
27+
## Requirements
28+
29+
- [AWS CDK v2](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html) installed and configured
30+
- [Node.js 20+](https://nodejs.org/) with npm
31+
- AWS account [bootstrapped for CDK](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html)
32+
- Python 3.12 (for AWS Lambda functions)
33+
- (Optional) Amazon EC2 instances tagged with `AutoRemediate=true` for SSM remediation
34+
35+
## Deployment
36+
37+
### Step 1: Install dependencies and synthesize
38+
39+
```bash
40+
cd cloudwatch-log-alarm-lambda-remediation-cdk/cdk
41+
npm install
42+
npx cdk synth
43+
```
44+
45+
### Step 2: Deploy the stack
46+
47+
```bash
48+
npx cdk deploy
49+
```
50+
51+
## Testing
52+
53+
### 1. Generate error logs to trigger the alarm
54+
55+
```bash
56+
LOG_GROUP="/app/monitored-service"
57+
58+
# Write ERROR messages to trigger the alarm (threshold = 5)
59+
for i in $(seq 1 6); do
60+
aws logs put-log-events \
61+
--log-group-name "$LOG_GROUP" \
62+
--log-stream-name "test-stream" \
63+
--log-events "[{\"timestamp\":$(date +%s000),\"message\":\"ERROR: Connection timeout to database (attempt $i)\"}]" \
64+
--sequence-token "$(aws logs describe-log-streams --log-group-name $LOG_GROUP --log-stream-name-prefix test --query 'logStreams[0].uploadSequenceToken' --output text 2>/dev/null)"
65+
sleep 1
66+
done
67+
68+
echo "Wrote 6 ERROR messages. Alarm will evaluate in ~5 minutes."
69+
```
70+
71+
### 2. Check alarm state (after 5 minutes)
72+
73+
```bash
74+
aws cloudwatch describe-alarms \
75+
--alarm-names "app-error-rate-alarm" \
76+
--alarm-types "LogAlarm" \
77+
--query 'LogAlarms[0].{State:StateValue,Reason:StateReason}'
78+
```
79+
80+
### 3. Verify the AWS Lambda function was invoked
81+
82+
```bash
83+
aws logs filter-log-events \
84+
--log-group-name "/aws/lambda/CloudwatchLogAlarmLambdaRem-RemediationFn*" \
85+
--filter-pattern "Executing remediation" \
86+
--query 'events[].message'
87+
```
88+
89+
### 4. Check AWS Systems Manager command history
90+
91+
```bash
92+
aws ssm list-commands \
93+
--filters "key=DocumentName,value=AWS-RunShellScript" \
94+
--max-results 5 \
95+
--query 'Commands[].{Id:CommandId,Status:Status,Comment:Comment}'
96+
```
97+
98+
## Cleanup
99+
100+
> **Warning:** This will delete the log group and all log data. The Amazon CloudWatch Log Alarm and all associated resources will be removed.
101+
102+
```bash
103+
cd cloudwatch-log-alarm-lambda-remediation-cdk/cdk
104+
npx cdk destroy
105+
```
106+
107+
## Services Used
108+
109+
| Service | Role |
110+
|---------|------|
111+
| Amazon CloudWatch Logs | Application log storage and query target |
112+
| Amazon CloudWatch Log Alarm | Scheduled Logs Insights query with threshold evaluation |
113+
| Amazon SNS | Alarm notification delivery to subscribers |
114+
| AWS Lambda | Auto-remediation logic (classify alarm, invoke SSM) |
115+
| AWS Systems Manager | Execute commands on tagged Amazon EC2 instances |
116+
117+
----
118+
Copyright 2026 Amazon.com, Inc. or its affiliates. All Rights Reserved.
119+
SPDX-License-Identifier: MIT-0

‎cloudwatch-log-alarm-lambda-remediation-cdk/cdk/.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
node_modules
2+
cdk.out
3+
cdk.context.json
4+
build
5+
*.js
6+
*.d.ts

‎cloudwatch-log-alarm-lambda-remediation-cdk/cdk/bin/app.ts‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
#!/usr/bin/env node
2+
// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
3+
// SPDX-License-Identifier: MIT-0 (2026)
4+
5+
import 'source-map-support/register';
6+
import * as cdk from 'aws-cdk-lib';
7+
import { CloudwatchLogAlarmLambdaRemediationStack } from '../lib/cloudwatch-log-alarm-lambda-remediation-stack';
8+
9+
const app = new cdk.App();
10+
new CloudwatchLogAlarmLambdaRemediationStack(app, 'CloudwatchLogAlarmLambdaRemediationStack', {
11+
description: 'Amazon CloudWatch Log Alarm with AWS Lambda auto-remediation (uksb-1tupboc57)',
12+
});

‎cloudwatch-log-alarm-lambda-remediation-cdk/cdk/cdk.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"app": "npx ts-node --prefer-ts-exts bin/app.ts"
3+
}

‎cloudwatch-log-alarm-lambda-remediation-cdk/cdk/lib/cloudwatch-log-alarm-lambda-remediation-stack.ts‎

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
// SPDX-License-Identifier: MIT-0 (2026)
3+
4+
import * as cdk from 'aws-cdk-lib';
5+
import * as lambda from 'aws-cdk-lib/aws-lambda';
6+
import * as iam from 'aws-cdk-lib/aws-iam';
7+
import * as sns from 'aws-cdk-lib/aws-sns';
8+
import * as snsSubscriptions from 'aws-cdk-lib/aws-sns-subscriptions';
9+
import * as logs from 'aws-cdk-lib/aws-logs';
10+
import { Construct } from 'constructs';
11+
import * as path from 'path';
12+
13+
export class CloudwatchLogAlarmLambdaRemediationStack extends cdk.Stack {
14+
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
15+
super(scope, id, props);
16+
17+
const region = cdk.Stack.of(this).region;
18+
const account = cdk.Stack.of(this).account;
19+
20+
// =========================================================
21+
// 1. Amazon CloudWatch Log Group (monitored application logs)
22+
// =========================================================
23+
const appLogGroup = new logs.LogGroup(this, 'AppLogGroup', {
24+
logGroupName: '/app/monitored-service',
25+
retention: logs.RetentionDays.ONE_WEEK,
26+
removalPolicy: cdk.RemovalPolicy.DESTROY,
27+
});
28+
29+
// =========================================================
30+
// 2. Amazon SNS Topic (alarm notification fan-out)
31+
// =========================================================
32+
const alarmTopic = new sns.Topic(this, 'AlarmTopic', {
33+
topicName: 'log-alarm-notifications',
34+
displayName: 'Amazon CloudWatch Log Alarm Notifications',
35+
});
36+
37+
// =========================================================
38+
// 3. AWS Lambda: Remediation Handler
39+
// =========================================================
40+
const remediationFn = new lambda.Function(this, 'RemediationFn', {
41+
runtime: lambda.Runtime.PYTHON_3_12,
42+
handler: 'handler.lambda_handler',
43+
code: lambda.Code.fromAsset(path.join(__dirname, '../../lambdas/remediation')),
44+
timeout: cdk.Duration.seconds(60),
45+
memorySize: 256,
46+
description: 'Auto-remediation: restarts service via AWS Systems Manager when error threshold breached',
47+
environment: {
48+
ALARM_TOPIC_ARN: alarmTopic.topicArn,
49+
},
50+
});
51+
52+
// SSM permissions for remediation (send commands to EC2 instances)
53+
remediationFn.addToRolePolicy(new iam.PolicyStatement({
54+
effect: iam.Effect.ALLOW,
55+
actions: [
56+
'ssm:SendCommand',
57+
'ssm:GetCommandInvocation',
58+
],
59+
resources: [
60+
`arn:aws:ssm:${region}::document/AWS-RunShellScript`,
61+
`arn:aws:ec2:${region}:${account}:instance/*`,
62+
],
63+
conditions: {
64+
StringEquals: {
65+
'aws:ResourceTag/AutoRemediate': 'true',
66+
},
67+
},
68+
}));
69+
70+
// CloudWatch Logs permissions for context retrieval
71+
remediationFn.addToRolePolicy(new iam.PolicyStatement({
72+
effect: iam.Effect.ALLOW,
73+
actions: [
74+
'logs:GetLogEvents',
75+
'logs:FilterLogEvents',
76+
],
77+
resources: [appLogGroup.logGroupArn, `${appLogGroup.logGroupArn}:*`],
78+
}));
79+
80+
// Subscribe Lambda to SNS topic
81+
alarmTopic.addSubscription(
82+
new snsSubscriptions.LambdaSubscription(remediationFn)
83+
);
84+
85+
// =========================================================
86+
// 4. IAM Role for Scheduled Query execution
87+
// =========================================================
88+
const scheduledQueryRole = new iam.Role(this, 'ScheduledQueryRole', {
89+
assumedBy: new iam.ServicePrincipal('logs.amazonaws.com'),
90+
description: 'Allows Amazon CloudWatch Logs to execute scheduled queries',
91+
inlinePolicies: {
92+
LogsQuery: new iam.PolicyDocument({
93+
statements: [
94+
new iam.PolicyStatement({
95+
effect: iam.Effect.ALLOW,
96+
actions: [
97+
'logs:StartQuery',
98+
'logs:GetQueryResults',
99+
'logs:StopQuery',
100+
],
101+
resources: [appLogGroup.logGroupArn, `${appLogGroup.logGroupArn}:*`],
102+
}),
103+
],
104+
}),
105+
},
106+
});
107+
108+
// =========================================================
109+
// 5. AWS::CloudWatch::LogAlarm (new CFN resource type)
110+
// Monitors error rate in application logs using Logs Insights
111+
// =========================================================
112+
const logAlarm = new cdk.CfnResource(this, 'ErrorRateLogAlarm', {
113+
type: 'AWS::CloudWatch::LogAlarm',
114+
properties: {
115+
AlarmName: 'app-error-rate-alarm',
116+
AlarmDescription: 'Triggers when error count exceeds threshold in application logs (monitored via Amazon CloudWatch Logs Insights query)',
117+
ComparisonOperator: 'GreaterThanOrEqualToThreshold',
118+
Threshold: 5,
119+
QueryResultsToAlarm: 1,
120+
QueryResultsToEvaluate: 1,
121+
TreatMissingData: 'notBreaching',
122+
ActionsEnabled: true,
123+
AlarmActions: [alarmTopic.topicArn],
124+
OKActions: [alarmTopic.topicArn],
125+
ScheduledQueryConfiguration: {
126+
QueryString: 'fields @timestamp, @message | filter @message like /ERROR/ | stats count(*) as error_count by bin(5m)',
127+
LogGroupIdentifiers: [appLogGroup.logGroupName],
128+
AggregationExpression: 'count(*)',
129+
ScheduleConfiguration: {
130+
ScheduleExpression: 'rate(5 minutes)',
131+
StartTimeOffset: 300,
132+
},
133+
ScheduledQueryRoleARN: scheduledQueryRole.roleArn,
134+
},
135+
},
136+
});
137+
138+
// Ensure role is created before the alarm
139+
logAlarm.addDependency(scheduledQueryRole.node.defaultChild as cdk.CfnResource);
140+
141+
// =========================================================
142+
// Outputs
143+
// =========================================================
144+
new cdk.CfnOutput(this, 'LogGroupName', {
145+
value: appLogGroup.logGroupName,
146+
description: 'Amazon CloudWatch Logs group being monitored',
147+
});
148+
149+
new cdk.CfnOutput(this, 'AlarmTopicArn', {
150+
value: alarmTopic.topicArn,
151+
description: 'Amazon SNS topic for alarm notifications',
152+
});
153+
154+
new cdk.CfnOutput(this, 'RemediationFunctionArn', {
155+
value: remediationFn.functionArn,
156+
description: 'AWS Lambda remediation function ARN',
157+
});
158+
159+
new cdk.CfnOutput(this, 'LogAlarmName', {
160+
value: 'app-error-rate-alarm',
161+
description: 'Amazon CloudWatch Log Alarm name',
162+
});
163+
}
164+
}

‎cloudwatch-log-alarm-lambda-remediation-cdk/cdk/package.json‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
{
2+
"name": "cloudwatch-log-alarm-lambda-remediation-cdk",
3+
"version": "1.0.0",
4+
"bin": { "app": "bin/app.ts" },
5+
"scripts": {
6+
"build": "tsc",
7+
"synth": "cdk synth",
8+
"deploy": "cdk deploy",
9+
"destroy": "cdk destroy"
10+
},
11+
"dependencies": {
12+
"aws-cdk-lib": "^2.185.0",
13+
"constructs": "^10.0.0",
14+
"source-map-support": "^0.5.21"
15+
},
16+
"devDependencies": {
17+
"typescript": "~5.4.0",
18+
"ts-node": "^10.9.0"
19+
}
20+
}

‎cloudwatch-log-alarm-lambda-remediation-cdk/cdk/tsconfig.json‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"compilerOptions": {
3+
"target": "ES2020",
4+
"module": "commonjs",
5+
"lib": ["es2020"],
6+
"declaration": true,
7+
"strict": true,
8+
"noImplicitAny": true,
9+
"strictNullChecks": true,
10+
"noImplicitThis": true,
11+
"alwaysStrict": true,
12+
"noUnusedLocals": false,
13+
"noUnusedParameters": false,
14+
"noImplicitReturns": true,
15+
"noFallthroughCasesInSwitch": false,
16+
"inlineSourceMap": true,
17+
"inlineSources": true,
18+
"experimentalDecorators": true,
19+
"strictPropertyInitialization": false,
20+
"outDir": "./build",
21+
"rootDir": "."
22+
},
23+
"exclude": ["node_modules", "build"]
24+
}

0 commit comments

Comments
 (0)