We all know it, we have created several CloudWatch alarms to monitor our application. If an alarm sets off we get a wall of text and don’t really know what to do. To get the problem solved asap the first contact with the alarm has to be clear about what the problems and actions are. For that I always recommend getting rid of the default message and use a HTML template for giving the specific information the person who receives the alarm needs. The main benefit here is really to use a HTML template so that you can be flexible with which information you want to give. Typical use cases are links to some portals and monitoring systems. The final alarm we receive looks like that:

The full code can be seen here.

---

The Architecture

The architecture loos like the following:

• CloudWatch Alarm: The alarm which checks on some metric
• SNS Topic: The topic gets notified when the alarm changes its state
• Lambda: The lambda is subscribed by the SNS topic
• SES: SES will be called within the lambda for sending out messages.

The additional lambda function for getting some metrics is not in here.

---

Deployment

If you just want to get started quickly follow these steps:

1. Clone the git repository: git clone git@github.com:AlessandroVol23/cloudwatch-custom-email-cdk.git

2. Define your source and destination email address:

   export DESTINATION_EMAIL=destination@gmail.com
   export SOURCE_EMAIL=src@gmail.com
   

3. Run cdk bootstrap to prepare your bucket for the lambda upload

4. Run cdk deploy for deploying the infrastructure.

That’s it. Test your alarm either by invoking your lambda function 5 times or with the AWS CLI aws cloudwatch set-alarm-state --alarm-name ALARM_NAME --state-value ALARM --state-reason "Testing" . The alarm name is in the output after the deployment.

Implementation in more detail

I’ve created the infrastructure we need in CDK here. Let’s go through the code and I’ll explain what happens.

Lambda for getting metrics

const fn = new lambda.Function(
  this,
  "lambda_cloudwatch_customize_notification",
  {
    runtime: lambda.Runtime.PYTHON_3_8,
    handler: "index.handler",
    code: new lambda.InlineCode(
      "def handler(event, context):\n\tprint(event)\n\treturn {'statusCode': 200, 'body': 'Hello, World'}"
    ),
  }
);

First of all, we create one lambda function for getting any metrics intro CloudWatch. I absolutely don’t recommend having the code inline but for the example it is fine.

Lambda for customising the alarm

const lambdaCustomizeMail = new lambda.Function(
  this,
  "customize_cloudwatch_alarm",
  {
    runtime: lambda.Runtime.PYTHON_3_8,
    handler: "index.handler",
    code: lambda.Code.fromAsset(`${path.resolve(__dirname)}/lambda`),
    environment: {
      SOURCE_EMAIL: process.env.SOURCE_EMAIL!,
      DESTINATION_EMAIL: process.env.DESTINATION_EMAIL!,
    },
  }
);

lambdaCustomizeMail.addToRolePolicy(
  new iam.PolicyStatement({
    actions: ["ses:SendEmail"],
    resources: ["*"],
    effect: iam.Effect.ALLOW,
  })
);

In the second part, we create the lambda function which gets the SNS alarm as an input, customises the message, and sends it out via SES. The code is in a subdirectory. Important here to note are the environment variables SOURCE_EMAIL and DESTINATION_EMAIL . These have to be set locally to add the required emails. Finally, a policy has to be added to allow lambda to execute SES. The best practice would be to allow just the resources you need 😉

Lambda code

def handler(event, context):
    print("Event: ", event)
    records = event.get('Records')
    first_record = records[0]

    sns_message = first_record.get('Sns')

    html = fill_html_file(sns_message)

    send_email(html)

It basically gets the first records, parses some attributes, fills the HTML template, and sends the e-mail. The full lambda is in the git repository.

SNS Subscription

const snsTopic = new sns.Topic(this, "cloudwatchAlarm");

snsTopic.addSubscription(
  new snsSubscription.LambdaSubscription(lambdaCustomizeMail)
);

This code adds an SNS topic and the subscription lambda function.

Alarm

const alarm = new cloudwatch.Alarm(this, "testCustomizeAlarm", {
  metric: fn.metric("Invocations"),
  threshold: 5,
  evaluationPeriods: 1,
});

alarm.addAlarmAction({
  bind(): AlarmActionConfig {
    return { alarmActionArn: snsTopic.topicArn };
  },
});

new cdk.CfnOutput(this, "AlarmToTrigger", { value: alarm.alarmName });

Finally, we just need our CloudWatch alarm, as a metric we use the invocations of the first lambda with some threshold. We bind the SNS topic to the alarm action and output the name of the alarm.

The code is fairly easy and that is what I really like about CDK.

---

Considerations

When executing this approach you have to think about a couple of considerations.

SES vs. SNS

First of all, when using SES instead of SNS you have to verify your e-mail addresses or domains. You need a destination address for SES. SNS on the other hand takes care of that for you and also allows subscribing and unsubscribing from emails. The only reason here for SES is that SNS does not support HTML emails.

Lambda failures

You have to be aware of any failures that happen. In the best case you design your lambda in a way that you also get a notification in case something goes wrong, for example if a field in the alarm description is missing. Because if your lambda fails you won’t get notified about any alarms.

No SES in China!

Something I learned the hard way. There is no SES service in AWS China! Take this into consideration when designing your alarms.

---

That’s it. I hope I could give you some insight and help you with getting nice-looking alarm messages. For more content on AWS, Serverless and building SaaS software follow my Twitter 🙂