Let’s Build Cron Jobs on Steroids ⏱💉

Let’s Build Cron Jobs on Steroids ⏱💉

Next feature alert 🚨

The main feature of ServerlessQ is a hosted message queue. But a really important feature is the ability to create Cron Jobs.

This feature enables Vercel users to use Vercel as an almost full-fleshed platform. Remember the goal is to not have the overhead of AWS.

But I didn't want to build simple Cron Jobs. I want to give the benefit of both worlds. That is why I'll build Cron Jobs with the power of message queues.

Speaking in AWS services I will combine EventBridge, SQS, Lambda, AppSync, and DynamoDB.

Let's dive in and see some details about the implementation.

What are Cron Jobs?

Cron Jobs are scheduled events that run in a pre-defined time interval. For example, if you want to run a task to clean up your database every month you can simply run a Cron Job and let it execute a simple script for that.

Since we are living in a serverless world we need a service for that. AWS built EventBridge (formerly known as CloudWatch Events) for exactly that. You can simply define a Cron Job and plug it into another service, for example, Lambda.

With ServerlessQ I want to enable users to use the great AWS Services without the need for an AWS Account.

Cron Expression

A cron expression has its own syntax. It is kind of similar for all providers but can have slight differences. For AWS it looks like that.

AWS EventBridge Cron Expression

There are 6 groups where you can define values. The page also shows some common examples.

Let's take this expression for example: 0/10 * ? * MON-FRI *

You can read the cron like that:

  1. Every ten minutes
  2. Each hour
  3. Any day of the month
  4. Any month
  5. Only Monday to Friday
  6. Any year

This task runs indefinitely till the end of time, each Monday to Friday, every ten minutes.

The Architecture.

The Whiteboard Architecture Sketch

Like in my last post let's start with a simple whiteboard architecture sketch.

Whiteboard Sketch of Cron Jobs

There are kind of three different architectures involved here:

  • General idea of the services (the orange one at the bottom)
  • Creating a new Cron Job (the left one)
  • Executing the Cron Job (the right one)

Let's see them in a bit more detail.

Creating a new Cron Job with AppSync, EventBridge, Lambda, SQS

Create a queue with EventBridge, SQS, Lambda, and AppSync

My main API is an AppSync GraphQL API. I'll create a new mutation and define an input:


input CreateCronJobInput {
  name: String!
  cronExpression: String!
  retries: Int
  target: String!
  method: Method!
}

type Mutation @aws_oidc @aws_api_key {
  createCronJob(input: CreateCronJobInput!): Cron!
}

A Cron Job has the following attributes:

  • Name: Simply a name for displaying it nicely
  • A cron expression, we'll pass this from the frontend
  • Number of retries if a call fails
  • Target URL
  • Target Method

The mutation will be called from the frontend (from a simple form) and the mutation forwards the call via a pipeline resolver to Lambda.

The lambda is executing the following actions:

  1. Create a new Cron Job in EventBridge
  2. Create a queue for the Cron Job
  3. Save all metadata in DynamoDB

This is all that needs to be done to create a new Cron Job and connect it to a queue. Both the Cron Job and the queue are mapped to each other in DynamoDB.

Execution of Cron Jobs

Execute a Cron Job with AWS Eventbridge, AWS Lambda, AWS DynamoDB

Let's have a look at what happens when a Cron Job is executed.

The following services are involved here:

  • EventBridge
  • SQS
  • Lambda
  • DynamoDB

A Cron Job will be executed in some pre-defined time. For example every 5 minutes.

This Cron Job calls a lambda function. This lambda function is named cronLambda here.

The lambda function executes three main steps:

  1. Get the Cron Job from DynamoDB
  2. Prepare message for SQS
  3. Send a message to SQS

Get the defined Cron Job

EventBridge sends an event to the lambda function. This event includes the ARN:

{
   "resources":[
      "arn:aws:events:us-east-2:661734112387:rule/customerCron-123"
   ]
}

From this ARN we can query our DynamoDB and get the defined Cron Job.

The most important data we need:

  • Target & Method: Which URL should be called
  • Queue ID: Which queue belongs to this Cron Job

Prepare SQS Message

The next step is to prepare the actual SQS message. The message follows a simple format. We add some headers (Accept & Content-Type) and some more metadata like:

  • Timestamp
  • Target
  • Method
  • Variant
  • Etc.

Send SQS Message

The final step is to send the message to SQS. SQS and our connected lambda function take care of the rest.

UI

This is what the backend looks like. How do we present everything to the user?

There are four different screens we need to work on:

  1. Cron Job Overview
  2. Creating a Cron Job
  3. Cron Job Details
  4. Including Cron Job Requests in the request screen

Cron Job Overview

A list of all cron jobs

Cron Job Overview is a simple list of all Cron Jobs. It shows the name, the state, and the next execution.

Creating a Cron Job

Creating a new Cron Job

This modal is needed to create a new Cron Job. I want to give the flexibility to either enter a custom cron expression. Or to enter an interval like every 5 minutes and we will create the cron expression.

On the right side of the pane, the user will see the execution dates of his Cron Job to get an idea if that is correct or not.

The user can also define the target URL and method.

Cron Job Details

Cron Job Details Screen

This screen is pretty similar to the queues. It will show some metadata (defined target, name, cron pattern). And it also shows the successful or unsuccessful invocations.

This screen should also show the upcoming executions of the Cron Job.

Requests

The requests screen simply adds a filter for requests coming either from a queue or from a Cron Job and the variant is shown in the table.

All Requests from Cron Jobs and Queues

These are all screens needed. Most of them are still in Figma or already implemented.

What's on Steroids now?

I called the post Cron Jobs on Steroids. Normally a Cron Job simply executes some script and that's it. I want to enhance this with retries, delays, etc. The great thing is that we can simply combine the flexibility and power of message queues with the simplicity of Cron Jobs. With that, it is super easy to build resilient, scalable, and performant systems, even without an AWS Account 😉

Final Words

Thanks! If you want to try it out head over to ServerlessQ and sign up.

I also see many possibilities to build the same workflows with Step Functions, but for now, I'll stick to lambda 🙂

If you want to read more about how I am building ServerlessQ make sure to follow me on Twitter

Did you find this article valuable?

Support Sandro Volpicella by becoming a sponsor. Any amount is appreciated!