Create a Notification Service using AWS, Twilio, Docker and Node.js

July 30, 2021

How to Create a Scalable, Reliable Notification Service using AWS, Twilio, Docker and Node.js

Staying in touch with your users is paramount, especially as a startup. But the complexities of setting up a scalable notification service that can handle increasingly large volumes of automated notifications delivered on multiple platforms can be overwhelming. Facing this exact situation while working at Skillbee, I created a viable solution.

This blog post is to ensure that you do not have to start from scratch. You can find the code, with a quick-start README here.

Requirements

Our system needed the following capabilities:

Handle communication on email, SMS, WhatsApp and Android notification
Personalize notifications for the target user
Have in-built failure-tolerance and redundancy to ensure message delivery
Have priority channels to deliver urgent communication
Do all of this at scale — our required throughput was around a million notification a day, at the time

I designed a micro-service based approach that had components which communicated with each other to get work done. The components were:

API Service: This service exposed REST endpoints which can be hit by our backend/admin-tools to trigger notifications
Queueing Service: When a notification was triggered, the API service vetted the request and upon verification, sent this request to the queueing service for consumption and delivery
Consumption Service: This service constantly listened to the queueing service and when a notification was found, it consumed the notification from the queue, and processed it based on metadata.
Delivery Service: This was called by the consumption service and was responsible for the final delivery of the notification to the target users.

API Service

The API service exposed different endpoints for all modes of notifications — that communicating services could hit with POST requests. The body of the POST request had information about the message and targets, while the endpoint being hit decided the mode of notification. Note that the body could contain multiple messages, and each message could have multiple targets.

For example, here’s a sample request body for an email notification sent to the endpoint host:PORT/api/v1/notify/email. Refer to the linked GitHub repository for more examples.

{
    "data":[
        {
            "content":{
                "subject" : "Sample Email's Subject",
                "html" : "<p> Sample email's body</p>"
            },
            "targets":[
                "email1@foo.bar",
                "email2@boo.bar"
            ]
        }
         {
            "content":{
                "subject" : "Sample Email 2's Subject",
                "html" : "<p> Sample email 2's body</p>"
            },
            "targets":[
                "email3@foo.bar"
            ]
        }
    ]
}

Queueing Service

I used Amazon’s Simple Queuing Service (SQS) as the queueing service for our notification system. SQS stored all vetted requests sent by the API service and exposed them for consumption. You can refer to my post on setting up SQS for queueing and consumption from NodeJS to help setup SQS for your notification service. More information on this is also available in the quick start README of the shared repository.

Consumption Service

The consumption service continuously polled SQS queues and consumed messages . It then processed them based on their intended target and channel, tapping into delivery services. You can refer to the repository for the code and instructions on setting up the consumption service and connecting it to SQS. I wrote the consumption service in NodeJS and used the AWS-SQS SDK for polling and consumption of SQS queues.

Delivery Service

I used Twilio as the primary delivery service for SMS and WhatsApp communication. SendGrid — which is owned by Twilio as well — was used as the primary email delivery service. Phone notifications were delivered using Google’s Firebase service. Notably, there is no vendor locking in this notification service, and new delivery services can be easily plugged in/removed. Additionally, logic-based redundancies can also be built in, in the event that a delivery service is down.

Feel free to get in touch with me if you need any further help setting your notification service up. This post, coupled with the repository will hopefully make this easier for you. Thank you for reading!