Integrating SQS with Lambda using Event Source Mapping

Introduction

In this article, we are going to learn about.

What is SQS and Lambda
How to integrate SQS with Lambda using Event Source Mapping
Way of processing the messages from SQS

What is SQS?

Amazon Simple Queue Service (SQS) is a managed message queuing service that is used to send, store, and retrieve multiple messages of various sizes asynchronously. We can hook the AWS Lambda function with SQS to process messages in real time.

What is Lambda?

AWS Lambda is a serverless computing service provided by Amazon Web Services (AWS). It allows you to run code without provisioning or managing servers. AWS Lambda is ideal for building microservices, data processing systems, and real-time file-processing applications

Integrating SQS with Lambda

Lambda with SQS allows to creation of applications that are highly available and also fault-tolerant. If a Lambda function fails to process a message, the message is returned to the queue and retried. This combination ensures that no message is lost and that each message is processed successfully. We can use the “Trigger” option in the lambda function to integrate with SQS and process the messages from the SQS queue.

Lambda with SQS

Step 1. Create SQS and Lambda.

I assume you are already aware of how to create AWS Lambda and SQS.

Step 2. Add the required permission to the AWS Lambda.

We need to add the required policies to the AWS Lambda IAM role.

Open the Lambda Page.
Choose the Configuration tab, and then choose Permissions.
Under Role name, choose the link to your execution role. This link opens the role in the IAM console.
Select Add Permission -> Attach Policies in the Permissions Policies section.
In the search field, enter AWSLambdaSQSQueueExecutionRole. Add this policy to the Lambda execution role. This is an AWS-managed policy that contains the permissions to read messages from an AWS SQS queue.

The above AWSLambdaSQSQueueExecutionRole policy contains the below permissions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

Step 3. Add Trigger to AWS Lambda (Event Source Mapping).

Open the Lambda Page
Under Configuration -> Triggers, click the Add trigger button
Choose a trigger type (Here trigger type is AWS SQS)
Configure the required options, and then choose Add

Message Process

We can process the message in the following ways.

One by One
Batch Process

One by One

We can configure the lambda to process the messages one by one. We can use the “Batch Size” property and assign value to 1.

Batch Size

Configuring an SQS batch size of 1 for Lambda with an SQS event source can lead to,

Increased costs.
Longer processing times for high message volumes, especially if the Lambda function needs to scale rapidly.
Exhaustion of available concurrency for Lambda functions in the AWS account.

Batch Process

When lambda is subscribed to an SQS, Lambda polls the queue as it waits for messages to arrive and consumes messages in batches. We can configure this with the “Batch Size” property in the lambda trigger.

How SQS and Lambda Batching Works?

Batch Size

The maximum number of messages that Lambda will retrieve from the SQS queue in a single batch.
The maximum is 10,000 for standard queues and 10 for FIFO queues.

Batch Window

The maximum time that Lambda will wait before processing a batch of messages. Lambda will process the batch when either the batch size is reached or the batch window time elapses, whichever occurs first.
The batch Window supports only for Standard queue.
When the batch size is greater than 10, the batch window should be at least 1 second.

	Scenario 1	Scenario 2	Scenario 3
	Batch size = 100 Batch Window = 300 seconds Messages = 30 Queue type = Standard	Batch size = 100 Batch Window = 300 seconds Messages = 205 Queue type = Standard	Batch size = 10 Messages = 95 Queue type = FIFO
Lambda invocations	1	3	10
Why?	With a small number of messages (fewer than the batch size of 100), Lambda will not wait for the batch window to elapse.	2Lambda invocations, each processing 100 messages. Since you have more than the batch size (100), Lambda will process the messages in batches of 100 as quickly as it can until the queue has fewer than 100 messages left. The remaining 5 messages are less than the batch size. Lambda waits for either the batch size to be met or the batch window to elapse. In this case, since no additional messages are arriving to meet the batch size of 100, Lambda waits for the entire batch window of 300 seconds before processing these messages.	9 Lambda invocations, each processing 100 messages. Since you have more than the batch size (100), Lambda will process the messages in batches of 100 as quickly as it can until the queue has fewer than 100 messages left. The remaining 5 messages are less than the batch size. Still, Lambda picks those messages as another batch and will process it. Here no waiting time (Batch Window) to meet the batch size.

In scenario 2, the number of messages in the queue is less than the batch size. Lambda waits for the batch window to complete. This is why the remaining 5 messages were processed after the 300-second batch window, causing the delay.

Recommendations to Avoid Delays

Adjust the Batch Window: To reduce latency for smaller batches, consider lowering the batch window. A shorter batch window will prompt Lambda to invoke more frequently with smaller batches.
Adjust the Batch Size: If you often process small batches of messages, lowering the batch size can ensure more frequent processing.

Example. We will create a demo setup for the batch process of the SQS messages.

Lambda Trigger Configuration

Lambda Trigger Configuration

Lambda Code

public SQSBatchResponse FunctionHandler(SQSEvent evnt, ILambdaContext context)
{
    Console.WriteLine("***************************  BATCH START  ********************************");

    for (int i = 0; i < evnt.Records.Count; i++)
    {
        try
        {
            Console.WriteLine("Actual Message:");
            Console.WriteLine(evnt.Records[i]);
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message + " - Message Id: " + evnt.Records[i].MessageId);
        }
    }

    Console.WriteLine("**************************** BATCH END  ******************************");
}

SQS Message

[
  {
    "id": "994c506e-ba4a-4712-9746-aa0429d67940",
    "message": "message-1"
  },
  {
    "id": "1487c0e4-b164-41a8-b950-db4805887527",
    "message": "message-2"
  },
  {
    "id": "64f9063c-2890-473e-b1e1-03adf2591d59",
    "message": "message-3"
  },
  {
    "id": "d907aee7-b1b3-4048-9874-a3de334b1f44",
    "message": "message-4"
  },
  {
    "id": "96415fa7-488e-4761-b7cd-6742c4dc4fd7",
    "message": "message-5"
  },
  {
    "id": "5b9ae959-9558-4c9c-87d3-7fc8d250ce92",
    "message": "message-6"
  }
]

Lambda Log

Lambda has processed the messages by batch, and the first batch has 4 messages, and the second batch has 2 messages.

Batch 1

Batch 2

Handling Failure

If any message in the batch fails during processing, Lambda considers the entire batch a failure, regardless of the success of the other messages in the batch. Those messages are returned to the queue. It means that if one message fails, the entire batch is retried, including those messages that were successfully processed.

There is a feature called "Partial batch response" in AWS Lambda, which is used to inform SQS about only the failed records that need to be processed again.

We have to Include ReportBatchItemFailures in the EventSourceMapping configuration to enable the "Partial batch response" in AWS Lambda.

Lambda Code

public SQSBatchResponse FunctionHandler(SQSEvent evnt, ILambdaContext context)
{
    List<BatchItemFailure> batchItemFailures = [];

    Console.WriteLine("***************************  BATCH START  ********************************");

    for (int i = 0; i < evnt.Records.Count; i++)
    {
        try
        {
            // Process the message
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message + " - Message Id: " + evnt.Records[i].MessageId);

            // Add failure messages to BatchItemFailure
            batchItemFailures.Add(new BatchItemFailure { ItemIdentifier = evnt.Records[i].MessageId });
        }
    }

    Console.WriteLine("**************************** BATCH END  ******************************");

    return new SQSBatchResponse(batchItemFailures); // Return the list of failure messages to SQS
}

Summary

In this article, you have learned the following topics.

What is SQS and Lambda
How to integrate SQS with Lambda using Event Source Mapping
Way of processing the messages from SQS