Processing & Handling multiple messages from WebHook

What is WebHook?

Webhooks and connectors help to connect the web services to channels and teams in Microsoft Teams. Webhooks are user-defined HTTP callback that notifies users about any action that has taken place in the Teams channel. It's a way for an app to get real-time data.

Reference taken

• https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/what-are-webhooks-and-connectors

Use Case

Client I have a Webhook which sends post API calls; the request or the webhook will send millions of records. You need to capture the data & process it without losing any data messages. Also, the messages contain a scenario where queues containing master data should be processed first, and queue messages with child data should be processed later. Child queues should not be processed until master data is available in the backend.

Answer

Let's go through the picture; the picture speaks a lot in gathering the text; let's go through the above architecture for a few mins before we further understand the design.

Things to Consider

1. The API endpoint listening to the WebHook should not have much business validation or DB transaction, as the data flowing from the WebHook is in millions of records. If we do any validation here, there would be a chance that the message could be lost, which we don't want to happen.
2. Until master data is available, we should not insert child data.
3. Scalability of the application to read multiple records.

Design Flow

1. As the data is received from the webhook, we are sending the message to the primary queue as we don't want to lose any messages from the WebHook.
2. Once the message is there in the queue, we can trigger the Azure function by reading the messages from the primary queue and doing any business validation if required at this function level and later send the queue containing master information to the master queue and the child containing information to the child queue.
3. We will have an Azure function, which gets triggers as we get a queue into a master queue and starts processing the message. If there is an issue while processing the message, then the message will be sent to the dead letter queue.
4. Once the message is read and there is no validation issue, we insert the data in the Azure SQL database.
5. We will have another Azure function that gets triggered as a message is pushed to the dead letter queue, and the message starts getting processing here.
6. Once the message is read from the dead letter and is processed successfully, then the message is inserted into the database.
7. If there is any issue while processing from the dead letter queue, then we can resend it back to the master queue to re-try again and avoid any loss of messages.
   Note. Suppose we want to avoid sending it to the master queue again. In that case, we can save the message in a storage-like blob and notify the stakeholder that the message is not processed successfully, where debugging would be required to analyze. In this case, we don't lose any message received from Webhook.
8. Once the queue is available in the Child data queue, the Azure function is triggered, where we validate if the master is available or not. In case the master data is available, then we can send a message to the message to azure service sub-subscriptions, where the Azure function would read the message and process the message into the database.
   Here, the reason why we are sending to Azure service bus subscription is that we want to leverage the option to use Fan out mechanism so that the large messages can be processed quickly. If we want to insert it in the processing order, we use the queue mechanism only.
9. If there is any issue while processing the queue message from the child data queue, we repeat the steps similar to the master queue from step 5 to step 7.
10. In the case of step 8, if the master data is unavailable, we resend the queue to the child data queue again until master data is available to avoid any messages from the queue.

This is one of the processes of processing multiple records from webhook, having a business validation child & master data dependency.

Client. The above design suits my requirement, but now I have an extra condition where irrespective of messages sent from webhook, I want to process a few messages at first and later the others. Let's say I have people who subscribed to my application using Credit cards & debit Cards. I want to process the refund amount for the Credit Card users first, followed by Debit Card Users.

Answer

Generally, the queue is processed in the order they are received. Still, since we need to process a set of higher priority queues first compared to low or medium priority, we can achieve this using the Priority Queue Pattern mechanism.

What is Priority Queue Pattern?

The queues are prioritized for the messages where the messages with higher priority are processed first, and later the queues with lower priority are processed.

How can we achieve this?

They are 3 ways in which this can be achieved.

1. Have separate priority-wise queues, like HighPriority & LowPriority, and try to have more consumers reading for HighPriority & fewer consumers or services reading from low priority so that the higher priority is processed faster.
2. The queue should have priority property like HighPriority Queue having messages with 1 and LowPriorty as 2. While reading the messages, we sort the order and process only the higher priority before the lower priority queue.
3. Create an Azure Service Bus subscription that filters where, depending on property value, the higher priority is sent to a dedicated subscription and the lower priority to another {Azure service bus subscription. Since the consumers on higher priority are more, higher priority messages are read faster than lower priority queue. We have more services to consume HigherPriority messages and fewer services creation or fewer consumers (azure function) to process the messages.

Summary

Webhooks enable the collection and processing of data without any data messages being lost. An Azure function is used in a specific use case to read messages from a primary queue, send messages containing master information to a master queue, and send messages containing child information to a child queue. The program ensures that child queues are not handled before the backend has access to master data. 

Hope you like the article :) .... Thanks for spending time to read my article.

Reference

• https://learn.microsoft.com/en-us/azure/architecture/patterns/priority-queue 
• https://www.serverlessnotes.com/docs/azure-service-bus-implementing-the-priority-queue-pattern