Explain Batch Processing in MuleSoft

This article is the continued part of the MuleSoft articles series; if you would like to understand the basics of MuleSoft, refer to these articles before going next.

• Getting Started MuleSoft
• How To Connect to Azure SQL Database with MuleSoft
• Load Data from S3 Bucket Using MuleSoft S3 Connector
• How To Call Stored Procedure with MuleSoft
• How To Post Data Using Stored Procedure with MuleSoft
• How To Set Variable and Payload in MuleSoft
• Fetch MySQL Data Using Database Connector in MuleSoft
• Retrieve Data From Database Based On Conditions In MuleSoft
• Create and Write CSV Files Using File Connector in MuleSoft
• Read Files Using File Connector in MuleSoft
• Default Error Handling Behavior of a Mule Application
• Basic Internal Error Handlers in Mule application
• Global Error Handling in Mule Application
• How To Invoke REST Web Service in MuleSoft
• How To Invoke SOAP Web Service in MuleSoft
• How To Create and Trigger Schedulers in MuleSoft
• Publish and Subscribe Messages using VM Connector in MuleSoft

In this article, you will learn how to create and use batch processing in the MuleSoft application.

What is Batch Processing, and Why?

Batch processing in MuleSoft is a mechanism that allows you to process large sets of data in a systematic and efficient manner. It is a feature that helps handle high volumes of data by breaking it down into smaller, manageable chunks, processing each chunk independently, and then aggregating the results. Batch processing is particularly useful for scenarios where data needs to be processed in batches, such as bulk data updates, transformations, or integrations.

Key features and benefits of batch processing in MuleSoft

1. Large Data Sets: Batch processing is designed to handle large data sets that may be impractical to process in a single transaction. It allows you to divide the data into smaller chunks, known as "records," and process them one batch at a time.
2. Parallel Execution: Batches can be processed in parallel, allowing for better utilization of system resources and improving overall performance. This is especially beneficial when dealing with high volumes of data.
3. Error Handling: Batch processing includes robust error-handling mechanisms. If an error occurs during the processing of a record in a batch, MuleSoft can manage the error and decide whether to skip the record, retry it, or mark it for manual review.
4. Transaction Management: Each batch is treated as a separate transaction. This means that if an error occurs in one batch, it doesn't affect the processing of other batches. It helps ensure data integrity and consistency.
5. Configurability: Batch processing in MuleSoft is highly configurable. You can define the size of each batch, the number of parallel threads, error-handling strategies, and more. This flexibility allows you to tailor batch processing to specific requirements.
6. Reusability: You can create reusable batch jobs that can be used across different parts of your application or across different applications. This promotes a modular and maintainable architecture.
7. Scalability: Batch processing supports scalability by allowing you to distribute the processing of batches across multiple nodes or instances of your MuleSoft application. This is particularly useful in scenarios where horizontal scaling is required.

Here is a simplified example of a batch processing flow in MuleSoft.

In this example, the `batch:job` defines a batch job named "myBatchJob" that logs each record being processed. The actual processing logic would go inside the `<batch:process-records>` section.

Batch processing is a valuable feature in MuleSoft when dealing with scenarios involving large volumes of data that require efficient, parallelized, and fault-tolerant processing. It helps streamline data integration, transformation, and update tasks in a way that is scalable and manageable.

Architecture

MuleSoft's batch processing architecture is designed to efficiently handle large volumes of data by breaking it down into smaller batches and processing each batch independently. The batch processing module in MuleSoft is built on top of the Anypoint Batch module, and it provides a set of components and features to facilitate batch processing in Mule applications. Below are key components and architectural concepts related to MuleSoft's batch processing.

1. Batch Job

• A batch job is the main container for the batch processing logic. It encapsulates the entire processing flow for a set of records.
• It defines the input source, the processing logic, and the output phase.

2. Batch Step

• A batch job is divided into one or more steps, where each step represents a specific phase in the processing.
• Common steps include the input phase (`batch:input`), the processing phase (`batch:process-records`), and the output phase (`batch:on-complete`).

3. Input Source

• The input source defines where the batch job retrieves its data. This can be a database query, a file, a message queue, or any other source.
• MuleSoft provides connectors and components to interact with various data sources.

4. Record Aggregation

• Records are processed in chunks, known as batches. The size of each batch is configurable.
• The `batch:process-records` phase is where the actual processing logic is applied to each record in the batch.

5. Error Handling

• MuleSoft's batch processing includes robust error-handling mechanisms. If an error occurs during the processing of a record, the system can manage the error based on configured strategies, such as skipping the record or retrying it.

6. Transaction Management

• Each batch is treated as a separate transaction, ensuring that if an error occurs in one batch, it doesn't affect the processing of other batches.

Transactional behavior helps maintain data integrity and consistency.

7. Parallel Execution

• Batches can be processed in parallel, utilizing multiple threads or nodes to improve performance.
• The level of parallelism is configurable based on the requirements of the application and the resources available.

8. Flow Reference

• Batch jobs often include references to existing Mule flows for processing. These flows contain the logic that should be applied to each record in the batch.

9. Completion Phase

• The `batch:on-complete` phase is where you define actions to be taken once the entire batch job has been processed. This could include logging, sending notifications, or executing additional logic.

Here is a simplified example of a MuleSoft batch-processing flow.

This example illustrates the basic structure of a MuleSoft batch job with input, processing, and completion phases. The actual processing logic and configuration would depend on the specific requirements of your integration scenario.

Getting Started with Anypoint Studio

Let's do a quick project setup and include the required global config files; these are the quick steps.

First, open Anypoint Studio and Create a New Mule Project.

Enter the project name, select runtime and project location, and click the Finish button.

In this article, I will demonstrate how to break a big flat file into multiple files with the same record counts, delete the file from the folder after processing, and write the file in a different folder.

First of all, drag and drop File > On New or Updated File connector from the Mule palette to message flow.

Click on the Add button so all required .jar files will be added to the solution.

Next, let’s configure the file connector.

Provide the connector configuration (my configuration is just a C drive), provide the file directory path, select the fixed frequency as a scheduling strategy, and provide post-processing actions.

Here is my file config connection.

Let’s drag and drop the logger and provide the logger message.

Also include a Transform message component as well to transform the payload into Java.

Now drag and drop Batch Job from the Core palette and give Batch block size, the file has 10000 records, and I am going to break and write the file one record in each file.

Drag and drop a Set Variable control to provide the file name from Java Payload and provide variable name and value.

Add a new Transform message to convert Java content to csv content before writing the files.

Next, drag and drop File > Write connector from the Core palette into the batch job Processor step.

Provide configuration and Path (Writing files based on Index file column values), and the content should be the entire payload.

Output folder configuration to write the files.

At last, Let's run the application to see the result; right-click on the file and click Run as Mule Application. Once you see the message Deployed, everything is good, and the application is deployed.

Go to the output folder location to see the files.

As you can see input folder file has 10k records, and the output folder file has 10k+1 as the original file, also moving to the output folder. The other good thing is just took seconds to write those many files.

Conclusion

In this article, we learned a basic understanding of Mulesoft batch processing and real-time samples using batch jobs in the MuleSoft application.