Many developers are talking about moving functionally to the cloud. One way to speed up the processing of data and/or files is to use a microservice. Microservices can fan out processing to almost instantly process many requests at roughly the same time. This will make your users happy since they will get a faster response. For this article, I am going to show you how to process invoice data using an AWS LAMBDA that I architected for a company this year. This could also be easily done with Azure Functions. This one is relatively simple and I will go through it step by step.
The Problem
For this company, they wanted to automate the submission of invoices. Currently, the vendors email their invoices to accounting, and then they must enter the data, by hand. This is time-consuming and costly. They wanted me to automate this process, so the data is inserted into SAP by a microservice. Additionally, the invoices that the vendors submit must be stored for 7 years and then automatically deleted, by a different microservice.
The Solution
I will go through the architecture step by step. This architecture is very similar to one that I have previously architected and written for a different company.
Vendor Submits Their Invoice
Vendors will be uploading their invoices in this first step. Before that, via our web application, they can pull down invoice data from Salesforce in an Excel spreadsheet. They can then make changes and then submit their final invoice data via the application to an API endpoint. In this case, I am using ASP.NET WebAPI.
As you can see, everything is secured by Okta.
API Stores File and Sends Data to a Queue
Once the invoice hits the API, the first thing that it will do is store the Excel file in blob storage. In this case, I am using AWS S3. We do this first because we need the id of the file in storage. That id will be placed on the queue with other metadata like the vendor id and the user’s id (email address) and invoice id from Salesforce.
Queues hold only text data and that is a limited size. I usually save this type of data as JSON so it can be easily parsed or deserialized into a class object.
Processing the Invoice Data
Placing the data onto the queue will immediately fire off the microservice and provide the message from the queue to the microservice. In this case, I am using an AWS LAMBDA. Next, I will describe the business logic steps that the microservice will perform.
Load Data
Using the id from blob storage, the microservice will pull down the Excel spreadsheet that the vendor submitted.
Change Invoice Status
In this step, the microservice will change the status of the invoice in Salesforce to “Submitted”. This invoice will also be locked in Salesforce.
Process the Data
Next, the microservice will group and process data so it can be sent to SAP. This includes getting cost data, tax data, calculating summaries, and more. There are about 5 steps in this process that I will not go over here since that is business logic and will not be useful to anyone else but my team.
Send Invoice Data to SAP
In this step, the data from step 3.3 will be inserted into SAP. After that completes, then the status of the invoice in Salesforce will be set to “Received”.
Load Data
The last step of this microservice will email the person that submitted the invoice that the processing of the invoice has completed. If there is a problem with the data, then they will be notified.
Processing Invoices Faster
After what I have written above, you might ask since a microservice only does one thing (that is all they should do), why not just put it in the API? Easy, this architecture can instantly handle a spike of invoices being submitted at around the same time. Have you ever gone to a website, like Comic-Con or even the Microsoft MVP Summit site when registration opens, and it does not work right, or you can’t even get into the site at all? An architecture like this could help or even eliminate those types of issues.
Another huge plus to using microservices like this is that as soon as the API call is complete, they can continue to use our app while the invoice is being processed. This makes for a great user experience.
Cleaning Up Old Invoices
For legal reasons, we are keeping the invoices that the vendors submitted for 7 years. To cut down on storage costs, there is also a scheduled microservice that will remove any of the invoices that are older than 7 years.
Summary
Using microservices allows you to decouple functionality (which is always a good idea) in the cloud. I have previously written how I used architecture very similar to this to speed up processing 600K data payloads from a mobile device from 7 - 70 seconds to 250ms. That is no joke, and it allowed that company to bring on very large contracts since, for them, performance was keeping them from landing these contracts. Then that company canceled my contract! Oh well.
If you have comments or questions, please leave them below. Stay tuned because I plan to write more microservice architecture articles in the future.