Distributed Transaction in C# Microservices using SAGA Pattern

What is a distributed transaction?

In the world of microservices, each service has its own local database and some operations we do in microservices may involve multiple database operations for multiple services.

So a distributed transaction may contain local transactions for multiple databases.

What we often encounter is the need to guarantee data consistency, updated in parallel by different actors.

There are some solutions handling the distributed transaction in microservice that we can follow, such as Seata, servicecomb-pack, etc.

But both Seata and servicecomb-pack are not friendly and hard to use in C# due to there being no SDK for C# version.

After some exploration, I found a distributed transaction framework named DTM, written in Golang, that was easy to use and supported multilingual technology stack.

And I will use a popular distributed transaction pattern named Saga to introduce how to do a distributed transaction using C# with this distributed transaction framework.

What is Saga?

Saga first appeared in a paper published by Hector Garcaa-Molrna & Kenneth Salem in 1987.

The core idea is to split a long transaction into multiple short transactions, coordinated by the Saga transaction coordinator, with the global transaction completing normally if each short transaction completes successfully, and invoking the compensating operations one at a time according to the reverse order if a step fails.

The Saga pattern is a widely used pattern for distributed transactions. It is asynchronous and reactive.

What is DTM?

DTM is a distributed transaction framework which provides cross-service eventual data consistency. It provides saga, tcc, xa, 2-phase message strategies for a variety of application scenarios. It also supports multiple languages and multiple store engine to form up a transaction.

There are some features of DTM:

• Extremely easy to adopt
• Easy to use
• Multi-language support
• Easy to deploy, easy to extend
• Multiple distributed transaction protocols

For more details about DTM, you can visit its official website or Github page.

https://en.dtm.pub/

https://github.com/dtm-labs/dtm

After a series of introductions, I will use a classic transfer of money across banks example to demonstrate how to use it.

Setup DTM server

Here use docker to setup DTM quickly.

docker run -d -p 36789:36789 -p 36790:36790 --name=dtm-svc yedf/dtm:1.13

To quickly experience DTM, boltdb is used to store data, and if you are deploying in a production environment, remember to switch to mysql or other storage engines

NOTE: It support using HTTP and GRPC to communicate with DTM server, 36789 is HTTP port and 36790 is GRPC port.

 

Basic Sample

Before creating the C# microservices, let's take a look at a typical timing diagram of a successfully completed SAGA transaction.

As we can see, in this timing diagram, the transaction initiator, after defining the orchestration information for the entire global transaction (including the forward and reverse compensation operations for each step), commits it to the DTM server, which then executes the previous SAGA logic step by step.

Now, we will create two microservices using .NET 6 minimal API.

The first is the transfer out microservice:

var builder = WebApplication.CreateBuilder(args);

// Add Dmtcli
builder.Services.AddDtmcli(x =>
{
    x.DtmUrl = "http://localhost:36789";
});

var app = builder.Build();

app.MapPost("/api/TransOut", (HttpContext httpContext, TransRequest req) => 
{
    Console.WriteLine($"TransOut, QueryString={httpContext.Request.QueryString}");
    Console.WriteLine($"User: {req.UserId}, transfer out {req.Amount} --- forward");

    return Results.Ok(TransResponse.BuildSucceedResponse());
});

app.MapPost("/api/TransOutCompensate", (HttpContext httpContext, TransRequest req) =>
{
    Console.WriteLine($"TransOutCompensate, QueryString={httpContext.Request.QueryString}");
    Console.WriteLine($"User: {req.UserId}, transfer out {req.Amount} --- reverse compensation");
    
    return Results.Ok(TransResponse.BuildSucceedResponse());
});

app.Run("http://*:10000");

And the second is the transfer in microservice

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddDtmcli(x => 
{
    x.DtmUrl = "http://localhost:36789";
});

var app = builder.Build();

app.MapPost("/api/TransIn", (HttpContext httpContext, TransRequest req) =>
{
    // 
    Console.WriteLine($"TransIn, QueryString={httpContext.Request.QueryString}");
    Console.WriteLine($"User: {req.UserId}, transfer in {req.Amount} --- forward");

    return Results.Ok(TransResponse.BuildSucceedResponse());
});

app.MapPost("/api/TransInCompensate", (HttpContext httpContext, TransRequest req) =>
{
    Console.WriteLine($"TransInCompensate, QueryString={httpContext.Request.QueryString}");
    Console.WriteLine($"User: {req.UserId}, transfer out {req.Amount} --- reverse compensation");

    return Results.Ok(TransResponse.BuildSucceedResponse());
});

app.Run("http://*:10001");

NOTE: For the sake of simplicity, specific database operations are not introduced here, but are replaced by log output.

Adding a console application to impersonate the transaction initiator. And we should add one of the following package from nuget.

dotnet add package Dtmcli --version 1.0.0

or

dotnet add package Dtmgrpc --version 1.0.0

NOTE: Dtmcli uses the HTTP protocol to communicate with the DTM server, and Dtmgrpc uses the GRPC protocol.

The following example will use Dtmcli to demonstrate how to commit a SAGA transaction.

var services = new ServiceCollection();

services.AddDtmcli(x =>
{
    x.DtmUrl = "http://localhost:36789";
});

var provider = services.BuildServiceProvider();
var dtmClient = provider.GetRequiredService<IDtmClient>();

var outApi = "http://192.168.0.101:10000/api";
var inApi = "http://192.168.0.101:10001/api";

var userOutReq = new Common.TransRequest() { UserId = "1", Amount = -30 };
var userInReq = new Common.TransRequest() { UserId = "2", Amount = 30 };

var cts = new CancellationTokenSource();

var gid = await dtmClient.GenGid(cts.Token);
var saga = new Saga(dtmClient, gid)
        .Add(outApi + "/TransOut", outApi + "/TransOutCompensate", userOutReq)
        .Add(inApi + "/TransIn", inApi + "/TransInCompensate", userInReq)
        .EnableWaitResult()
        ;

await saga.Submit(cts.Token);

Console.WriteLine($"Submit SAGA transaction, gid is {gid}.");

A few simple steps, a complete SAGA distributed transaction has been written!!

Let's see what the result of this example runs.

After running this example, you can see from the application logs that both the transfer in and out are successful, and you can see from the dtm server's logs how the DTM server calls the microservice API.

Although a successful example has been completed, this example is too simple, both the transfer in and out are one-time successes, and do not reflect the problems encountered in the microservices world, such as network jitter, machine downtime, and process crash.

In the transactional domain, exceptions are a key consideration, that can lead to inconsistencies. When we are doing distributed transactions, then exceptions in distribution appear more frequently, and the design and handling of exceptions is even more important.

For handling those exceptions, DTM pioneered the sub-transaction barrier technology that can help us easily to solve exception problems and greatly reduces the threshold of using distributed transactions.

The sub-transaction barrier can achieve the following effect, see the figure.

 

Sub-transaction Barrier Sample

Similarly, let's first look at the timing diagram of this failed SAGA distributed transaction.

In this example, we will simulate some abnormal situations to see if the sub-transaction barrier can be handled correctly to ensure the accuracy of the transaction

Let's add an API to simulate a failed transfer in operation.

app.MapPost("/api/TransInError", (HttpContext httpContext, TransRequest req) =>
{
    Console.WriteLine($"TransIn, QueryString={httpContext.Request.QueryString}");
    Console.WriteLine($"User: {req.UserId}, transfer in {req.Amount} --- forward");

    // status code = 409 || content contains FAILURE
    // return Ok(TransResponse.BuildFailureResponse());
    return Results.StatusCode(409);
});

And change the saga instance.

var saga = new Saga(dtmClient, gid)
        .Add(outApi + "/TransOut", outApi + "/TransOutCompensate", userOutReq)
        .Add(inApi + "/TransInError", inApi + "/TransInCompensate", userInReq)
        .EnableWaitResult()
        ;

What happens in this case?

The most immediate result is that the distributed transaction fails directly because the transaction initiator threw an exception.

Transaction failures are common, in which we want to ensure that the consistency of the data is correct.

But there is also a more serious case where the forward operation of the transfer fails, but the rollback operation of the transfer is also performed! Originally, the transferred party should have had some more money in this transaction, but in fact it was a loss.

Doing the compensation of forward operation is error prone, because the failure in the forward operation may happen before or after the commitment of the transaction. sub-transaction barrier has taken care of these cases for us. If the failure in forward operation happened after the commitment, the barrier will call the compensation; and if the failure happened before the commitment, the barrier will ignore the compensation.

The principle of subtransaction barrier technology is to create a branch operation status table dtm_barrier in the local database, with the unique key of gid-branch_id-branch_op.

1. open the local transaction
2. for the current operation op (forward|try|confirm|cancel), insert ignore a row gid-branchid-[op], if the insertion is unsuccessful, commit the transaction returns success (common idempotency control method)
3. if the current operation is cancel|reverse, then insert ignore a row gid-branchid-try, if the insertion is successful (note that it is successful), then commit the transaction and returns success
4. call the business logic within the barrier, if the business returns success, the transaction is committed to return success; if the business returns failure, the transaction is rollbacked and returns failure

Let's try this sub-transaction barrier technique next.

SDK provides the following method for us.

public Task Call(DbConnection db, Func<DbTransaction, Task> busiCall);

What we need to do is write our own logic inside busiCall.

The following code improves the compensation API for transfer in and demonstrates how to use the sub-transaction barrier.

app.MapPost("/api/BarrierTransInCompensate", async (HttpContext httpContext, TransRequest req, IBranchBarrierFactory factory) =>
{
    Console.WriteLine($"TransIn, QueryString={httpContext.Request.QueryString}");
    
    // create barrier from query
    var barrier = factory.CreateBranchBarrier(httpContext.Request.Query);

    using var db = Db.GeConn();
    await barrier.Call(db, async (tx) =>
    {
        // some exception occure, should not output this one.
        Console.WriteLine($"User: {req.UserId}, transfer in {req.Amount} --- sub-transaction handle!");

        await Task.CompletedTask;
    });

    return Results.Ok(TransResponse.BuildSucceedResponse());
});

We expect that sub-transaction handle log will not be output here.

var saga = new Saga(dtmClient, gid)
        .Add(outApi + "/TransOut", outApi + "/TransOutCompensate", userOutReq)
        .Add(inApi + "/TransInError", inApi + "/BarrierTransInCompensate", userInReq)
        .EnableWaitResult()
        ;

The result of this transaction call is indeed as we expected, and some information is also output from the sdk for our reference.

Will not exec busiCall, isNullCompensation=True, isDuplicateOrPend=False

The SDK tells us that it will not perform the current business operation because the current operation is null compensation.

Compared with the previous example, the sub-transaction barrier technology can really help us easily handle abnormal situations and ensure data consistency.

Here is the source code you can find on my Github page.

• DtmSagaSample

Summary

This article introduces some simple distributed transaction knowledge, as well as DTM, a very simple and easy-to-use distributed transaction framework, and uses it to complete a few simple examples.

SAGA pattern is one of the most commonly used modes in DTM, and 2-phase message, TCC, XA patterns are also supported by DTM.

We can also explore distributed transaction solutions in many scenarios with DTM.

I hope this will help you!

Reference

• DTM tutorial page
• SAGA of DTM tutorial
• DTM github page
• Saga Pattern: how to manage distributed transactions with microservices