Logging is one of the most important factors to trace any issue in the system. Multiple requests will reach the system at the same time. Each request will have its own way. In Microservices architecture the complexity will be more. The same request will get forwarded to multiple systems. Each layer in each system will add some logs to trace the issue. But we need to link all the logs belonging to the request. Then only we identify how the request is behaving on each system. To Link all the logs into a single link, that is why correlation is required.
There are multiple frameworks available to correlate all the logs.
Azure is providing azure app insights to link and showcase all the requests into a single pipeline. Kibana and Serilog also provide to link the logs. Here we are going to see to link the logs without any third-party components.
Setup the CorrelationId
The request will originate from any client application or any subsystem. Each request can be identified by its reuestId. So the origin system can send its requestId or we can create the Id in case the origin didn't send it.
public class CorrelationHeaderMiddleware {
private readonly RequestDelegate _next;
public CorrelationHeaderMiddleware(RequestDelegate next) {
_next = next;
}
public async Task InvokeAsync(HttpContext context) {
var correlationHeaders = context.Request.Headers["CorrelationId"];
string correlationId;
if (correlationHeaders.Any()) {
correlationId = correlationHeaders[0];
} else {
correlationId = Guid.NewGuid().ToString();
context.Request.Headers.Add("CorrelationId", correlationId);
}
context.Items["CorrelationId"] = correlationId;
await _next(context);
}
}
We have created new middleware that will fetch the correlation from the httpcontext or it will create the new id and set it into the request. Middleware is added into the request pipeline. All the requests will be processed and they will be added with CorrelationId.
Setup Logs
The request will get into multiple flows and add its logs. While adding the logs it will send the information alone to the log. While saving the logs we are fetching the CorrelationId from the request and adding it with the logs. So all the logs will get added with CorrelationId. After that, we can store the logs based on our format.
public class Logger: ILogType {
private readonly ILogger < Logger > _logger;
private readonly IHttpContextAccessor _httpContextAccessor;
public Logger(ILogger < Logger > logger, IHttpContextAccessor httpContextAccessor) {
_logger = logger;
_httpContextAccessor = httpContextAccessor;
}
private string GetCorrelationId() {
return _httpContextAccessor.HttpContext.Items["CorrelationId"] as string;
}
public void ProcessError(string message) {
_logger.LogError(GetCorrelationId() + ": " + message);
}
public void ProcessLogs(string message) {
_logger.LogInformation(GetCorrelationId() + ": " + message);
}
public void ProcessWarning(string message) {
_logger.LogWarning(GetCorrelationId() + ": " + message);
}
}
The request needs to be forwarded to multiple systems to complete it. To send the request to multiple subsystems, we need to use any protocol like HTTP or AMQP. In this case, we need to attach the CorrelationId as a header with the corresponding protocol. The corresponding subsystem will read the id and logs the stuff.
All the log data will be get stored in the same or multiple systems. But CorrelationId will be the key to linking all the logs. Using this id we can fetch the request flow order. It will help to understand the system's flow as well as to trace any issue in the system's behavior. Adding more logs will increase the system space. We need to ensure the log type on each log also.