AWS Cloudwatch Integration Using Serilog Sinks In .NET Core API And Custom Logging Mechanism

This blog explains how to integrate AWS cloudwatch service with .NET Core API using Serilog sinks and how to capture custom log details for a required functionality. With this approach you will be able to send all your log messages directly to Cloudwatch service and view them from Cloudwatch UI. Also demonstrates how to collect required/necessary custom logs in a separate log group.

Introduction

 

This blog explains a simple and tested approach on how to integrate .NET Core API with AWS Cloudwatch service using serilog sinks and send log messages directly to the AWS Cloudwatch service where it can be viewed with filter capabilities and troubleshoot your development or production environment problems easily. This blog also gives a bonus approach to capture custom log messages from anywhere in your application irrespective of LogLevel.

 

Prerequisite

 

To be able to follow through this blog, you need to have basic knowledge of the following things:

1. Basics of .NET Core API and how it works
2. Basic Knowledge about AWS services
3. Basics of Serilog Logging framework and how it works

Why Do We Need This?

 

In web application development, almost all developers (or most) would have faced this problem at least once in their experience "Urgh! Code works fine in my local machine but breaks when deployed to Testing/Prod Server". To handle that scenario, we use logging frameworks across our application code to log all important/critical steps with multiple levels such as Trace, Debug, Info, Warning, Error & Fatal. With the use of logging, we can easily troubleshoot/detect why/where our application code breaks and solve the problem immediately. Serilog is such a framework with a variety of logging capabilities.

 

Instead of storing logs to your database or file system and managing it on your own, we are going one step ahead since AWS provides a cloud based service called Cloudwatch to collect all your application logs and display it in Console UI with filter enabled capabilities. In the current business model, web applications are becoming smart by using more cloud computing services rather building/managing on their own.

In certain cases, we want to log all the actions happening around a critical functionality of our application all the time, for example payment gateway or third party service calling module. Due to this scenario, either you need to flood your AWS cloudwatch storage with information logs OR log INFO level details in the name of ERROR level. Neither way is correct. This blog illustrates an approach to solve this problem by enabling custom logging feature and capture only required details with correct log level.

 

Okay, enough theory. Let's implement it with a step by step approach.

 

Create a .NET Core API application (using Visual Studio 2017 or higher version)

 

This is just an easy and well-known step to start. From Visual Studio, create a new .Net Core Web application and select API with Controllers type which will create a runnable API project with controller having CRUD HTTP methods. This is enough to demonstrate the scope of this blog but in real time, you need to apply the following steps in your existing Core API service.

 

Add necessary NuGet Libraries

 

Following are the required Nuget packages which should be added in order to store logs into AWS cloudwatch.

• AWSSDK.Core - This SDK establishes connection to AWS services and makes the difficult and time consuming deployment process into an easy task.
• Serilog - This library provides all the necessary functions to capture logs from your application and write it in the desired target.
• Serilog.Sinks.AwsCloudwatch - This library helps us to create a connection between .net core application and AWS cloudwatch service

AWS SDK Installation

 

Once AWS SDK library is installed in a .Net core application, you will be able to open AWS Toolkit under View menu. After opening it for the first time, you need to provide your AWS account access key details to login to your account. This will create a default profile which will be used for all your deployments or access to direct AWS services automatically.

 

 

If you want to create and use a different profile other than default, you can easily do that from AWS Toolkit and set AWS_PROFILE environment variable to your profile name. Setting AWS_PROFILE environment variable to your profile lets AWS SDK pick up required profile information from .NET credentials file stored in AWS SDK and connect it to AWS service. Please refer to this link for more information here.

 

Setup Logger for .NET Core API

 

Once AWS account setup is done through SDK, we need to add required configurations in Startup.cs file to start capturing log information and to let .NET Core API know that we are going to send log messages to AWS Cloudwatch. Add the below lines of code to your Startup.cs file.

public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
            int.TryParse(Configuration.GetSection("Serilog").GetSection("LogLevel").Value, out int level);
            int.TryParse(Configuration.GetSection("Serilog").GetSection("RententionPolicy").Value, out int retentionConfig);
            var logLevel = typeof(LogEventLevel).IsEnumDefined(level) ?
                                (LogEventLevel)level : LogEventLevel.Error;
            var retentionPolicy = typeof(LogGroupRetentionPolicy).IsEnumDefined(retentionConfig) ?
                              (LogGroupRetentionPolicy)retentionConfig : LogGroupRetentionPolicy.OneWeek;
            var region = RegionEndpoint.GetBySystemName(Configuration.GetSection("Serilog").GetSection("Region").Value);
            var levelSwitch = new LoggingLevelSwitch();
            levelSwitch.MinimumLevel = logLevel;
            // customer formatter
            var formatter = new CustomLogFormatter();
            var options = new CloudWatchSinkOptions
            {
                // the name of the CloudWatch Log group from config
                LogGroupName = Configuration.GetSection("Serilog").GetSection("LogGroup").Value,
                // the main formatter of the log event
                TextFormatter = formatter,
                // other defaults
                MinimumLogEventLevel = logLevel,
                BatchSizeLimit = 100,
                QueueSizeLimit = 10000,
                Period = TimeSpan.FromSeconds(10),
                CreateLogGroup = true,
                LogStreamNameProvider = new DefaultLogStreamProvider(),
                RetryAttempts = 5,
                LogGroupRetentionPolicy = retentionPolicy
            };
            // setup AWS CloudWatch client
            var client = new AmazonCloudWatchLogsClient(region);
            // Attach the sink to the logger configuration
            Log.Logger = new LoggerConfiguration()
                .WriteTo.Logger(l1 => l1
                    .MinimumLevel.ControlledBy(levelSwitch)
                    .WriteTo.AmazonCloudWatch(options, client))
              .CreateLogger();
        }

Add a CustomFormatter class to your application and add the below lines of code. This will help you format your log messages to your desired view. You can modify it based on your requirement.

public class CustomLogFormatter : ITextFormatter
   {
       public void Format(LogEvent logEvent, TextWriter output)
       {
           output.Write("Timestamp - {0} | Level - {1} | Message {2} {3} {4}", logEvent.Timestamp, logEvent.Level, logEvent.MessageTemplate, JsonConvert.SerializeObject(logEvent.Properties), output.NewLine);
           if (logEvent.Exception != null)
           {
               output.Write("Exception - {0}", logEvent.Exception);
           }
       }
   }

I have configured all the necessary details related to logging such as Log Level, Retention Policy, Log Bucket Name from appSettings.json file. Below are the set of configurations that need to be added to the appSettings.json file.

"Serilog": {
    "LogLevel": 4,
    "LogGroup": "SSOAPI_Development",
    "CustomLogGroup": "SSOAPI_DevCustomInfo",
    "Region": "eu-central-1",
    "RententionPolicy": 5
  }

Test Your Implementation By Logging from Code

 

Now is the time to test our logging setup by adding logs from any part of the application. Add the below lines to the values controller GET method since it is the easiest part to test.

// GET api/values

[HttpGet]

public ActionResult<IEnumerable<string>> Get()

{
      Log.Information("This is logged from Values Controller Get Method");

      return new string[] { "value1", "value2" };
}

Now run the application and you will be able to see logs getting saved into AWS Cloudwatch account under the logGroup created from our application.

 

Custom Logging using Serilog Properties

 

You can add only the required/necessary logs from any log level all the time to a custom log bucket under AWS cloudwatch. This approach will be useful to keep track of any critical functionalities such as user payment gateways as such. We need to add another cloudwatch option configuration and use Serilog filter option to gather custom logs based on property which you will include while logging messages. Add the below lines of code along with previous logging setup.

var customOptions = new CloudWatchSinkOptions
            {
                // the name of the CloudWatch Log group for logging
                LogGroupName = Configuration.GetSection("Serilog").GetSection("CustomLogGroup").Value,

                // the main formatter of the log event
                TextFormatter = formatter,

                // other defaults defaults
                MinimumLogEventLevel = LogEventLevel.Information,
                BatchSizeLimit = 100,
                QueueSizeLimit = 10000,
                Period = TimeSpan.FromSeconds(10),
                CreateLogGroup = true,
                LogStreamNameProvider = new DefaultLogStreamProvider(),
                RetryAttempts = 5,
                LogGroupRetentionPolicy = retentionPolicy
            };
            // setup AWS CloudWatch client
            var client = new AmazonCloudWatchLogsClient(region);
            // Attach the sink to the logger configuration
            Log.Logger = new LoggerConfiguration()
                .WriteTo.Logger(l1 => l1
                    .MinimumLevel.ControlledBy(levelSwitch)
                    .WriteTo.AmazonCloudWatch(options, client))
              .WriteTo.Logger(l2 => l2
                    .Filter.ByIncludingOnly(Matching.WithProperty("custom"))
                    .WriteTo.AmazonCloudWatch(customOptions, client))
              .CreateLogger();

While adding log messages from the controller, you need to add a custom property to let it get logged into the custom log bucket. Add the below code to your values controller GET method.

// GET api/values
        [HttpGet]
        public ActionResult<IEnumerable<string>> Get()
        {
            Log.ForContext("Custom", "").Information("This is logged from Values Controller Get Method");
            Log.Error("This is an error message from ValuesController - Not logged to Custom log group");
            return new string[] { "value1", "value2" };
        }

After running this code, you will see only logs with custom properties added will be logged to Custom Log Group.

 

Conclusion

 

Hopefully after going through this blog you will have a clear understanding about how to send logs to AWS Cloudwatch and integrate it with .NET Core API using serilog logging framework. Please feel free to post your questions related to this blog in the comments section. Also point out if something you feel important is missing in this.

 

Thanks for reading !

 

Happy Coding