Diagnosing Microservices Performance with Azure Application Insights and OpenTelemetry

Introduction

Implementing distributed tracing with Azure Application Insights or OpenTelemetry to diagnose performance issues is a crucial aspect of monitoring and troubleshooting applications in a distributed system. Below, I'll provide a step-by-step example with a real-world scenario using Azure Application Insights and .NET Core.

Scenario. You have a microservices-based e-commerce application that consists of multiple services, such as a product catalog service, a shopping cart service, and an order processing service. You want to trace user requests as they flow through these services to diagnose performance bottlenecks.

Prerequisites

1. Visual Studio or Visual Studio Code
2. .NET Core SDK installed
3. Azure subscription (for Azure Application Insights)

Step 1. Create a New .NET Core Web API Project

Create a new .NET Core Web API project for one of your microservices. You can repeat this step for each microservice:

dotnet new webapi -n ProductCatalogService

Step 2. Install the Required Packages

In each microservice project, install the necessary NuGet packages:

dotnet add package Microsoft.ApplicationInsights.AspNetCore
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Exporter.ApplicationInsights

Step 3. Configure OpenTelemetry and Application Insights

In the Startup.cs file of your microservice project, configure OpenTelemetry and Application Insights:

using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using OpenTelemetry.Exporter.ApplicationInsights;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

Author: Sardar Mudassar Ali Khan
namespace ProductCatalogService
{
    public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        public void ConfigureServices(IServiceCollection services)
        {
            services.AddControllers();

            services.AddOpenTelemetryTracing(builder =>
            {
                builder
                    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("ProductCatalogService"))
                    .AddAspNetCoreInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddConsoleExporter()
                    .AddApplicationInsightsExporter(options =>
                    {
                        options.ConnectionString = Configuration["ApplicationInsights:InstrumentationKey"];
                    });
            });
        }

        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }

            app.UseRouting();
            app.UseEndpoints(endpoints =>
            {
                endpoints.MapControllers();
            });
        }
    }
}

Step 4. Configure Application Insights

In the appsettings.json file of each microservice project, add your Application Insights Instrumentation Key:

{
  "ApplicationInsights": {
    "InstrumentationKey": "YOUR_INSTRUMENTATION_KEY"
  }
}

Step 5. Add Tracing to Your Code

In your microservice controller or service methods, you can add custom tracing to trace specific operations. For example:

using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;
using System;
using System.Diagnostics;

Author: Sardar Mudassar Ali Khan
namespace ProductCatalogService.Controllers
{
    [ApiController]
    [Route("api/[controller]")]
    public class ProductsController : ControllerBase
    {
        private readonly ILogger<ProductsController> _logger;

        public ProductsController(ILogger<ProductsController> logger)
        {
            _logger = logger;
        }

        [HttpGet("{id}")]
        public IActionResult GetProduct(int id)
        {
            var operationName = "ProductCatalogService.GetProduct";
            var activity = new Activity(operationName).Start();

            try
            {
                // Your code to fetch the product by ID

                activity.SetStatus(Status.Ok);
                return Ok("Product details here");
            }
            catch (Exception ex)
            {
                activity.SetStatus(Status.Error);
                _logger.LogError(ex, "Error while fetching the product.");
                return StatusCode(500, "Internal Server Error");
            }
            finally
            {
                activity.Stop();
            }
        }
    }
}

Step 6. Deploy and Monitor

Deploy your microservices to Azure or your preferred hosting environment. Ensure that the Application Insights Instrumentation Key is correctly configured in the appsettings.json of each microservice.

As requests flow through your microservices, Azure Application Insights will collect and display traces, allowing you to diagnose performance issues and monitor the entire flow of user requests.

By following these steps, you'll have implemented distributed tracing using Azure Application Insights and OpenTelemetry in your microservices-based application to diagnose and troubleshoot performance issues effectively. Repeat these steps for each microservice in your application.

Conclusion

In this example, we've walked through the process of implementing distributed tracing using Azure Application Insights and OpenTelemetry in a real-world scenario for a microservices-based e-commerce application. Here's a brief summary of the key points:

1. Create Microservices: We started by creating .NET Core Web API projects for each microservice in our application. These microservices represent different components like product catalogs, shopping carts, and order processing.
2. Install Packages: We added the necessary NuGet packages to our microservice projects. These packages include Microsoft.ApplicationInsights.AspNetCore for Application Insights and OpenTelemetry packages for tracing.
3. Configure OpenTelemetry and Application Insights: In the Startup.cs file of each microservice, we configured OpenTelemetry to trace incoming requests and added an Application Insights exporter to send trace data to Azure Application Insights. We also set up resource information and other instrumentation as needed.
4. Configure Application Insights: We added the Application Insights Instrumentation Key to the appsettings.json file of each microservice. This key is essential for sending trace data to Azure Application Insights.
5. Add Tracing to Code: Within the microservice code, we demonstrated how to add custom tracing using OpenTelemetry. We created an activity for tracing specific operations and set their status based on the success or failure of those operations.
6. Deploy and Monitor: After deploying the microservices to your chosen hosting environment, you can use Azure Application Insights to monitor and analyze the traces and performance data. This allows you to diagnose performance issues, track the flow of user requests, and gain insights into your application's behavior in a distributed system.

By following these steps, you can effectively implement distributed tracing in your microservices architecture to improve performance analysis and debugging, ensuring the smooth operation of your complex, distributed applications. Azure Application Insights, in combination with OpenTelemetry, provides powerful tools for monitoring and diagnosing issues in your applications.