Observability in ASP.NET Core with OpenTelemetry & Azure Monitor

Table of Contents

1. What is OpenTelemetry?
2. Setting Up OpenTelemetry in ASP.NET Core
   • Metrics
   • Logging
   • Tracing
   • Exporting to Azure Monitor
3. Setting Up Azure Application Insights, Alerts, and Monitoring
4. Viewing Custom Metrics in Azure Monitor
5. Query Custom Metrics with KQL
6. Deep Dive into Telemetry Signals
   • Distributed Tracing
   • Metrics
   • Logs
7. Advanced Telemetry Patterns
8. Conclusion

What is OpenTelemetry?

OpenTelemetry is a vendor-neutral instrumentation framework for generating, collecting, processing, and exporting telemetry data such as traces, metrics, and logs. It provides unified APIs and SDKs to help developers monitor applications effectively.

Setting Up OpenTelemetry in ASP.NET Core

To start, you need to install the required NuGet packages:

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Instrumentation.Runtime
dotnet add package Azure.Monitor.OpenTelemetry.Exporter

Metrics

Metrics help track the behavior and performance of an application over time.

var meter = new Meter("MyCustomMeter", "1.0");
var counter = meter.CreateCounter<int>("my_custom_counter");
counter.Add(1, new KeyValuePair<string, object?>("path", Request.HttpContext.Request));

You can also use Histogram to measure latency:

var sw=Stopwatch.StartNew();
var meter = new Meter("MyCustomMeter", "1.0");
var requestDuration = meter.CreateHistogram<double>("myApp.request_duration", unit: "ms");
  var result= Enumerable.Range(1, 5).Select(index => new WeatherForecast
  {
      Date = DateOnly.FromDateTime(DateTime.Now.AddDays(index)),
      TemperatureC = Random.Shared.Next(-20, 55),
      Summary = Summaries[Random.Shared.Next(Summaries.Length)]
  })
  .ToArray();
 sw.Stop();
 requestDuration.Record(sw.ElapsedMilliseconds, new KeyValuePair<string, object?>("path", Request.HttpContext.Request));

Logging

To log-structured data:

_logger.LogInformation("Request duration: {@Duration} ms", sw.ElapsedMilliseconds);

If you use @ before the object name, it ensures structured logging, preserving the shape of the object.

Tracing

To enable distributed tracing:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
    {
        tracerProviderBuilder
            .AddSource("opentelemetryverify")
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddConsoleExporter()
            .AddAzureMonitorTraceExporter(o =>
            {
                // Set the connection string for Azure Monitor
                o.ConnectionString = connectionString;
            });
    });

Exporting to Azure Monitor

To send telemetry data to Azure Monitor:

1. Create an Application Insights resource in Azure.
2. Copy the connection string from the Overview blade.
3. In appsettings.json or secrets:

"APPLICATIONINSIGHTS_CONNECTION_STRING": "InstrumentationKey=..."

Configure exporters:

builder.Services.AddOpenTelemetryMetrics(builder =>
{
    builder.AddAspNetCoreInstrumentation()
           .AddHttpClientInstrumentation()
           .AddAzureMonitorMetricExporter(options =>
           {
               options.ConnectionString = configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"];
           });
});

Setting Up Azure Application Insights, Alerts, and Monitoring

1. Application Insights Setup

• Go to Azure Portal.
• Create a new Application Insights resource.
• Choose the corresponding resource group and region.
• Copy the Connection String after creation.

2. Connect Your App

Paste the connection string into your application’s configuration and use Azure Monitor exporters in the OpenTelemetry setup.

3. Custom Metrics

• Navigate to your Application Insights resource.
• Go to Metrics > Add custom metric namespace and filter by name (e.g., myapp.request_count).

4. Alerts Configuration

• Go to Alerts > + New Alert Rule.
• Set:
  • Scope: Application Insights resource.
  • Condition: Metric signal (e.g., myapp.request_count > 1000).
  • Action Group: Create or choose email/SMS/webhook notifications.
  • Alert Rule Details: Set name and severity.

5. Workbooks and Dashboards

• Use Workbooks in Application Insights for rich dashboards.
• Customize time ranges and add charts and KQL-based visualizations.

6. Monitoring with Log Analytics

You can write custom queries for logs and metrics using KQL (Kusto Query Language):

customMetrics
| where name == "myapp.request_count"
| summarize count() by bin(timestamp, 1h), name
| render timechart

Viewing Custom Metrics in Azure Monitor

To view metrics:

• Go to your Application Insights resource.
• Click on Metrics.
• Choose the custom metric name like myapp.request_count.
• Apply dimensions (e.g., path) to analyze traffic patterns.

Query Custom Metrics with KQL

In Azure Monitor Logs:

customMetrics
| where name == "myapp.request_count"
| summarize count() by bin(timestamp, 1h), name
| render timechart

Deep Dive into Telemetry Signals

OpenTelemetry provides three core types of signals for observability: Traces, Metrics, and Logs. Each serves a different purpose, but together, they provide a complete view of your application’s health and performance.

🧵 Distributed Tracing

Purpose: Traces help you understand the journey of a request as it flows through various components and services in a distributed system.

Key Concepts

• Span: A single unit of work, e.g., an HTTP request or a database call.
• Trace: A tree of spans representing a complete workflow or transaction.
• Attributes: Key-value pairs to describe a span (e.g., URL, status code).
• Context Propagation: Allows you to pass trace information across process boundaries.

Example Use Case

You’re debugging why a request is slow. A trace shows that while your API responded quickly, the delay was due to a downstream database call taking too long.

Code

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
    {
        tracerProviderBuilder
        .AddSource("opentelemetryverify")
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddConsoleExporter()
            .AddAzureMonitorTraceExporter(o =>
            {
                o.ConnectionString = connectionString; // Set the connection string for Azure Monitor
            });
    })

Metrics

Purpose: Metrics provide quantitative information over time to measure application performance and health.

Key Metric Instruments

• Counter: Monotonically increasing value (e.g., number of requests).
• UpDownCounter: Can increase or decrease (e.g., active connections).
• Histogram: Measures distribution of values (e.g., request durations).

Example Use Case

You want to monitor request rates and latency. Metrics help you visualize traffic patterns and performance trends over time.

var sw=Stopwatch.StartNew();
var meter = new Meter("MyCustomMeter", "1.0");
var requestDuration = meter.CreateHistogram<double>("myApp.request_duration", unit: "ms");
var counter = meter.CreateCounter<int>("my_custom_counter");
counter.Add(1, new KeyValuePair<string, object?>("path", Request.HttpContext.Request));
 
var dbConnection = meter.CreateUpDownCounter<int>("dbconnection_count");
dbConnection.Add(1);
var result= Enumerable.Range(1, 5).Select(index => new WeatherForecast
{
    Date = DateOnly.FromDateTime(DateTime.Now.AddDays(index)),
    TemperatureC = Random.Shared.Next(-20, 55),
    Summary = Summaries[Random.Shared.Next(Summaries.Length)]
})
.ToArray();
dbConnection.Add(-1);
sw.Stop();
requestDuration.Record(sw.ElapsedMilliseconds, new KeyValuePair<string, object?>("path", Request.HttpContext.Request));
_logger.LogInformation("Request duration: {Duration} ms", sw.ElapsedMilliseconds);

WithMetrics(metrics =>
{
    metrics
        .AddAspNetCoreInstrumentation()
        .AddRuntimeInstrumentation()
        .AddHttpClientInstrumentation()
        .AddMeter("MyCustomMeter") // Optional: For custom metrics
        .AddConsoleExporter()     // Export metrics to console
        .AddAzureMonitorMetricExporter(o =>
        {
            o.ConnectionString = connectionString; // Set the connection string for Azure Monitor
        });
});

Logs

Purpose: Logs provide contextual, human-readable messages about events or errors occurring in your app.

Structured Logging

Logging with structured data (objects, dictionaries) makes it easier to search and analyze logs in Azure Monitor or any observability platform.

Example Use Case

You log a response payload for auditing or an exception trace for troubleshooting. Structured logs help you query specific fields like status codes or response times.

Code

builder.Logging.ClearProviders();
builder.Logging.AddOpenTelemetry(x =>
{
    x.SetResourceBuilder(ResourceBuilder.CreateDefault()
        .AddService("OpenTelemetryVerify", serviceVersion: "1.0.0")); // Set the service name and version
    x.IncludeFormattedMessage = true; // Include formatted message in logs
    x.IncludeScopes = true; // Include scopes in logs
    x.ParseStateValues = true; // Parse state values in logs
    x.AddConsoleExporter(); // Export logs to console
    x.AddAzureMonitorLogExporter(o =>
    {
        o.ConnectionString = connectionString; // Set the connection string for Azure Monitor
    });
});

_logger.LogInformation("Response of the weather forecast:{@result}", data);

Bonus Tip

To view structured logs correctly in Azure Monitor, ensure your logger supports scopes and enrichment (e.g., with Serilog or OpenTelemetry exporters).

Advanced Telemetry Patterns

1. Correlation Across Services

Correlate telemetry data using TraceId and SpanId across services to track a request’s flow. This enables full end-to-end visibility in distributed systems.

2. Redaction and Enrichment

Use processors to add metadata like userId or region and to remove or mask sensitive fields before telemetry is exported.

3. Sampling Strategies

Reduce data volume by configuring sampling. Use TraceIdRatioBasedSampler(0.1) to export 10% of traces while keeping error traces always.

4. Span Events and Annotations

Add custom ActivityEvents to spans to track the lifecycle stages of an operation (e.g., “CacheMiss”, “PaymentValidated”).

5. Custom Instrumentation

Instrument your domain-specific workflows like Checkout or VerifyOTP with custom spans or metrics. This gives business-level observability.

6. Error and Exception Tracking

Auto-capture exceptions in spans and logs. Use AddExceptionInstrumentation() to bind exception telemetry with traces.

7. Real-Time Alerting

Leverage Azure Monitor alerts based on custom metrics or logs. Use KQL queries to define thresholds and trigger notifications.

8. Dashboards and SLO Monitoring

Build dashboards using Azure Workbooks to monitor SLIs (e.g., error rate < 1%) and SLOs (e.g., 95th percentile latency < 300ms).

9. Cold Start and Warm Path Differentiation

Track cold start delays in serverless or autoscaling systems by tagging spans with startup metadata.

10. Multi-Exporter Strategy

Send telemetry to multiple backends (e.g., Azure Monitor and local file) for redundancy or migration testing.

Conclusion

By integrating OpenTelemetry in ASP.NET Core and exporting to Azure Monitor, you gain deep insights into your application using logs, metrics, and traces. This setup ensures robust observability, enabling proactive monitoring and faster issue resolution.

With advanced telemetry patterns, you can enrich and extend your observability to meet the demands of modern, distributed, and dynamic systems.