惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

TaoSecurity Blog
TaoSecurity Blog
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
F
Fortinet All Blogs
Cisco Talos Blog
Cisco Talos Blog
D
Darknet – Hacking Tools, Hacker News & Cyber Security
S
Secure Thoughts
美团技术团队
雷峰网
雷峰网
Hugging Face - Blog
Hugging Face - Blog
博客园_首页
C
CXSECURITY Database RSS Feed - CXSecurity.com
Engineering at Meta
Engineering at Meta
人人都是产品经理
人人都是产品经理
月光博客
月光博客
T
Tor Project blog
P
Privacy & Cybersecurity Law Blog
Recorded Future
Recorded Future
I
Intezer
博客园 - 【当耐特】
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
GbyAI
GbyAI
罗磊的独立博客
V
V2EX
Google DeepMind News
Google DeepMind News
D
DataBreaches.Net
Last Week in AI
Last Week in AI
T
Tailwind CSS Blog
www.infosecurity-magazine.com
www.infosecurity-magazine.com
A
About on SuperTechFans
Scott Helme
Scott Helme
Vercel News
Vercel News
Spread Privacy
Spread Privacy
T
Threat Research - Cisco Blogs
Recent Announcements
Recent Announcements
Hacker News: Ask HN
Hacker News: Ask HN
C
CERT Recently Published Vulnerability Notes
G
Google Developers Blog
B
Blog
博客园 - 叶小钗
WordPress大学
WordPress大学
博客园 - 聂微东
Threat Intelligence Blog | Flashpoint
Threat Intelligence Blog | Flashpoint
Jina AI
Jina AI
IT之家
IT之家
C
Cybersecurity and Infrastructure Security Agency CISA
P
Palo Alto Networks Blog
小众软件
小众软件
博客园 - Franky
Microsoft Azure Blog
Microsoft Azure Blog
AWS News Blog
AWS News Blog

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant Common SOC 2 Failures (Real World) Stop Vibe-Checking Your AI App: A Practical Guide to Evals How to Use SonarQube and SonarScanner Locally to Level Up Your Code Quality Your Next To-Do App Is Dead — I Replaced Mine with an OpenClaw AI Sign a Nostr event in 60 lines of Python using coincurve — no nostr-sdk, no nbxplorer, no rust toolchain ITGC Audit Explained Like You’re in Big 4 Patch Tuesday abril 2026: Microsoft parcha 163 vulnerabilidades y un zero-day en SharePoint Stop scraping everything: a better way to track competitor price changes Listing on MCPize + the Official MCP Registry while routing payments OUTSIDE the marketplace — how I kept 100% of my x402 revenue Building an AI-Powered Risk Intelligence System Using Serverless Architecture Why We Ripped Function Overloading Out of Our AI Toolchain Testing AI-Generated Code: How to Actually Know If It Works SaaS Churn Is Killing Your Business. Here Is What to Do About It (Without a Support Team) The Speed of AI Is No Longer Linear - And Self-Improving Models Are Why How to Implement RBAC for MCP Tools: A Practical Guide for Engineering Teams From Standard Quote to Persuasive Proposal: AI Automation for Arborists I built a CLI that scaffolds complete multi-tenant SaaS apps Axios CVE-2025–62718: The Silent SSRF Bug That Could Be Hiding in Your Node.js App Right Now The dashboard that ended our friendship Data Pipelines Explained Simply (and How to Build Them with Python) The Hidden Cost of AI Systems Nobody Talks About. undefined vs undeclared, and how typeof behaves Switching from file-based jobs to NATS/Kafka in Rust without changing code io_uring Adventures: Rust Servers That Love Syscalls Why Agentic AI is Killing the Traditional Database The POUR principles of web accessibility for developers and designers Quantum Neural Network 3D — A Deep Dive into Interactive WebGL Visualization How To Install Caveman In Codex On macOS And Windows Automation Pipeline Reliability: Why Your Workflow Breaks When Nobody Is Watching I Built an 'Open World' AI Coding Agent — It Works From ANY Folder From Freelancing to Product: A Tech Service Company's SaaS Transformation China's AI Giants: Adding Tencent Hunyuan & ByteDance Doubao to AI University (74 Providers) On the Vibe Coders and Their Lies clerk: Auto-Summarize Your Claude Code Sessions AI Weekly — 2026/04/10–04/17 | The Model Lockdown Is Here, but the Toolchain Is the Real Battleground AI 週報 — 2026/04/10–2026/04/17 模型封鎖潮來了,但工具鏈才是真戰場 Maybe this is how Open-Source apps are born... 🚀 Fine-Tune LLMs with LoRA and QLoRA: 2026 Guide tRPC v11 + Next.js App Router: End-to-End Type Safety Without the Boilerplate ShadCN UI in 2026: Why I Stopped Installing Component Libraries and Started Owning My Components SaaS Billing in React Server Components: Stripe + Supabase Without a Single `useEffect` Join our DEV Weekend Challenge — $1,000 in Prizes Across TEN winners! Submissions Due April 20 at 6:59 AM UTC. Implementing FSRS Spaced Repetition in Flutter + Supabase — Adding Memory Science to an AI Learning App "I Texted My Localhost From the Train — Claude Code Fixed the Bug Before I Got Home" I Built a Sales Prep AI and It Went Deeper Than Expected Design to Code #2: One JSON, Eleven Outputs Solving the 100M-Row Problem: A Summary Table Pattern for High-Volume Push Notification Logs Flutter Web With Wasm: What Actually Changes For Developers I Built 50 Royalty-Free Soundtracks for My Side Project in a Weekend Using AI Music Generation The Vibe Coding Security Checklist: 7 Things to Check Before You Ship Stop Letting Googlebot Guess Fix Your React App's SEO Right Desconstruindo o Streaming do LinkedIn: Como Criar um Engine de Extração de Vídeo de Alta Performance com HLS e FFmpeg (EDA Part-1) EDA (Exploratory Data Analysis) Explained With Real Life — Why Looking at Your Data Is the Most Important Step in Machine Learning Brand Relationship Management at Scale: Our 4-Touch Outreach System for 200+ Brands Why String.fromEnvironment() Might Return an Empty String in Dart JGuardrails 1.0.0 — Hardening Java LLM Apps Against Jailbreaks, Toxicity, and Prompt Injection Plan and Schedule a Full Week of Threads Content From One Claude Conversation Coding Cat Oran Ep3, Five Tables Changed Everything Updated: BFF Pattern I'm done watching freelancers get buried by 200 proposals. So I'm building the alternative. This is my first post BFS Algorithm in Java Step by Step Tutorial with Examples Tracking LLM Pricing Monthly: An Open Dataset for 22 AI Models How We Measure Content ROI on a Comparison Site: Revenue Attribution Without Perfect Data Introducing Nova AI Ops: The AI-Native Operating System for SRE Teams I built a free desktop video downloader for Windows — Grabbit How Talkie OCR Helps Vision-Impaired & Dyslexic Users Read the World Around Them VRCFaceTracking安装和iPhone面捕配置教程,有bug Even CrowdStrike Can't See Your Agents The Automation Gold Rush: What n8n Workflows and Claude Are Opening Up for Developers Right Now
OpenTelemetry in .NET Microservices: A Practical Guide to Observable Distributed Systems
Adrián López · 2026-04-30 · via DEV Community

You've deployed your fifth microservice. Requests flow through an API gateway, hit an order service, fan out to inventory and payment, and eventually land in a notification pipeline. Everything works — until it doesn't. A customer reports that checkout takes 12 seconds. Your logs show nothing unusual. Your metrics say average latency is fine. But somewhere in that chain of services, something is dragging, and you have no idea where.

This is the observability gap that kills microservice architectures. Monoliths gave us stack traces. Microservices gave us distributed amnesia.

OpenTelemetry exists to close that gap — and in the .NET ecosystem, it's matured from an experimental curiosity into the de facto standard for instrumentation. This post covers everything you need to go from zero to production-grade observability in your .NET microservices, with concrete C# examples, real debugging scenarios, and deployment patterns for Azure.


Why Observability Is Hard in Microservices

If you've operated distributed systems, you already know the pain points:

  • Distributed tracing is non-trivial. A single user request might touch 8 services, 3 databases, and 2 message queues. Correlating that flow requires propagating context across HTTP calls, gRPC channels, and async messaging — consistently.
  • Logs are scattered. Each service writes its own logs to its own sink. Reconstructing a timeline means grepping across dozens of streams and mentally stitching timestamps together.
  • Metrics lack context. Knowing that P99 latency spiked on the payment service tells you where to look, but not why. Was it a slow database query? A downstream timeout? A garbage collection pause?
  • Vendor lock-in is real. You instrumented everything for Datadog, and now leadership wants to move to Azure Monitor. Ripping out and replacing telemetry code across 20 services is not how anyone wants to spend a quarter.

OpenTelemetry addresses all of these problems with a single, vendor-neutral instrumentation framework.


What Is OpenTelemetry?

OpenTelemetry (OTel) is a CNCF project that provides a unified set of APIs, SDKs, and conventions for generating, collecting, and exporting telemetry data — specifically traces, metrics, and logs.

The key value proposition is decoupling instrumentation from backends. You instrument your code once with OTel APIs, and then choose where to send the data: Azure Monitor, Jaeger, Grafana, Coralogix, Datadog, Honeycomb, or any OTLP-compatible backend. Swap backends without touching application code.

OpenTelemetry in the .NET Ecosystem

The .NET OTel story is particularly strong because it builds on primitives that already exist in the runtime:

  • System.Diagnostics.Activity is the native .NET distributed tracing API, and OTel's .NET SDK wraps it directly — no parallel abstraction layer.
  • System.Diagnostics.Metrics (introduced in .NET 6) provides the metrics API that OTel hooks into.
  • ILogger<T> integration means your existing structured logging flows naturally into OTel's log pipeline.

This isn't a foreign framework bolted onto .NET. It's a thin orchestration layer over APIs you're probably already using.


Core Concepts

Traces, Spans, and Context Propagation

A trace represents the entire lifecycle of a request as it flows through your system. Each trace is composed of spans — individual units of work. A span records an operation's name, start/end timestamps, status, and key-value attributes.

Spans form a tree: a root span (e.g., the incoming HTTP request) has child spans (e.g., a database query, an outbound HTTP call), which may themselves have children. The parent-child relationship is established via context propagation — passing a trace ID and span ID across service boundaries.

In .NET, context propagation over HTTP is handled automatically when you use HttpClient with the OTel SDK configured. The SDK injects W3C traceparent headers on outbound calls and extracts them on inbound requests. For messaging scenarios (RabbitMQ, Azure Service Bus), you'll need to propagate context manually through message headers.

[API Gateway] ──► [Order Service] ──► [Payment Service]
  span: gateway       span: order         span: payment
  traceId: abc123     traceId: abc123     traceId: abc123
  spanId: 001         spanId: 002         spanId: 003
                      parentId: 001       parentId: 002

Enter fullscreen mode Exit fullscreen mode

Metrics and Logs

Metrics capture aggregated, numerical measurements over time: request counts, histogram of response times, gauge of active connections. Unlike traces (which capture individual requests), metrics are cheap at scale because they're pre-aggregated.

Logs provide the richest contextual detail — exception stack traces, business-level events, debug output — but are the most expensive signal to store and query.

The three signals complement each other:

  • Metrics alert you that something is wrong (latency spike).
  • Traces show you where in the call chain the problem is.
  • Logs tell you why (the specific error, the slow query text, the unexpected null).

OpenTelemetry correlates all three by attaching trace and span IDs to log records and metric exemplars, so you can jump from a metric anomaly to the specific traces that caused it.

Instrumentation: Automatic vs. Manual

Automatic instrumentation (also called library instrumentation) is provided by community packages that hook into common libraries. For .NET, this includes:

  • OpenTelemetry.Instrumentation.AspNetCore — traces for incoming HTTP requests
  • OpenTelemetry.Instrumentation.HttpClient — traces for outgoing HTTP calls
  • OpenTelemetry.Instrumentation.SqlClient — traces for SQL queries
  • OpenTelemetry.Instrumentation.EntityFrameworkCore — traces for EF Core operations
  • OpenTelemetry.Instrumentation.GrpcNetClient — traces for gRPC calls

These packages generate spans automatically with no code changes beyond registration.

Manual instrumentation is what you add for your own business logic — things automatic instrumentation can't know about. Processing a payment? Validating inventory? Calling a third-party API with retry logic? You create custom spans and record custom metrics for these.

Exporters and Backends

An exporter is the component that sends telemetry data to a backend. Common exporters in .NET:

  • OTLP (OpenTelemetry Protocol): The standard wire format. Works with the OpenTelemetry Collector, Grafana Tempo, Jaeger, and most modern backends.
  • Azure Monitor: Sends data to Application Insights / Azure Monitor via the Azure.Monitor.OpenTelemetry.Exporter package.
  • Console: Writes telemetry to stdout — useful for local development and debugging.
  • Zipkin / Jaeger (direct): Native exporters for these backends, though OTLP is generally preferred now.

Hands-On: Setting Up OpenTelemetry in ASP.NET Core

Step 1: Install the Packages

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.HttpClient
dotnet add package OpenTelemetry.Instrumentation.SqlClient
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Exporter.Console
dotnet add package Azure.Monitor.OpenTelemetry.Exporter

Enter fullscreen mode Exit fullscreen mode

Step 2: Configure the SDK in Program.cs

using OpenTelemetry;
using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;
using OpenTelemetry.Logs;
using OpenTelemetry.Resources;

var builder = WebApplication.CreateBuilder(args);

// Define a shared resource describing this service
var serviceName = "OrderService";
var serviceVersion = "1.0.0";

var resourceBuilder = ResourceBuilder.CreateDefault()
    .AddService(serviceName: serviceName, serviceVersion: serviceVersion)
    .AddAttributes(new Dictionary<string, object>
    {
        ["deployment.environment"] = builder.Environment.EnvironmentName,
        ["host.name"] = Environment.MachineName
    });

// Configure Tracing
builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService(serviceName, serviceVersion: serviceVersion))
    .WithTracing(tracing =>
    {
        tracing
            .SetResourceBuilder(resourceBuilder)
            .AddAspNetCoreInstrumentation(opts =>
            {
                // Filter out health check noise
                opts.Filter = ctx => !ctx.Request.Path.StartsWithSegments("/health");
            })
            .AddHttpClientInstrumentation()
            .AddSqlClientInstrumentation(opts =>
            {
                opts.SetDbStatementForText = true; // Capture SQL text (careful in prod)
                opts.RecordException = true;
            })
            .AddSource("OrderService.Activities") // Custom ActivitySource
            .AddOtlpExporter(opts =>
            {
                opts.Endpoint = new Uri("http://otel-collector:4317");
            });

        // Add Azure Monitor in deployed environments
        if (!builder.Environment.IsDevelopment())
        {
            tracing.AddAzureMonitorTraceExporter(opts =>
            {
                opts.ConnectionString = builder.Configuration
                    .GetValue<string>("AzureMonitor:ConnectionString");
            });
        }
        else
        {
            tracing.AddConsoleExporter();
        }
    })
    .WithMetrics(metrics =>
    {
        metrics
            .SetResourceBuilder(resourceBuilder)
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddMeter("OrderService.Metrics") // Custom Meter
            .AddOtlpExporter()
            .AddPrometheusExporter(); // /metrics endpoint for scraping
    });

// Configure Logging
builder.Logging.AddOpenTelemetry(logging =>
{
    logging
        .SetResourceBuilder(resourceBuilder)
        .AddOtlpExporter()
        .IncludeFormattedMessage = true;
});

var app = builder.Build();
app.MapPrometheusScrapingEndpoint(); // Expose /metrics
// ... rest of pipeline
app.Run();

Enter fullscreen mode Exit fullscreen mode

Step 3: Add Custom Instrumentation

using System.Diagnostics;
using System.Diagnostics.Metrics;

public class OrderProcessingService
{
    // ActivitySource is the .NET equivalent of a Tracer
    private static readonly ActivitySource ActivitySource =
        new("OrderService.Activities", "1.0.0");

    // Custom metrics
    private static readonly Meter Meter = new("OrderService.Metrics", "1.0.0");
    private static readonly Counter<long> OrdersProcessed =
        Meter.CreateCounter<long>("orders.processed", "orders", "Total orders processed");
    private static readonly Histogram<double> OrderProcessingDuration =
        Meter.CreateHistogram<double>("orders.processing_duration", "ms",
            "Time to process an order");

    private readonly ILogger<OrderProcessingService> _logger;
    private readonly PaymentClient _paymentClient;
    private readonly InventoryClient _inventoryClient;

    public OrderProcessingService(
        ILogger<OrderProcessingService> logger,
        PaymentClient paymentClient,
        InventoryClient inventoryClient)
    {
        _logger = logger;
        _paymentClient = paymentClient;
        _inventoryClient = inventoryClient;
    }

    public async Task<OrderResult> ProcessOrderAsync(Order order)
    {
        // Start a custom span
        using var activity = ActivitySource.StartActivity(
            "ProcessOrder",
            ActivityKind.Internal);

        // Add structured attributes to the span
        activity?.SetTag("order.id", order.Id);
        activity?.SetTag("order.item_count", order.Items.Count);
        activity?.SetTag("order.customer_id", order.CustomerId);

        var stopwatch = Stopwatch.StartNew();

        try
        {
            // Validate inventory — this creates a child span automatically
            // because HttpClient instrumentation is active
            var inventoryResult = await _inventoryClient
                .CheckAvailabilityAsync(order.Items);

            if (!inventoryResult.AllAvailable)
            {
                activity?.SetStatus(ActivityStatusCode.Error, "Insufficient inventory");
                activity?.AddEvent(new ActivityEvent("inventory.insufficient",
                    tags: new ActivityTagsCollection
                    {
                        { "unavailable_skus", string.Join(",",
                            inventoryResult.UnavailableSkus) }
                    }));

                return OrderResult.InsufficientInventory(inventoryResult.UnavailableSkus);
            }

            // Process payment
            var paymentResult = await _paymentClient
                .ChargeAsync(order.CustomerId, order.Total);

            activity?.SetTag("payment.transaction_id", paymentResult.TransactionId);

            _logger.LogInformation(
                "Order {OrderId} processed successfully. Transaction: {TransactionId}",
                order.Id, paymentResult.TransactionId);

            activity?.SetStatus(ActivityStatusCode.Ok);
            OrdersProcessed.Add(1,
                new KeyValuePair<string, object?>("order.status", "completed"));

            return OrderResult.Success(paymentResult.TransactionId);
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            OrdersProcessed.Add(1,
                new KeyValuePair<string, object?>("order.status", "failed"));

            _logger.LogError(ex, "Failed to process order {OrderId}", order.Id);
            throw;
        }
        finally
        {
            stopwatch.Stop();
            OrderProcessingDuration.Record(stopwatch.Elapsed.TotalMilliseconds,
                new KeyValuePair<string, object?>("order.status",
                    activity?.Status == ActivityStatusCode.Ok ? "completed" : "failed"));
        }
    }
}

Enter fullscreen mode Exit fullscreen mode

Step 4: Correlating Requests Across Services

For HTTP-based communication, correlation is automatic. The HttpClient instrumentation injects traceparent headers, and the receiving service's AspNetCore instrumentation picks them up.

For message-based communication (e.g., Azure Service Bus, RabbitMQ), you need to propagate context manually:

// Producer: inject trace context into message headers
public async Task PublishOrderEventAsync(OrderEvent orderEvent)
{
    using var activity = ActivitySource.StartActivity(
        "PublishOrderEvent", ActivityKind.Producer);

    activity?.SetTag("messaging.system", "servicebus");
    activity?.SetTag("messaging.destination", "order-events");

    var message = new ServiceBusMessage(JsonSerializer.Serialize(orderEvent));

    // Inject current context into message properties
    if (activity?.Context is ActivityContext context)
    {
        message.ApplicationProperties["traceparent"] =
            $"00-{context.TraceId}-{context.SpanId}-{(context.TraceFlags.HasFlag(ActivityTraceFlags.Recorded) ? "01" : "00")}";
    }

    await _sender.SendMessageAsync(message);
}

// Consumer: extract trace context from message headers
public async Task HandleMessageAsync(ServiceBusReceivedMessage message)
{
    ActivityContext parentContext = default;

    if (message.ApplicationProperties.TryGetValue("traceparent", out var traceparent)
        && traceparent is string tp)
    {
        // Parse W3C traceparent header
        var parts = tp.Split('-');
        if (parts.Length == 4)
        {
            var traceId = ActivityTraceId.CreateFromString(parts[1]);
            var spanId = ActivitySpanId.CreateFromString(parts[2]);
            var flags = parts[3] == "01"
                ? ActivityTraceFlags.Recorded : ActivityTraceFlags.None;
            parentContext = new ActivityContext(traceId, spanId, flags, isRemote: true);
        }
    }

    using var activity = ActivitySource.StartActivity(
        "ProcessOrderEvent",
        ActivityKind.Consumer,
        parentContext);

    activity?.SetTag("messaging.system", "servicebus");
    activity?.SetTag("messaging.operation", "process");

    // Process the message...
}

Enter fullscreen mode Exit fullscreen mode


Real-World Scenarios

Scenario 1: Debugging Latency Across Services

A customer complains that placing an order takes too long. With OpenTelemetry configured, you query your tracing backend for traces where order.processing_duration > 5000ms.

You pull up a trace and see:

[OrderController.Post]          0ms ─────────────────────── 6200ms
  [ProcessOrder]                5ms ──────────────────────  6195ms
    [HTTP GET /inventory]       10ms ─── 120ms
    [HTTP POST /payments]       125ms ────────────────────  6100ms
      [SQL: INSERT payment]     130ms ── 180ms
      [HTTP POST /fraud-check]  200ms ────────────────────  6050ms  ← BOTTLENECK

Enter fullscreen mode Exit fullscreen mode

The fraud-check service is taking nearly 6 seconds. Without distributed tracing, you'd be staring at order service logs wondering why ProcessOrderAsync is slow, with no visibility into what the payment service is doing downstream.

Scenario 2: Tracking End-to-End Request Flow

A webhook delivery to a partner is failing intermittently. You search for traces by webhook.partner_id = "acme" and find that successful deliveries take 200ms, but failures show a pattern: the notification service's outbound HTTP call to the partner endpoint returns 503 after exactly 30 seconds — a timeout.

The trace reveals the full chain: API → Order → Notification → Partner webhook. The partner's endpoint is intermittently overloaded. You add a retry policy with exponential backoff and add a span event to track each retry attempt.

Scenario 3: Identifying Production Bottlenecks

Your orders.processing_duration histogram shows P95 degradation over two weeks. You correlate with exemplars — specific trace IDs attached to the slow metric data points — and find they all share a common pattern: the SQL span for inventory reservation is growing linearly.

Digging into the span attributes, you see db.statement contains a query that's missing an index on the sku column. As the product catalog grew, that query went from 5ms to 500ms. The trace data told you exactly which query, in which service, against which table.


Best Practices

Naming Conventions

Use semantic conventions from the OpenTelemetry specification. This ensures consistency across services and teams:

  • Span names should be low-cardinality: ProcessOrder, HTTP GET /api/orders/{id} (not /api/orders/12345). Include the verb and resource, not dynamic IDs.
  • Attribute keys follow dot.notation: order.id, payment.method, customer.tier. Avoid inventing your own when a semantic convention exists (http.method, db.system, rpc.service).
  • Meter and instrument names use reverse-domain style: orderservice.orders.processed, orderservice.processing.duration.

Reducing Telemetry Noise and Cost

Not every span is worth sending to your backend:

  • Filter health checks and readiness probes — as shown in the setup code, exclude /health and /ready endpoints from trace instrumentation.
  • Use the Filter callback on instrumentation options to drop low-value spans programmatically.
  • Limit db.statement capture in production. Full SQL text is invaluable for debugging but expensive to store at scale. Consider enabling it only in non-production environments or for sampled traces.
  • Control attribute cardinality. Adding user.email to every span creates millions of unique attribute values and explodes storage costs. Prefer IDs over free-text values.

Sampling Strategies

Sampling controls which traces get recorded and exported:

  • AlwaysOn: Record every trace. Fine for low-traffic services or development.
  • AlwaysOff: Record nothing. Useful for suppressing telemetry in batch jobs that would generate excessive noise.
  • TraceIdRatioBased: Record a fixed percentage of traces (e.g., 10%). Simple and predictable, but you might miss the rare-but-critical slow request.
  • ParentBased: Respect the sampling decision of the upstream service. If the parent was sampled, sample the child too. This is critical for keeping traces complete.
tracing.SetSampler(new ParentBasedSampler(
    new TraceIdRatioBasedSampler(0.1))); // 10% base rate, but always honor parent

Enter fullscreen mode Exit fullscreen mode

In practice, tail-based sampling (deciding after the trace is complete, based on whether it was slow or errored) is far more useful than head-based sampling. This requires the OpenTelemetry Collector with a tail-sampling processor — the SDK doesn't support it natively.

Securing Telemetry Data

Telemetry data can contain sensitive information: query parameters, user IDs, database statements, error messages with PII.

  • Scrub sensitive attributes before export. Use the SDK's Enrich and Filter callbacks to redact or remove PII.
  • Use TLS for OTLP exports. The OTLP exporter supports gRPC and HTTP/protobuf — ensure endpoints use https:// or encrypted gRPC channels.
  • Restrict access to your telemetry backend. Traces contain internal architecture details. Treat your observability platform with the same access control rigor as production databases.
  • Never log secrets. This sounds obvious, but automatic instrumentation on HttpClient can capture request headers. Explicitly exclude Authorization and other sensitive headers.
.AddHttpClientInstrumentation(opts =>
{
    opts.FilterHttpRequestMessage = req =>
    {
        // Don't trace calls to the token endpoint
        return !req.RequestUri!.AbsolutePath.Contains("/oauth/token");
    };
    opts.EnrichWithHttpRequestMessage = (activity, request) =>
    {
        // Remove sensitive query parameters
        activity?.SetTag("http.url",
            request.RequestUri?.GetLeftPart(UriPartial.Path));
    };
})

Enter fullscreen mode Exit fullscreen mode


Deploying and Monitoring in Azure

Integration with Application Insights / Azure Monitor

The Azure.Monitor.OpenTelemetry.Exporter package sends traces, metrics, and logs directly to Application Insights without requiring the OpenTelemetry Collector as an intermediary. For many teams on Azure, this is the fastest path to production observability.

Alternatively, the newer Azure Monitor OpenTelemetry Distro (Azure.Monitor.OpenTelemetry.AspNetCore) bundles common instrumentation libraries with the exporter in a single package:

builder.Services.AddOpenTelemetry()
    .UseAzureMonitor(opts =>
    {
        opts.ConnectionString = builder.Configuration["AzureMonitor:ConnectionString"];
    });

Enter fullscreen mode Exit fullscreen mode

This auto-configures ASP.NET Core, HttpClient, and SQL Client instrumentation with the Azure Monitor exporter. It's opinionated but gets you to 80% coverage with minimal code.

In Application Insights, you get:

  • Transaction search: Find traces by operation ID, custom attributes, or time range.
  • Application Map: Visualize service dependencies and identify failure hotspots.
  • Performance blade: Drill from P95 latency to individual slow traces.
  • Failures blade: Aggregate exceptions by type and trace them back to the originating request.

Observability in Containerized Environments (Kubernetes)

When running .NET microservices on AKS or any Kubernetes cluster, the recommended architecture is:

  1. Deploy the OpenTelemetry Collector as a DaemonSet or sidecar. Services export telemetry via OTLP to the Collector on localhost or the node's IP — fast, low-latency, no cross-network traffic.
  2. The Collector handles routing, batching, and export. It can fan out to multiple backends (Azure Monitor + Grafana Tempo), apply tail-based sampling, enrich spans with Kubernetes metadata (k8s.pod.name, k8s.namespace.name), and buffer during backend outages.
  3. Use the Kubernetes Attributes Processor in the Collector to automatically attach pod, node, and deployment metadata to every span — invaluable for correlating application behavior with infrastructure events.

A minimal Collector config for Azure:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  k8sattributes:
    extract:
      metadata:
        - k8s.pod.name
        - k8s.namespace.name
        - k8s.deployment.name
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces
        type: latency
        latency: { threshold_ms: 2000 }
      - name: probabilistic
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  azuremonitor:
    connection_string: ${APPLICATIONINSIGHTS_CONNECTION_STRING}
  otlp/tempo:
    endpoint: tempo.monitoring.svc.cluster.local:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [k8sattributes, tail_sampling, batch]
      exporters: [azuremonitor, otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [k8sattributes, batch]
      exporters: [azuremonitor]
    logs:
      receivers: [otlp]
      processors: [k8sattributes, batch]
      exporters: [azuremonitor]

Enter fullscreen mode Exit fullscreen mode

This gives you tail-based sampling (always capture errors and slow requests, sample 10% of everything else), Kubernetes metadata enrichment, and dual export to both Azure Monitor and Grafana Tempo.


Key Takeaways

  • OpenTelemetry is the industry standard for vendor-neutral observability. In .NET, it builds on native APIs (Activity, Meter, ILogger), making adoption low-friction.
  • The three signals — traces, metrics, and logs — are complementary. Metrics detect anomalies, traces locate the problem, logs explain the cause. Correlation via trace IDs ties them together.
  • Automatic instrumentation covers the infrastructure layer (HTTP, SQL, gRPC). Manual instrumentation is where you capture business-domain semantics (order processing, payment flow, inventory checks).
  • Sampling and filtering are essential at scale. Tail-based sampling in the OTel Collector is the most effective strategy for capturing interesting traces without drowning in data.
  • The OTel Collector is your telemetry control plane. Use it to decouple services from backends, apply processing rules, and route data to multiple destinations.

Implementation Checklist

  • [ ] Install OTel SDK and instrumentation packages for ASP.NET Core, HttpClient, and your data access layer.
  • [ ] Configure ResourceBuilder with service name, version, and environment attributes.
  • [ ] Register automatic instrumentation for all infrastructure libraries in use.
  • [ ] Create an ActivitySource and Meter for each service's custom business telemetry.
  • [ ] Add meaningful span attributes and events to manual instrumentation — follow semantic conventions.
  • [ ] Implement context propagation for asynchronous messaging (Service Bus, RabbitMQ, Kafka).
  • [ ] Configure exporters: OTLP to a Collector for production, Console for local development.
  • [ ] Set up sampling: ParentBasedSampler with TraceIdRatioBasedSampler as a starting point; evaluate tail-based sampling in the Collector.
  • [ ] Filter out noise: health checks, readiness probes, internal polling.
  • [ ] Scrub PII from span attributes and suppress sensitive headers.
  • [ ] Deploy the OpenTelemetry Collector in your Kubernetes cluster with batching, K8s metadata enrichment, and your chosen exporters.
  • [ ] Validate end-to-end trace correlation across at least two services before going to production.

Next Steps

Once you have baseline observability in place, consider exploring:

  • Distributed context baggage for passing business-context values (tenant ID, feature flags) across service boundaries without adding them to every call explicitly.
  • Exemplars — linking specific metric data points to the traces that produced them — for bridging from metric alerts to root-cause traces.
  • Custom span processors for advanced enrichment: attaching git commit SHAs, feature flag states, or canary deployment markers to every span.
  • SLO-driven alerting built on OTel metrics: define error budgets and burn rates using the histogram data you're already collecting.
  • OpenTelemetry instrumentation for Azure SDK calls (Azure.Core tracing) to get visibility into blob storage, Key Vault, and other Azure service interactions.
  • Performance tuning the telemetry pipeline itself: batch sizes, export intervals, memory limits on the Collector — telemetry infrastructure at scale needs its own observability.

Observability isn't a feature you ship once. It's a practice you refine as your system grows. OpenTelemetry gives you the foundation — a single, portable instrumentation layer that evolves with your architecture. Start with automatic instrumentation, add manual spans for the operations that matter most to your business, and iterate from there.