Lesson 62 of 66 · 18 min read · Updated

Instrumenting .NET Apps on Kubernetes with OpenTelemetry

In the previous lesson Prometheus scraped a /metrics endpoint and you could see request rates, latency percentiles and an alert. Metrics answer "is something wrong, and how much?". They rarely answer "why?". When the 95th percentile of checkout doubles, you want to open one slow request and see where the time went: in your API, in the pricing service it called, or in the database behind that. And you want the log lines written during that exact request, not a grep through every Pod.

That needs the other two signals: traces and logs, correlated with each other and with the metrics. OpenTelemetry (OTel) is the open standard for producing all three, and the OpenTelemetry Collector is the component that receives them inside the cluster, enriches them with Kubernetes metadata, and sends them wherever you want. In this lesson you instrument two .NET 10 services, run a Collector, follow one request across both services, find its log lines by trace id, and push the metrics into the Prometheus you installed last time over OTLP.

We will cover the following topics:

  1. The three signals, and how trace context ties them together
  2. The OpenTelemetry .NET SDK in .NET 10: traces, metrics and logs with a single UseOtlpExporter()
  3. Configuring the SDK through OTEL_* environment variables in a Pod spec
  4. The OpenTelemetry Collector: receivers, processors, exporters and pipelines
  5. Deployment vs DaemonSet collectors
  6. The k8sattributes processor and the RBAC it needs
  7. Sending metrics to Prometheus over OTLP, and choosing a backend for traces and logs

Technical Requirements

  1. The multi-node kind cluster used throughout this series (Docker Desktop's Kubernetes with the kind provisioner, one control-plane and three workers, Kubernetes v1.36), or an equivalent kind create cluster setup.
  2. kubectl, Helm, Docker and the .NET 10 SDK.
  3. kube-prometheus-stack from the previous lesson, installed as release kps in namespace monitoring. If you skipped it, you can still follow everything except Step 8.

Core Concepts

Three Signals, One Context

SignalQuestion it answersUnit of data
MetricsHow many, how fast, how full, over time?Aggregated numbers with labels
TracesWhat happened during this request, across services?Spans: named, timed operations with a parent
LogsWhat did the code say at that moment?Timestamped records with a message and attributes

A trace is a tree of spans. The root span is usually the incoming HTTP request. Every outgoing call, database query or interesting piece of work becomes a child span. Every span carries the same trace id.

When service A calls service B, A's HttpClient adds a traceparent header to the request (the W3C Trace Context standard). B's ASP.NET Core reads it and continues the same trace. No code is needed in either service for that.

Logs written while a span is active carry its trace id and span id. That is the link between the signals: from a slow span you jump to the exact log lines of that request, and from an error log line you jump to the full trace.

OpenTelemetry in .NET 10

.NET does not need a separate tracing API. It has had one built in for years:

OTel concept.NET API
Tracer / spanActivitySource / Activity
Meter / instrumentSystem.Diagnostics.Metrics.Meter, Counter<T>, Histogram<T>
LoggerMicrosoft.Extensions.Logging.ILogger

ASP.NET Core, HttpClient, Kestrel and the runtime already emit activities and metrics. The OpenTelemetry SDK listens to them and exports them. The packages used in this lesson:

PackageVersionPurpose
OpenTelemetry.Extensions.Hosting1.20.1AddOpenTelemetry() on the service collection
OpenTelemetry.Exporter.OpenTelemetryProtocol1.20.1OTLP exporter and UseOtlpExporter()
OpenTelemetry.Instrumentation.AspNetCore1.20.0Server spans and metrics for incoming requests
OpenTelemetry.Instrumentation.Http1.20.0Client spans and metrics for HttpClient
OpenTelemetry.Instrumentation.Runtime1.20.0GC, thread pool, exceptions

UseOtlpExporter() registers one OTLP exporter for all three signals. Without it you would call AddOtlpExporter() separately on tracing, metrics and logging.

Configure with Environment Variables, Not Code

The SDK reads a standard set of environment variables. This is what makes it fit Kubernetes so well: the same image runs in every environment and the Pod spec decides where the data goes.

VariableExampleMeaning
OTEL_SERVICE_NAMEcatalog-apiThe service.name resource attribute. The most important one.
OTEL_EXPORTER_OTLP_ENDPOINThttp://otel-collector.obs-otel:4317Where to send OTLP data
OTEL_EXPORTER_OTLP_PROTOCOLgrpc (default) or http/protobufWire protocol. gRPC uses port 4317, HTTP uses 4318
OTEL_RESOURCE_ATTRIBUTESservice.version=2.0,deployment.environment.name=devExtra resource attributes, comma-separated
OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARGparentbased_traceidratio / 0.1Keep 10% of new traces
OTEL_METRIC_EXPORT_INTERVAL60000Milliseconds between metric exports (default 60 s)

A resource describes who produced the telemetry: service name, version, environment, and in Kubernetes the Pod, namespace, node and Deployment. Every span, metric point and log record is attached to a resource.

Why a Collector?

Applications could send OTLP straight to a backend. In practice you put an OpenTelemetry Collector in between:

  1. Applications only know one address. Switching from Jaeger to Tempo, or adding SigNoz next to Prometheus, is a Collector change. No redeploys.
  2. The Collector batches, retries and limits memory, so a slow backend does not slow your API.
  3. It adds information the application does not know, such as the Deployment that owns the Pod.
  4. It can filter, redact, sample and transform in one place.

A Collector is configured with four kinds of building blocks:

BlockExamplesRole
Receiversotlp, prometheus, file_log, kubelet_statsHow data gets in
Processorsmemory_limiter, k8s_attributes, batch, transform, filterWhat happens to it on the way
Exportersotlp_grpc, otlp_http, debugWhere it goes
Pipelinestraces, metrics, logsWhich receivers, processors and exporters are wired together for each signal

A note on names. Recent Collector releases (0.162 is used here) renamed components to snake_case: k8sattributes became k8s_attributes, the otlp exporter became otlp_grpc, and otlphttp became otlp_http. Most blog posts still use the old names. The Helm chart translates the old names for you, but new configuration should use the new ones.

Deployment vs DaemonSet

ModeWhat it looks likeUse it for
Deployment (gateway)A few replicas behind a ClusterIP ServiceCentral processing: tail sampling, routing, export with credentials. Simplest to start with.
DaemonSet (agent)One Collector per nodeReading container log files from the node, kubelet stats, host metrics. Apps send to the Collector on their own node (status.hostIP) and never cross the network for telemetry.

Large clusters often run both: an agent on every node that forwards to a gateway. This lesson uses a single Deployment, which is enough for application traces, metrics and logs sent over OTLP.

The k8s_attributes Processor

When a Pod sends data to the Collector, the Collector sees the Pod's IP on the connection. The k8s_attributes processor watches the Kubernetes API, finds the Pod with that IP, and adds attributes such as k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.node.name and k8s.container.name to the resource.

To do that it needs to read Pods, Namespaces and ReplicaSets across the cluster. The ReplicaSet is how it gets from a Pod to its Deployment name. So the Collector's ServiceAccount needs a ClusterRole with get, list and watch on those resources. The Helm chart's kubernetesAttributes preset creates the processor configuration and that RBAC for you.

Where Do Traces and Logs Go?

The Collector is not a database. For the hands-on part we use the debug exporter, which prints everything to the Collector's own log, so you can see exactly what arrives without installing anything else. Metrics go to Prometheus, which has a native OTLP receiver since version 3.

For real use, point the Collector at a backend that stores and searches traces and logs:

BackendSignalsNotes
JaegerTracesCNCF project, simple to run, accepts OTLP natively
Grafana Tempo + LokiTraces + logsPairs with the Grafana and Prometheus you already have
SigNozTraces, metrics, logsOpen-source, OTLP-native, one UI for all three
Elastic, Datadog, Azure Monitor, and othersAllCommercial or managed, all accept OTLP

Swapping between them is a change of exporter in the Collector configuration.

Hands-On: Kubernetes Commands

Add the OpenTelemetry Helm repository

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

Render the Collector configuration before installing

Shows the final config, with preset processors and RBAC included.

helm template otel-collector open-telemetry/opentelemetry-collector --version 0.176.1 -n obs-otel -f collector-values.yaml | less

List the components compiled into a Collector image

A component you configure must exist in the distribution you run.

docker run --rm otel/opentelemetry-collector-k8s:0.162.0 components

Follow what the Collector receives

With the debug exporter, the Collector's log is your first trace viewer.

kubectl -n obs-otel logs deploy/otel-collector -f

Check the Collector's permissions

Needed for the k8s_attributes processor.

kubectl auth can-i list replicasets --as=system:serviceaccount:obs-otel:otel-collector -A

Print a Pod's OTEL environment

Quickly confirms where a workload sends its telemetry.

kubectl -n obs-shop exec deploy/catalog-api -- printenv | grep OTEL

Step-by-Step Example

The Scenario

A storefront has two services. catalog-api serves product pages and handles checkout. For both, it asks pricing-api for the current price over HTTP. You want to:

  1. See every checkout as one trace spanning both services.
  2. Read the log lines of that exact checkout.
  3. Know which Pod, Deployment and node every piece of telemetry came from, without writing that into the application.
  4. Keep using Prometheus and PromQL for metrics.

Step 1 — Instrument pricing-api

dotnet new web -n PricingApi
cd PricingApi
dotnet add package OpenTelemetry.Extensions.Hosting --version 1.20.1
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol --version 1.20.1
dotnet add package OpenTelemetry.Instrumentation.AspNetCore --version 1.20.0
dotnet add package OpenTelemetry.Instrumentation.Runtime --version 1.20.0

Program.cs:

using OpenTelemetry;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
.WithTracing(t => t.AddAspNetCoreInstrumentation())
.WithMetrics(m => m.AddAspNetCoreInstrumentation().AddRuntimeInstrumentation())
.WithLogging()
.UseOtlpExporter();

var app = builder.Build();

app.MapGet("/prices/{id:int}", async (int id, ILogger<Program> logger) =>
{
await Task.Delay(Random.Shared.Next(10, 80)); // pretend to call a pricing database
var amount = 10m + id % 7 * 2.5m;
logger.LogInformation("Calculated price {Price} for product {ProductId}", amount, id);
return Results.Ok(new { productId = id, amount });
});

app.Run();

There is no service name, endpoint or version in the code. All of that comes from the environment. WithLogging() adds an OpenTelemetry logger provider to the normal ILogger pipeline, so existing log statements are exported without changes. Structured logging with message templates ({Price}, {ProductId}) matters here: the values arrive as separate attributes you can filter on, not baked into a string.

Step 2 — Instrument catalog-api, with a Custom Span and Metric

Create CatalogApi the same way and also add OpenTelemetry.Instrumentation.Http --version 1.20.0, because this service makes outgoing calls.

using System.Diagnostics;
using System.Diagnostics.Metrics;
using OpenTelemetry;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

var activitySource = new ActivitySource("Storefront.Catalog");
var meter = new Meter("Storefront.Catalog");
var checkouts = meter.CreateCounter<long>("catalog.checkouts", description: "Completed checkouts");

// Service name, version and extra resource attributes come from OTEL_* environment variables.
builder.Services.AddOpenTelemetry()
.WithTracing(t => t
.AddSource("Storefront.Catalog")
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation())
.WithMetrics(m => m
.AddMeter("Storefront.Catalog")
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddRuntimeInstrumentation())
.WithLogging()
.UseOtlpExporter(); // one exporter for traces, metrics and logs, configured by OTEL_EXPORTER_OTLP_*

builder.Services.AddHttpClient("pricing", c =>
c.BaseAddress = new Uri(builder.Configuration["PRICING_URL"] ?? "http://pricing-api"));

var app = builder.Build();

app.MapGet("/products/{id:int}", async (int id, IHttpClientFactory factory, ILogger<Program> logger) =>
{
var price = await factory.CreateClient("pricing").GetFromJsonAsync<PriceResponse>($"/prices/{id}");
logger.LogInformation("Served product {ProductId} with price {Price}", id, price!.Amount);
return Results.Ok(new { id, name = $"Product {id}", price = price.Amount });
});

app.MapPost("/checkout/{id:int}", async (int id, IHttpClientFactory factory, ILogger<Program> logger) =>
{
using var activity = activitySource.StartActivity("ReserveStock");
activity?.SetTag("product.id", id);
await Task.Delay(Random.Shared.Next(20, 120));

var price = await factory.CreateClient("pricing").GetFromJsonAsync<PriceResponse>($"/prices/{id}");
checkouts.Add(1);
logger.LogInformation("Checkout completed for product {ProductId} at {Price}", id, price!.Amount);
return Results.Ok(new { status = "paid", price = price.Amount });
});

app.Run();

record PriceResponse(int ProductId, decimal Amount);

AddSource("Storefront.Catalog") is easy to forget. An ActivitySource that is not registered produces no spans: StartActivity simply returns null, which is why the code uses activity?.. The same applies to meters and AddMeter.

Use the same multi-stage Dockerfile as in the previous lesson (with the right .csproj and .dll names) and build both images:

docker build -t pricing-api:1.0 ./PricingApi
docker build -t catalog-api:2.0 ./CatalogApi

With a plain kind cluster, load them with kind load docker-image afterwards. Docker Desktop's Kubernetes sees them directly.

Step 3 — Write the Collector Values

Save as collector-values.yaml:

fullnameOverride: otel-collector
mode: deployment
image:
repository: otel/opentelemetry-collector-k8s # the chart requires you to choose a distribution
presets:
kubernetesAttributes:
enabled: true # adds the k8s_attributes processor and the RBAC it needs
resources:
limits:
memory: 512Mi # memory_limiter is derived from this
config:
receivers:
jaeger: null # we only use OTLP
zipkin: null
exporters:
debug:
verbosity: basic # one summary line per batch
debug/detailed:
verbosity: detailed # every span and log record
otlp_http/prometheus:
endpoint: http://kps-kube-prometheus-stack-prometheus.monitoring:9090/api/v1/otlp
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, k8s_attributes, batch]
exporters: [debug/detailed]
metrics:
receivers: [otlp]
processors: [memory_limiter, k8s_attributes, batch]
exporters: [otlp_http/prometheus, debug]
logs:
receivers: [otlp]
processors: [memory_limiter, k8s_attributes, batch]
exporters: [debug/detailed]
ports:
jaeger-compact:
enabled: false
jaeger-thrift:
enabled: false
jaeger-grpc:
enabled: false
zipkin:
enabled: false

Things to notice:

  1. otel/opentelemetry-collector-k8s is the distribution built for Kubernetes. It includes k8s_attributes, kubelet_stats, file_log and the OTLP components, but not every exporter in the contrib project. For example, it has no prometheus exporter, which is one reason we push metrics to Prometheus over OTLP instead of having Prometheus scrape the Collector.
  2. The order of processors matters. memory_limiter goes first so it can refuse data before anything else allocates memory. batch goes last so it batches the final result.
  3. exporter/name syntax (debug/detailed, otlp_http/prometheus) creates several instances of the same component type with different settings.

Step 4 — Install the Collector

helm install otel-collector open-telemetry/opentelemetry-collector --version 0.176.1 -n obs-otel --create-namespace -f collector-values.yaml --wait
kubectl -n obs-otel get pods,svc
NAME READY STATUS RESTARTS AGE
pod/otel-collector-9b6b99b46-52bmp 1/1 Running 0 5m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/otel-collector ClusterIP 10.96.155.23 <none> 4317/TCP,4318/TCP 5m

The chart also created a ClusterRole for the processor:

kubectl get clusterrole otel-collector -o yaml
rules:
- apiGroups: [""]
resources: [pods, namespaces]
verbs: [get, watch, list]
- apiGroups: [apps]
resources: [replicasets]
verbs: [get, list, watch]

Step 5 — Deploy the Services with OTEL Environment Variables

Save as apps.yaml:

apiVersion: v1
kind: Namespace
metadata:
name: obs-shop
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: pricing-api
namespace: obs-shop
spec:
replicas: 2
selector:
matchLabels: { app: pricing-api }
template:
metadata:
labels: { app: pricing-api }
spec:
containers:
- name: pricing-api
image: pricing-api:1.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
env:
- name: OTEL_SERVICE_NAME
value: pricing-api
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://otel-collector.obs-otel:4317
- name: POD_NAME
valueFrom: { fieldRef: { fieldPath: metadata.name } }
- name: OTEL_RESOURCE_ATTRIBUTES
value: service.version=1.0,deployment.environment.name=dev,k8s.pod.name=$(POD_NAME)
---
apiVersion: v1
kind: Service
metadata:
name: pricing-api
namespace: obs-shop
spec:
selector: { app: pricing-api }
ports:
- port: 80
targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: catalog-api
namespace: obs-shop
spec:
replicas: 2
selector:
matchLabels: { app: catalog-api }
template:
metadata:
labels: { app: catalog-api }
spec:
containers:
- name: catalog-api
image: catalog-api:2.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
env:
- name: PRICING_URL
value: http://pricing-api
- name: OTEL_SERVICE_NAME
value: catalog-api
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://otel-collector.obs-otel:4317
- name: POD_NAME
valueFrom: { fieldRef: { fieldPath: metadata.name } }
- name: OTEL_RESOURCE_ATTRIBUTES
value: service.version=2.0,deployment.environment.name=dev,k8s.pod.name=$(POD_NAME)
---
apiVersion: v1
kind: Service
metadata:
name: catalog-api
namespace: obs-shop
spec:
selector: { app: catalog-api }
ports:
- port: 80
targetPort: 8080
kubectl apply -f apps.yaml
kubectl -n obs-shop wait --for=condition=Available deploy --all --timeout=180s

Two techniques are combined here. $(POD_NAME) is Kubernetes dependent environment variable syntax: a variable can reference another variable defined earlier in the same list, and the downward API fills POD_NAME from the Pod's metadata. The k8s_attributes processor would add the Pod name anyway; setting it in the SDK as well is a safety net if telemetry ever bypasses the Collector. The endpoint uses the Service's cross-namespace DNS name, otel-collector.obs-otel.

Step 6 — Follow One Checkout Across Both Services

Send some traffic:

kubectl -n obs-shop port-forward svc/catalog-api 8080:80
curl -s localhost:8080/products/3
{"id":3,"name":"Product 3","price":17.5}
curl -s -X POST localhost:8080/checkout/4
{"status":"paid","price":20.0}
for i in $(seq 1 30); do curl -s localhost:8080/products/$i >/dev/null; curl -s -X POST localhost:8080/checkout/$i >/dev/null; done

Now read the Collector's log. Each batch of spans starts with the resource that produced it:

kubectl -n obs-otel logs deploy/otel-collector --since=1m
ResourceSpans #0
Resource attributes:
-> service.name: Str(pricing-api)
-> service.version: Str(1.0)
-> deployment.environment.name: Str(dev)
-> k8s.pod.name: Str(pricing-api-78499668db-phl77)
-> telemetry.sdk.language: Str(dotnet)
-> telemetry.sdk.version: Str(1.20.1)
-> service.namespace: Str(obs-shop)
-> k8s.replicaset.name: Str(pricing-api-78499668db)
-> k8s.deployment.name: Str(pricing-api)
-> k8s.node.name: Str(desktop-worker2)
-> k8s.namespace.name: Str(obs-shop)
-> k8s.pod.uid: Str(a6c44ae6-be6b-49b0-b80d-0f23ca29cb88)
-> k8s.container.name: Str(pricing-api)
-> container.image.name: Str(pricing-api)
-> service.instance.id: Str(obs-shop.pricing-api-78499668db-phl77.pricing-api)

The first six attributes came from the SDK and the environment variables. Everything from service.namespace down was added by k8s_attributes. The application never mentioned its Deployment, ReplicaSet, node or namespace.

Then come the spans. Picking the spans of one trace id from that log gives the whole checkout:

ServiceSpan nameKindSpan IDParent ID
catalog-apiPOST /checkout/{id:int}Server1f8553c8fa7d44b4(root)
catalog-apiReserveStockInternal0b205bdefe7d3fd11f8553c8fa7d44b4
catalog-apiGETClient54d65468a4735ba40b205bdefe7d3fd1
pricing-apiGET /prices/{id:int}Serverb7d7244928f5b61b54d65468a4735ba4

Read the Parent ID column from top to bottom and you have the call tree: the incoming checkout request, your custom ReserveStock span, the outgoing HttpClient call, and pricing-api's handling of that call, in another Pod and possibly on another node. The traceparent header did that without a line of propagation code.

A server span looks like this in detail:

Span #1
Trace ID : b9f7f60e0888d6bdc80825f68908a3ce
Parent ID : 54d65468a4735ba4
ID : b7d7244928f5b61b
Name : GET /prices/{id:int}
Kind : Server
Attributes:
-> http.request.method: Str(GET)
-> url.path: Str(/prices/4)
-> http.route: Str(/prices/{id:int})
-> http.response.status_code: Int(200)

Step 7 — Find the Logs of That Request

Log records arrive through the logs pipeline with the trace id and span id of the span that was active when they were written:

LogRecord #1
SeverityText: Information
Body: Str(Calculated price {Price} for product {ProductId})
Attributes:
-> Price: Str(20.0)
-> ProductId: Int(4)
Trace ID: b9f7f60e0888d6bdc80825f68908a3ce
Span ID: b7d7244928f5b61b

Same trace id as the checkout, and the span id of pricing-api's server span. In a backend such as SigNoz, Grafana (Tempo + Loki) or Jaeger with a log store, this is the "show logs for this span" button. Notice that the body is the message template, and Price and ProductId are separate attributes, so "all checkouts for product 4" becomes a simple filter.

Step 8 — Send Metrics to Prometheus over OTLP

The Collector's metrics pipeline already points at http://kps-kube-prometheus-stack-prometheus.monitoring:9090/api/v1/otlp. Prometheus only accepts OTLP when the receiver is switched on. Save as kps-otlp-values.yaml:

prometheus:
prometheusSpec:
enableOTLPReceiver: true
otlp:
# turn these resource attributes into labels on every series
promoteResourceAttributes:
- k8s.namespace.name
- k8s.deployment.name
- k8s.pod.name
helm upgrade kps prometheus-community/kube-prometheus-stack --version 92.11.0 -n monitoring --reuse-values -f kps-otlp-values.yaml --wait

By default Prometheus maps service.namespace/service.name to the job label and service.instance.id to instance. All other resource attributes go into a separate target_info series. promoteResourceAttributes copies the ones you query by most often onto every series, with dots turned into underscores.

After a couple of minutes of traffic (the SDK exports metrics every 60 seconds by default, and rate() needs two points):

catalog_checkouts_total
catalog_checkouts_total{instance="obs-shop.catalog-api-5c449c554b-ghg8f.catalog-api", job="obs-shop/catalog-api", k8s_deployment_name="catalog-api", k8s_namespace_name="obs-shop", k8s_pod_name="catalog-api-5c449c554b-ghg8f"} 26

sum by (k8s_deployment_name, http_route) (rate(http_server_request_duration_seconds_count{k8s_namespace_name="obs-shop"}[5m]))
{http_route="/prices/{id:int}", k8s_deployment_name="pricing-api"} 2.82
{http_route="/products/{id:int}", k8s_deployment_name="catalog-api"} 2.67
{http_route="/checkout/{id:int}", k8s_deployment_name="catalog-api"} 2.67

histogram_quantile(0.95, sum by (le) (rate(http_client_request_duration_seconds_bucket{k8s_deployment_name="catalog-api"}[5m])))
{} 0.081

The last query is the 95th percentile of catalog-api's outgoing calls to pricing-api, about 81 ms. Same PromQL as in the previous lesson, but no ServiceMonitor was needed: the data was pushed. Both models coexist happily in one Prometheus.

The debug exporter in the metrics pipeline prints one line per batch, a quick way to see that metrics flow:

info Metrics "resource metrics": 1, "metrics": 28, "data points": 39

Step 9 — Point Traces and Logs at a Real Backend

Replace the debug exporter in the traces and logs pipelines with an OTLP exporter to your backend. The applications do not change. For example, for an OTLP endpoint such as SigNoz, Jaeger or Tempo running in the cluster:

config:
exporters:
otlp_grpc/backend:
endpoint: my-backend-collector.observability:4317
tls:
insecure: true
service:
pipelines:
traces:
exporters: [otlp_grpc/backend]
logs:
exporters: [otlp_grpc/backend]

For a hosted backend, add the API key with the headers setting of the exporter and keep the key in a Secret mounted as an environment variable, referenced as ${env:BACKEND_API_KEY}.

Step 10 — Clean Up

kubectl delete namespace obs-shop
helm uninstall otel-collector -n obs-otel
kubectl delete namespace obs-otel

If you no longer need Prometheus either, remove it as described at the end of the previous lesson, including the monitoring.coreos.com CRDs.

Summary

  1. Metrics say that something is wrong, traces show where, logs say why. Trace and span ids tie them together.
  2. .NET already speaks OpenTelemetry through Activity, Meter and ILogger. The OpenTelemetry SDK exports them; UseOtlpExporter() sends all three signals over OTLP.
  3. Keep endpoints, service names and resource attributes out of code. Set OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_RESOURCE_ATTRIBUTES in the Pod spec.
  4. Register every custom ActivitySource and Meter, or they produce nothing.
  5. The OpenTelemetry Collector decouples applications from backends: receivers, processors, exporters and per-signal pipelines. Put memory_limiter first and batch last.
  6. Run it as a Deployment to start, add a DaemonSet agent when you need node-level data such as container log files.
  7. The k8s_attributes processor adds Pod, Deployment, namespace and node to every signal. It needs a ClusterRole to read Pods, Namespaces and ReplicaSets.
  8. Prometheus 3 can receive OTLP directly; promote the resource attributes you filter on into labels.

In the next lesson you take a closer look at the ASP.NET Core side of running on Kubernetes: health checks that map to probes, graceful shutdown during rolling updates, and configuration from ConfigMaps and Secrets.