Monitoring Kubernetes with Prometheus and Grafana

Up to this point in the series you have learned to run, configure, expose and scale workloads. You have checked on them with kubectl get pods and kubectl logs. That works while you are staring at the terminal. It tells you nothing about what happened at three in the morning, whether checkout got slower after the last release, or which namespace is quietly eating all the memory.

That is the job of monitoring: collect numbers from every part of the system on a schedule, keep them as time series, query them, draw them and alert on them. In the Kubernetes world the de facto standard for this is Prometheus, with Grafana on top for dashboards. In this lesson you install both with a single Helm chart, teach Prometheus to scrape a .NET 10 API, write your first PromQL queries, build a dashboard from a ConfigMap, and create an alert that actually fires.

We will cover the following topics:

1. How Prometheus collects metrics with the pull model
2. What kube-prometheus-stack installs and why it uses an Operator
3. The Operator's custom resources: ServiceMonitor, PodMonitor and PrometheusRule
4. Exposing metrics from a .NET 10 API with OpenTelemetry and the Prometheus exporter
5. A practical PromQL primer: rate, sum by, histogram_quantile
6. Grafana dashboards, including dashboards provisioned from a ConfigMap
7. Alerting rules and Alertmanager

Technical Requirements

1. A multi-node cluster. The examples use Docker Desktop's built-in Kubernetes with the kind provisioner: one control-plane node and three workers (desktop-control-plane, desktop-worker, desktop-worker2, desktop-worker3), Kubernetes v1.36. A plain kind create cluster with one control-plane and three workers behaves the same way.
2. kubectl and Helm (the commands were run with Helm 4).
3. Docker and the .NET 10 SDK to build the sample image.
4. About 2 GB of free memory in the cluster for the monitoring stack.

Core Concepts

The Pull Model

Most monitoring systems you may have used before receive data: the application pushes numbers to a central server. Prometheus works the other way around. Every application exposes an HTTP endpoint, usually /metrics, that returns the current value of all its metrics as plain text. Prometheus pulls (scrapes) that endpoint on a fixed interval, adds a timestamp, and stores the result as a time series.

A scrape returns lines like these:

Each line is a metric name, a set of labels in curly braces, and a value. A unique combination of name and labels is one time series. The labels are what make Prometheus powerful: you can slice the same metric by route, status code, Pod, namespace or node at query time.

The pull model fits Kubernetes particularly well. Prometheus asks the Kubernetes API which Pods exist right now and scrapes them. When a Deployment scales from 2 to 10 replicas, nothing has to be reconfigured. If a Pod stops answering, Prometheus records up == 0 for that target, which is itself a useful signal.

The Four Metric Types

The golden rule: never graph a raw counter. A counter only tells you "60 requests since this process started". What you want is "how many per second right now", which is rate().

What kube-prometheus-stack Installs

You could install Prometheus, Grafana, Alertmanager and the exporters one by one and wire them together with a large configuration file. The community chart kube-prometheus-stack does all of it in one release:

The chart also scrapes the kubelet (and through it cAdvisor, which provides per-container CPU and memory), the API server and CoreDNS.

Why an Operator?

A plain Prometheus server is configured with one big prometheus.yml. In a cluster where twenty teams deploy their own services, nobody wants to edit a shared file every time a new API appears. The Prometheus Operator solves this by adding Kubernetes custom resources. Each team describes what should be scraped next to its own application, in its own namespace, and the Operator rewrites the Prometheus configuration for them.

ServiceMonitor Selection — The Classic Trap

A Prometheus resource does not pick up every ServiceMonitor in the cluster. It has a serviceMonitorSelector and a serviceMonitorNamespaceSelector. By default the chart sets the selector to "only ServiceMonitors labelled release: <helm release name>". The most common question about this stack is "I created a ServiceMonitor and nothing happens", and the answer is almost always this label.

You have two options:

1. Add the label release: kps (your release name) to every ServiceMonitor, PodMonitor and PrometheusRule.
2. Tell the chart to pick up all of them, regardless of labels. That is what the values file in this lesson does with serviceMonitorSelectorNilUsesHelmValues: false. The resulting Prometheus resource has an empty selector, serviceMonitorSelector: {}, which matches everything.

The second option is convenient on a single-team or learning cluster. On a shared cluster, a label convention gives the platform team more control.

Exposing Metrics from .NET 10

.NET has a built-in metrics API, System.Diagnostics.Metrics. ASP.NET Core, Kestrel, HttpClient and the runtime already publish metrics through it, for example http.server.request.duration. Your own code creates a Meter and instruments on it.

OpenTelemetry is the vendor-neutral way to collect those metrics and send them somewhere. For Prometheus there are two common choices:

This lesson uses the OpenTelemetry exporter, because the next lesson builds on the same SDK to add traces and logs. The beta label refers to the exporter only. The OpenTelemetry SDK and instrumentation packages are stable.

OpenTelemetry names use dots and units (http.server.request.duration, unit s). The Prometheus exporter translates them to Prometheus conventions: dots become underscores, the unit becomes a suffix, and counters get _total. So catalog.product.views appears as catalog_product_views_total, and http.server.request.duration appears as http_server_request_duration_seconds.

A PromQL Primer

PromQL is the query language of Prometheus. Five patterns cover most day-to-day questions:

How to read them:

1. [5m] is a range selector: "the samples from the last 5 minutes". rate() turns those samples of a counter into an average per-second increase, and copes with counter resets when a Pod restarts. It needs at least two samples, so a freshly scraped target returns nothing for the first scrape interval.
2. sum by (label) aggregates across all other labels. Without it you get one line per Pod.
3. =~ is a regular-expression match, so "5.." matches every 5xx status code.
4. histogram_quantile needs the le ("less than or equal") bucket label preserved in the sum by.

Alerting

An alerting rule is a PromQL expression plus a duration. When the expression returns a result, the alert becomes pending. If it keeps returning a result for the whole for: duration, it becomes firing and Prometheus sends it to Alertmanager. Alertmanager decides who to notify. A recording rule is the other kind of rule: it evaluates an expensive expression on a schedule and stores the result as a new, cheap-to-query time series.

Hands-On: Kubernetes Commands

Install the stack with Helm

One release installs the Operator, Prometheus, Alertmanager, Grafana and the exporters.

List the Operator's custom resources

These CRDs are what you create to configure monitoring.

See which monitors and rules exist

Monitors live next to the applications they describe, in any namespace.

Check what the Prometheus resource selects

Empty selectors ({}) mean "everything".

Open the Prometheus UI

Go to http://localhost:9090/targets to see every scrape target and its health.

Query Prometheus from the command line

The HTTP API is handy for scripts and quick checks.

Open Grafana

Log in as admin with the password from the Secret.

Open Alertmanager

Firing alerts appear here after Prometheus sends them.

Step-by-Step Example

The Scenario

You run a small catalog-api for an online store. It serves product pages and handles checkout. Checkout calls a payment provider that occasionally times out. You want to know the request rate per route, how long checkout takes, which products are viewed most, and you want an alert when more than 5% of checkouts fail.

Step 1 — Prepare the Helm Values

Save this as kps-values.yaml. Every setting has a reason.

The last four settings deserve a word. In kind clusters (and in many managed clusters such as AKS, EKS and GKE), etcd, the controller manager, the scheduler and kube-proxy are either bound to localhost or hidden from you. Leaving their scrape jobs enabled only produces permanently red targets and alerts that can never clear. On a cluster where you control those components you can expose their metrics endpoints and turn the jobs back on.

Step 2 — Install kube-prometheus-stack

Notice two things. There is one node-exporter Pod per node, including the control-plane, because it is a DaemonSet that tolerates the control-plane taint. And the node-exporter Pods have the node's IP (172.18.0.x), not a Pod IP, because they run with hostNetwork: true to see the node's real network interfaces. That also means each one occupies port 9100 on its node. If another node-exporter already runs with host networking on your cluster, you would get a port conflict; change prometheus-node-exporter.service.port in the values or disable it with nodeExporter.enabled: false.

The Prometheus and Alertmanager Pods were not created by the chart directly. The chart created Prometheus and Alertmanager custom resources and the Operator turned them into StatefulSets:

Step 3 — Build a .NET 10 API That Exposes Metrics

Create the project and add the OpenTelemetry packages:

Replace Program.cs:

Two details are worth understanding.

Label cardinality. The product.id tag uses id % 5, not the raw id. Every distinct label value creates a new time series. A label with millions of possible values (user ids, order ids, full URLs) can take down a Prometheus server. Use labels for small, bounded sets: route templates, status codes, regions, product categories.

Histogram buckets. The advice argument sets bucket boundaries in seconds. Without it, OpenTelemetry uses its default boundaries 0, 5, 10, 25, 50, 75, 100, 250, 500, ..., which were designed for milliseconds. Our first build of this API did not set them and every checkout landed in the same bucket:

A histogram where everything sits in the first bucket cannot produce a meaningful percentile. ASP.NET Core's built-in http.server.request.duration already ships with sensible second-based boundaries; your own histograms need them set explicitly.

Add a Dockerfile and a .dockerignore containing bin/ and obj/:

Build it:

Docker Desktop's Kubernetes sees images you build locally, so no push is needed. With a plain kind cluster, run kind load docker-image catalog-api:1.1 first.

Step 4 — Deploy the API

Save as 1-app.yaml. The Service carries the label app: catalog-api, which the ServiceMonitor will select, and names its port http, which the ServiceMonitor will reference.

Generate a little traffic and look at the raw endpoint:

Read the histogram from top to bottom: 11 checkouts finished within 0.1 s, 28 within 0.2 s, all 54 within 0.5 s. Buckets are cumulative. Note also that 60 checkout requests were sent but only 54 were recorded: the failed ones returned before Record was called. The HTTP metric, recorded by ASP.NET Core itself, still counts all of them with their real status code.

Step 5 — Tell Prometheus to Scrape It with a ServiceMonitor

Save as 2-servicemonitor.yaml:

The ServiceMonitor never talks to the Service's ClusterIP. The Operator uses it to find the Service, then generates a scrape job that targets each Pod behind it individually, through the Service's EndpointSlices. That is what you want: averaging metrics through a load balancer would mix up the numbers of different Pods.

After about 30 seconds, open http://localhost:9090/targets (port-forward from the Hands-On section) or ask the API:

Two targets, one per Pod, both up. Prometheus also attached Kubernetes labels to every series it scrapes from them: namespace, pod, service, container, endpoint and job. You did not have to put those in your .NET code.

If you see no targets, check in this order: does the Service have the label from spec.selector? Does the port name match? Is the ServiceMonitor selected by the Prometheus resource (the release label trap)? Does curl against /metrics work through a port-forward?

Step 6 — Write PromQL Queries

Keep some traffic flowing in another terminal:

Then run these queries in the Prometheus UI (Graph tab) or in Grafana's Explore view.

Requests per second by route and status:

The /metrics line is Prometheus itself scraping two Pods every 15 seconds. The http_route label holds the route template /products/{id:int}, not /products/7. ASP.NET Core does that deliberately to keep cardinality bounded.

95th percentile checkout duration:

95% of successful checkouts took less than about 0.46 seconds. The value is an estimate interpolated inside a bucket, which is why sensible boundaries matter.

The three most viewed product groups:

Step 7 — Add a Recording Rule and an Alert

Save as 3-prometheusrule.yaml:

The recording rule's name follows the convention level:metric:operations. Querying catalog:http_requests:rate5m now returns the pre-computed result instantly, which matters for dashboards that many people open.

Our API fails about 10% of checkouts on purpose, so the alert goes pending first and firing after one minute. Check it on the Prometheus Alerts page or with the API:

Prometheus forwarded it to Alertmanager:

Out of the box Alertmanager has no receivers configured, so the alert goes nowhere. In a real cluster you configure receivers (Slack, Teams via webhook, e-mail, PagerDuty) in the chart values under alertmanager.config, or per namespace with an AlertmanagerConfig resource.

Step 8 — Explore the Built-In Grafana Dashboards

Open http://localhost:3000 and log in as admin / prom-operator. Under Dashboards you will find around 25 dashboards that came with the chart, including:

1. Kubernetes / Compute Resources / Namespace (Pods): CPU and memory of every Pod in a namespace, compared with their requests and limits. Pick obs-demo.
2. Kubernetes / Compute Resources / Node (Pods): what runs on each node and how much it uses.
3. Node Exporter / Nodes: CPU, memory, disk and network of each node.
4. Kubernetes / Networking / Namespace (Pods): bytes and packets per Pod.
5. CoreDNS and Kubernetes / API server: the health of the control plane you can see.

These dashboards are driven by cAdvisor, kube-state-metrics and node-exporter. They work for every workload in the cluster without any change to the applications.

Step 9 — Ship Your Own Dashboard as a ConfigMap

Clicking a dashboard together in the UI is fine for exploring, but it lives only in Grafana's database. The chart runs a sidecar next to Grafana that watches all namespaces for ConfigMaps labelled grafana_dashboard: "1" and loads the JSON inside them. That means a dashboard can live in Git next to the application, like every other manifest.

Save as 4-dashboard.yaml:

The data source uid prometheus is the one the chart provisions. Within a few seconds the Catalog API dashboard appears in Grafana. You can check without a browser:

A practical workflow: design the dashboard in the UI, use Export → Export as JSON, and paste the result into the ConfigMap.

Step 10 — Clean Up

Remove the application first, then the stack. Helm does not delete CRDs on uninstall, so remove them explicitly if you want a clean cluster.

If you are continuing with the next lesson, keep the stack installed: it reuses Prometheus.

Summary

1. Prometheus pulls metrics from /metrics endpoints on a schedule and stores them as labelled time series. Kubernetes service discovery means scaling needs no reconfiguration.
2. kube-prometheus-stack installs the Prometheus Operator, Prometheus, Alertmanager, Grafana, kube-state-metrics, node-exporter and a large set of default rules and dashboards in one Helm release.
3. The Operator replaces a hand-edited prometheus.yml with custom resources: ServiceMonitor and PodMonitor for scraping, PrometheusRule for recording and alerting rules. Teams keep them next to their applications.
4. A Prometheus resource only picks up monitors that match its selectors. The release label requirement is the most common reason a ServiceMonitor "does nothing".
5. In .NET 10, ASP.NET Core and the runtime already publish metrics. OpenTelemetry with the Prometheus exporter serves them, plus your own Meter instruments, on /metrics.
6. Keep labels low-cardinality and give your own histograms explicit bucket boundaries in the unit you record.
7. Use rate() on counters, sum by to aggregate, and histogram_quantile on bucket rates for percentiles.
8. Ship dashboards as ConfigMaps labelled grafana_dashboard: "1" so they live in Git.

In the next lesson you add the other two signals, traces and logs, with OpenTelemetry, and route all three through an OpenTelemetry Collector running in the cluster.