Lesson 49 of 66 · 15 min read · Updated

Horizontal Pod Autoscaler with metrics-server

So far every Deployment in this series has had a fixed replicas value. You picked three, Kubernetes kept three, and that was the end of it. Real traffic does not work like that. A .NET API that is idle at 03:00 and saturated at 12:00 either wastes money all night or falls over at lunch.

The HorizontalPodAutoscaler (HPA) fixes this by changing replicas for you. It watches a metric, compares it to a target, and adds or removes Pods to keep the metric near that target. This article builds one from scratch on a local multi-node cluster and watches it scale a CPU-hungry .NET API from one Pod to eight and back again.

We will cover the following topics:

  1. How the HPA control loop works and where its numbers come from
  2. Installing metrics-server, the component that feeds the HPA
  3. Why resource requests are the foundation of utilization-based scaling
  4. Writing an autoscaling/v2 HPA with CPU and memory targets
  5. Tuning scale-up and scale-down speed with the behavior field
  6. Where the Cluster Autoscaler and Karpenter fit in

Technical Requirements

  1. A local multi-node cluster. The examples use Docker Desktop's built-in Kubernetes with the kind provisioner: one control-plane node and three workers (desktop-worker, desktop-worker2, desktop-worker3), Kubernetes v1.36. A plain kind create cluster with one control-plane and three workers behaves the same way.
  2. kubectl v1.36 and Docker.
  3. The .NET 10 SDK if you want to build the sample image yourself.

Core Concepts

The Control Loop

The HPA is a controller inside kube-controller-manager. Every 15 seconds it runs the same loop for each HPA object:

  1. Read the current value of each metric for every Pod of the target workload.
  2. Compute how many replicas would bring that metric back to the target.
  3. Apply the behavior rules that limit how fast it may move.
  4. Write the result to the replicas field of the Deployment.

The core formula is short:

desiredReplicas = ceil( currentReplicas * currentMetricValue / targetMetricValue )

If one Pod is running at 250% CPU utilization and the target is 50%, the HPA wants ceil(1 * 250 / 50) = 5 Pods. When the HPA watches several metrics it computes a replica count for each one and takes the largest. Scaling is driven by whichever resource is under the most pressure.

The HPA never touches Pods directly. It only changes replicas on the Deployment, and the Deployment's ReplicaSet does the rest, exactly as if you had run kubectl scale yourself.

Where the Metrics Come From

The HPA does not measure anything. It asks the Kubernetes metrics APIs:

APIServed byUsed for
metrics.k8s.iometrics-serverCPU and memory of Pods and nodes (type: Resource)
custom.metrics.k8s.ioAn adapter such as Prometheus AdapterMetrics attached to Kubernetes objects, for example requests per second per Pod
external.metrics.k8s.ioAn adapter such as KEDAMetrics from outside the cluster, for example queue length

A fresh cluster serves none of them. Without metrics-server, kubectl top fails and every CPU-based HPA reports <unknown>. Managed clusters (AKS, EKS, GKE) usually install it for you. Local clusters usually do not.

metrics-server scrapes the kubelet on every node, keeps only the latest sample in memory, and serves it through the API. It is deliberately simple. It is not a monitoring system and keeps no history.

Utilization Is Measured Against Requests

This is the single most important rule of CPU and memory autoscaling:

utilization % = actual usage / resource request * 100

The percentage is relative to the container's request, not its limit and not the node's capacity. A container that requests 200m CPU and uses 500m is at 250% utilization. That is not an error. It simply means the container is borrowing spare CPU on the node, up to its limit.

Two consequences follow:

  1. A container with no CPU request cannot be scaled on CPU utilization. The HPA reports failed to get cpu utilization: missing request for cpu.
  2. The request you choose decides how aggressive the HPA is. With a tiny request, normal load already looks like 300% and the HPA scales out early. With a huge request, the HPA waits too long.

If you are unsure what the request should be, the next article on the Vertical Pod Autoscaler shows how to measure it.

Target Types

An autoscaling/v2 metric can use three kinds of target:

Target typeMeaningExample
UtilizationAverage percentage of the request across PodsaverageUtilization: 50
AverageValueAverage absolute value per PodaverageValue: 300m
ValueOne total value for the whole workloadvalue: "100" for an external metric

CPU and memory usually use Utilization. Memory is a weaker scaling signal than CPU for .NET: the garbage collector holds on to memory it has already allocated, so memory often does not drop when load drops. Treat a memory target as a safety net, not as the main driver.

The behavior Field

Without tuning, the HPA scales up quickly (by default it may double the replica count or add four Pods every 15 seconds, whichever is more) and scales down cautiously. The behavior field lets you set both directions explicitly:

FieldPurpose
stabilizationWindowSecondsLook back over this window and use the most conservative recommendation. It prevents flapping. The default is 0 for scale-up and 300 for scale-down.
policiesLimits on how much can change in one periodSeconds, either as a number of Pods or as a Percent of current replicas.
selectPolicyMax picks the policy allowing the biggest change, Min the smallest, and Disabled turns that direction off entirely.

A common production shape is fast up, slow down. Add capacity as soon as it is needed, and remove it one Pod at a time once the load has clearly gone.

What the HPA Cannot Do

The HPA adds Pods. It does not add nodes. If the new Pods do not fit, they stay Pending and the HPA has done its job but nothing improves. On a cloud cluster a node autoscaler closes this gap:

  1. Cluster Autoscaler watches for Pods that are Pending because of insufficient resources and grows a node group (an AKS node pool, an EKS managed node group). When nodes sit underused, it drains and removes them.
  2. Karpenter works on the same trigger but skips node groups. It looks at the pending Pods' requirements and launches whichever instance type fits them best. It started on AWS and also has an Azure provider.

Neither can run on a local kind cluster, because there is no cloud API to create machines. The principle still matters: Pod autoscaling and node autoscaling are two separate layers, and production needs both.

Hands-On: Kubernetes Commands

Check that the metrics API is available

If this returns NotFound, metrics-server is not installed.

kubectl get apiservice v1beta1.metrics.k8s.io

See live CPU and memory per node and per Pod

These are the same numbers the HPA reads.

kubectl top nodes
kubectl top pods -n as-hpa

Create an HPA imperatively

Handy for experiments. In kubectl v1.36 the flags are --cpu and --memory, which accept a percentage or an absolute value.

kubectl autoscale deployment scaling-api -n as-hpa --cpu=50% --min=1 --max=8

Generate the YAML instead of applying it

The safest way to learn the autoscaling/v2 shape.

kubectl autoscale deployment scaling-api -n as-hpa --cpu=50% --min=1 --max=8 --dry-run=client -o yaml

Watch the HPA make decisions

The TARGETS column shows current value against target for every metric.

kubectl get hpa -n as-hpa -w

Read the HPA's reasoning

The Conditions and Events sections explain every scaling decision and every failure.

kubectl describe hpa scaling-api -n as-hpa

Step-by-Step Example

The Scenario

You run scaling-api, a .NET 10 minimal API with a /burn endpoint that spends a configurable number of milliseconds of pure CPU per request. It stands in for any CPU-heavy work: image resizing, PDF generation, JSON transformation. You want it to run on one Pod when idle and grow to at most eight under load.

Step 1 — Install metrics-server

The official manifest works on any cluster. kind's kubelets use self-signed serving certificates, so metrics-server must be told not to verify them with --kubelet-insecure-tls. Never use this flag on a production cluster with properly signed kubelet certificates.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl -n kube-system patch deployment metrics-server --type=json \
-p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
kubectl -n kube-system rollout status deploy/metrics-server

After about 20 seconds the first samples arrive:

kubectl top nodes
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
desktop-control-plane 520m 6% 1694Mi 10%
desktop-worker 317m 3% 918Mi 5%
desktop-worker2 614m 7% 1641Mi 10%
desktop-worker3 225m 2% 768Mi 4%

Step 2 — Build the Application

Create the project:

dotnet new web -n ScalingApi
cd ScalingApi

Replace Program.cs:

using System.Diagnostics;

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

string podName = Environment.GetEnvironmentVariable("POD_NAME") ?? Environment.MachineName;

app.MapGet("/", () => Results.Ok(new { servedByPod = podName }));

// Burns CPU for the requested number of milliseconds on the request thread.
app.MapGet("/burn", (int? ms) =>
{
int duration = Math.Clamp(ms ?? 100, 1, 2000);
var sw = Stopwatch.StartNew();
double x = 0;
while (sw.ElapsedMilliseconds < duration)
{
x += Math.Sqrt(x + 1);
}
return Results.Ok(new { servedByPod = podName, burnedMs = duration });
});

app.MapGet("/healthz", () => Results.Ok("ok"));

app.Run();

Add a Dockerfile and build:

FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY . .
RUN dotnet publish ScalingApi.csproj -c Release -o /app

FROM mcr.microsoft.com/dotnet/aspnet:10.0
WORKDIR /app
COPY --from=build /app .
ENV ASPNETCORE_HTTP_PORTS=8080
EXPOSE 8080
ENTRYPOINT ["dotnet", "ScalingApi.dll"]
docker build -t scaling-api:1.0 .

Docker Desktop's kind cluster can use images from your local Docker image store directly. The first Pod on each node may sit in ContainerCreating for a minute while the node fetches the image. With a standalone kind cluster, load the image first with kind load docker-image scaling-api:1.0.

Step 3 — Deploy With Requests

Save as 2-scaling-api.yaml. The requests block is what makes CPU utilization meaningful.

apiVersion: v1
kind: Namespace
metadata:
name: as-hpa
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: scaling-api
namespace: as-hpa
spec:
replicas: 1
selector:
matchLabels:
app: scaling-api
template:
metadata:
labels:
app: scaling-api
spec:
containers:
- name: scaling-api
image: scaling-api:1.0
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
resources:
requests:
cpu: 200m # 100% utilization = 200m of real CPU
memory: 128Mi
limits:
cpu: 500m
memory: 256Mi
readinessProbe:
httpGet:
path: /healthz
port: http
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: scaling-api
namespace: as-hpa
spec:
selector:
app: scaling-api
ports:
- name: http
port: 80
targetPort: http
kubectl apply -f 2-scaling-api.yaml
kubectl -n as-hpa rollout status deploy/scaling-api

Notice that replicas: 1 is still in the manifest. Once an HPA owns the Deployment, the HPA's value wins. If you later re-apply this file, replicas: 1 will briefly fight the HPA. Many teams remove replicas from the manifest once an HPA exists for exactly that reason.

Step 4 — Create the HPA

Save as 3-hpa.yaml:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: scaling-api
namespace: as-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: scaling-api
minReplicas: 1
maxReplicas: 8
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
behavior:
scaleUp:
stabilizationWindowSeconds: 0
selectPolicy: Max
policies:
- type: Percent
value: 100 # may double the replica count...
periodSeconds: 15
- type: Pods
value: 2 # ...or add 2 Pods, whichever is more
periodSeconds: 15
scaleDown:
stabilizationWindowSeconds: 60
selectPolicy: Min
policies:
- type: Pods
value: 1 # remove at most one Pod...
periodSeconds: 30 # ...every 30 seconds
kubectl apply -f 3-hpa.yaml
kubectl get hpa -n as-hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
scaling-api Deployment/scaling-api cpu: 2%/50%, memory: 15%/80% 1 8 1 31s

The idle API uses about 2% of its 200m CPU request and 15% of its 128Mi memory request. Both are under target, so one replica is correct.

Step 5 — Generate Load

Save as 4-load-generator.yaml. Four busybox Pods call /burn?ms=200 in a tight loop through the Service.

apiVersion: apps/v1
kind: Deployment
metadata:
name: load-generator
namespace: as-hpa
spec:
replicas: 4
selector:
matchLabels:
app: load-generator
template:
metadata:
labels:
app: load-generator
spec:
containers:
- name: load
image: busybox:1.37
command: ["/bin/sh", "-c"]
args:
- while true; do wget -q -O- "http://scaling-api/burn?ms=200" > /dev/null; done
resources:
requests:
cpu: 10m
memory: 16Mi

Open a second terminal and start watching before you apply it:

kubectl get hpa scaling-api -n as-hpa -w
kubectl apply -f 4-load-generator.yaml

Step 6 — Watch It Scale Out

This is the real output from the test cluster, with the REFERENCE, MINPODS and MAXPODS columns removed:

TARGETS REPLICAS AGE
cpu: 2%/50%, memory: 15%/80% 1 36s
cpu: 17%/50%, memory: 16%/80% 1 45s
cpu: 250%/50%, memory: 20%/80% 1 60s
cpu: 249%/50%, memory: 21%/80% 3 76s
cpu: 241%/50%, memory: 22%/80% 5 91s
cpu: 172%/50%, memory: 21%/80% 5 106s
cpu: 181%/50%, memory: 24%/80% 8 2m1s
cpu: 149%/50%, memory: 23%/80% 8 2m16s
cpu: 157%/50%, memory: 25%/80% 8 3m1s

Read it line by line:

  1. At 250%, the formula asks for ceil(1 * 250 / 50) = 5 Pods. The scale-up policies allow at most max(100% of 1, 2) = 2 extra Pods per 15 seconds, so the HPA goes to 3.
  2. Fifteen seconds later the formula still wants more. Now the policies allow max(100% of 3, 2) = 3 extra Pods, but only 5 are needed, so it goes to 5.
  3. It reaches 8, the maxReplicas ceiling, and stops. CPU stays above target because the load generator is relentless, but the HPA is not allowed to go higher.

The describe output confirms it:

kubectl describe hpa scaling-api -n as-hpa
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited True TooManyReplicas the desired replica count is more than the maximum replica count
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 3m4s horizontal-pod-autoscaler failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
Normal SuccessfulRescale 2m19s horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 2m3s horizontal-pod-autoscaler New size: 5; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 93s horizontal-pod-autoscaler New size: 8; reason: cpu resource utilization (percentage of request) above target

Two details are worth noticing:

  1. ScalingLimited: TooManyReplicas tells you the HPA wanted more than maxReplicas. In production, that condition is a signal to raise the ceiling or to make each Pod more efficient.
  2. The early FailedGetResourceMetric warning appeared in the first seconds after the HPA was created, before metrics-server had a sample for the Pod. It is harmless at startup. If it persists, check that metrics-server is healthy and that the Pods are Ready.

kubectl get pods -o wide shows the new Pods spread across the three worker nodes. That spread is the scheduler's job, not the HPA's.

Step 7 — Watch It Scale In

Stop the load:

kubectl delete deployment load-generator -n as-hpa

CPU drops to almost zero within a minute, but the replica count does not move straight away:

TARGETS REPLICAS AGE
cpu: 87%/50%, memory: 21%/80% 8 4m16s
cpu: 1%/50%, memory: 22%/80% 8 4m31s
cpu: 0%/50%, memory: 22%/80% 8 5m1s

The 60-second stabilization window is holding the old, higher recommendation. After that, the scale-down policy removes one Pod every 30 seconds:

Normal SuccessfulRescale 4m15s horizontal-pod-autoscaler New size: 7; reason: All metrics below target
Normal SuccessfulRescale 3m45s horizontal-pod-autoscaler New size: 6; reason: All metrics below target
Normal SuccessfulRescale 3m15s horizontal-pod-autoscaler New size: 5; reason: All metrics below target
Normal SuccessfulRescale 2m45s horizontal-pod-autoscaler New size: 4; reason: All metrics below target
Normal SuccessfulRescale 2m15s horizontal-pod-autoscaler New size: 3; reason: All metrics below target
Normal SuccessfulRescale 105s horizontal-pod-autoscaler New size: 2; reason: All metrics below target
Normal SuccessfulRescale 75s horizontal-pod-autoscaler (combined from similar events): New size: 1; reason: All metrics below target
kubectl get hpa -n as-hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
scaling-api Deployment/scaling-api cpu: 0%/50%, memory: 26%/80% 1 8 1 9m31s

Going from 8 to 1 took about five minutes. That is deliberate. If traffic comes back after a short pause, the Pods are still there. With the default scaleDown behavior (a 300-second window), the wait would be even longer.

Memory utilization ended slightly higher than it started (26% against 15%) even with no load. That is the .NET garbage collector keeping its heap after the burst, and it is the reason memory alone is a poor scale-down signal.

Step 8 — Clean Up

Deleting the namespace removes the Deployment, the Service and the HPA together:

kubectl delete namespace as-hpa

Keep metrics-server installed if you continue with the next article. To remove it:

kubectl delete -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Summary

  1. The HPA changes replicas on a Deployment so that a metric stays near a target. The formula is ceil(currentReplicas * current / target), evaluated every 15 seconds, and the largest result across metrics wins.
  2. CPU and memory metrics come from metrics-server, which local clusters usually lack. On kind it needs --kubelet-insecure-tls.
  3. Utilization is a percentage of the container's request. No request means no utilization-based scaling, and the size of the request sets how aggressive the HPA is.
  4. behavior controls speed: stabilizationWindowSeconds prevents flapping, policies cap each step, and selectPolicy chooses between them. Fast up and slow down is a sound default.
  5. Memory is a weak scaling signal for .NET because the GC keeps its heap. Use CPU, or a business metric, as the main driver.
  6. The HPA scales Pods, not nodes. On a cloud cluster, the Cluster Autoscaler or Karpenter adds the machines those Pods need.

The HPA changes how many Pods you run. In the next article we look at the other dimension, how big each Pod is, with the Vertical Pod Autoscaler and in-place Pod resize.