Node Autoscaling with Cluster Autoscaler and Karpenter

The previous lessons in this section taught the cluster to resize workloads. The HorizontalPodAutoscaler adds replicas, the Vertical Pod Autoscaler grows requests, and KEDA scales on queue length. All three share one blind spot: they create or enlarge Pods, but they never create a single byte of capacity. When the nodes are full, the new Pods sit in Pending and the autoscaler that created them has done everything it can.

Node autoscaling closes that gap. A node autoscaler watches for Pods that cannot be scheduled, adds machines that can hold them, and removes machines again when they are no longer needed. In this lesson you run the real Kubernetes Cluster Autoscaler on a local kind cluster. KWOK supplies the "cloud", so you can watch it add and remove nodes in seconds without a cloud bill. You will also see how Karpenter approaches the same problem differently, and how the managed clouds package both.

We will cover the following topics:

1. Why pod autoscalers need a node autoscaler, and why Pending Pods are the trigger
2. How Cluster Autoscaler works: node groups, scheduling simulation, expanders, and scale-down based on the utilization threshold and unneeded time
3. What blocks scale-down: PodDisruptionBudgets, the safe-to-evict annotation, Pods that cannot move, and nodes marked scale-down-disabled
4. How Karpenter differs: NodePools, NodeClasses, provisioning from Pod requirements, consolidation and disruption budgets
5. Why requests drive every decision, not actual usage
6. Overprovisioning with low-priority placeholder Pods to hide node start-up time
7. How AKS, EKS and GKE expose node autoscaling

Technical Requirements

1. Docker Desktop (or Docker Engine) with about 2 CPUs and 3 GB of memory free.
2. The kind CLI. The examples use kind v0.33.0 with the node image kindest/node:v1.36.4, so the cluster runs Kubernetes v1.36 like the rest of this series.
3. kubectl v1.36 and Helm 4 (the commands were run with Helm v4.3.1).
4. The .NET 10 SDK (10.0.402 was used) to build the sample API.
5. Internet access to pull the KWOK v0.8.0 manifests and the Cluster Autoscaler v1.36.1 image.

Why a separate cluster this time? Every other lesson in this series uses Docker Desktop's built-in Kubernetes with the kind provisioner. This lesson does not, for two reasons. First, the cluster autoscaler is a cluster-wide controller. It evaluates every node and every Pending Pod in the cluster, so it would also react to whatever else you have running on your main cluster. Second, the KWOK-based node provider registers fake Node objects and deletes them again. You do not want either of those things on the cluster you use every day. A disposable kind cluster created with the kind CLI, as shown in the lesson "Setting Up a Local Multi-Node Cluster with kind and Docker Desktop", keeps the experiment isolated, and you remove it with a single command at the end. The lesson also writes its kubeconfig to a separate file, so your default context is never changed.

Core Concepts

Pod Autoscalers Create Demand, Node Autoscalers Create Supply

Every autoscaler you have met so far changes demand. The HPA increases replicas, the VPA raises requests, KEDA turns a queue length into replicas. The scheduler then has to find room for that demand. When no node has enough unrequested CPU or memory, the scheduler gives up on the Pod and records the reason:

The Pod stays Pending with the condition PodScheduled=False and reason Unschedulable. A node autoscaler is the controller that reacts to exactly this state. It does not watch CPU graphs, request rates or queue lengths. Its only scale-up signal is: there are Pods the scheduler could not place, and new nodes would let them be placed.

When traffic arrives, the HPA adds Pods, some go Pending, and the node autoscaler adds a node. When traffic leaves, the HPA removes Pods, some nodes become nearly empty, and the node autoscaler drains and removes them.

How Cluster Autoscaler Is Organized

Cluster Autoscaler (CA) is a SIG Autoscaling project that runs as a Deployment, usually in kube-system. It does not create machines itself. A cloud provider implementation (AWS, Azure, GCP, Cluster API, Hetzner and many more) resizes groups of identical machines for it.

The central idea is the node group: nodes with the same shape, labels and taints that the cloud can grow or shrink, such as an AKS node pool, an EKS managed node group or a GKE node pool. Each group has a minimum and maximum size, and CA never leaves those bounds. Because all nodes in a group are identical, CA describes a node that does not exist yet with a template node: its labels, taints and allocatable resources. Every decision is a scheduler simulation against these templates.

Scale-Up: Simulate, Choose, Resize

CA runs a loop every --scan-interval (10 seconds by default). On each pass it:

1. Lists Pods that are unschedulable.
2. Filters out Pods that would fit on existing nodes anyway, for example on a node that just became ready.
3. For each node group, runs the scheduler's own predicates against that group's template node: resource fit, node selectors and affinity, taints and tolerations, topology spread and so on.
4. Estimates how many new nodes each candidate group would need to hold the Pending Pods.
5. Asks an expander to pick the best group, then increases that group's target size.

Step 3 is what makes CA accurate. It uses the same scheduling framework as kube-scheduler, so a Pod with nodeSelector: gpu=true, or one that does not tolerate a group's taint, never triggers a group that could not run it.

When several groups could help, the expander decides:

Expanders can be chained, for example --expander=priority,least-waste, where the second breaks ties left by the first. This lesson uses least-waste, and the logs show the waste percentages CA calculated for each candidate.

If a group is already at its maximum, CA skips it and tries the next one. If no group can help, the Pod gets a NotTriggerScaleUp event that says why.

Scale-Down: Underused, Movable, and Unneeded for Long Enough

Removing nodes is the harder half, because it means evicting running Pods. CA treats a node as a scale-down candidate only when all of the following hold:

1. Utilization is low. The sum of the requests of the Pods on the node, divided by its allocatable resources, is below --scale-down-utilization-threshold (default 0.5) for both CPU and memory. DaemonSet Pods count towards the requests, but CA knows they are recreated on every node and does not try to move them.
2. Every Pod can move. CA simulates removing the node and re-runs the scheduler for each of its Pods against the remaining nodes. If even one Pod has nowhere to go, the node stays.
3. Nothing blocks eviction. The next section lists what counts as a blocker.
4. It has been unneeded for long enough. The node must stay a candidate for --scale-down-unneeded-time (default 10 minutes). A node is also not considered for --scale-down-delay-after-add (default 10 minutes) after any scale-up.

CA then taints the node, evicts its Pods (respecting PodDisruptionBudgets and grace periods) and asks the cloud to delete the machine. Empty nodes go in bulk, non-empty ones a few at a time.

One more timer matters in practice. When CA decides a node is unremovable, for example because its Pods cannot move, it does not check that node again for --unremovable-node-recheck-timeout (default 5 minutes). In the Step-by-Step example you will see a node linger for exactly that long after its blocker was removed.

What Blocks Scale-Down

The opposite annotation, cluster-autoscaler.kubernetes.io/safe-to-evict: "true", tells CA that a Pod which would normally block (for example one with an emptyDir) may be evicted. The PodDisruptionBudget rules are the same ones kubectl drain follows, which the lesson "PriorityClass, Preemption and PodDisruptionBudgets" covers in detail.

Requests Drive Every Decision, Not Usage

Cluster Autoscaler never looks at real CPU or memory usage. It does not need metrics-server. Both directions are pure bookkeeping on requests:

1. Scale-up happens when the requests of Pending Pods do not fit into any node's free allocatable. A Pod that requests 100m but burns four cores will never cause a scale-up, it just competes for CPU on its node.
2. Scale-down compares the sum of requests on a node with the threshold. A node full of idle Pods that each request one core looks 95% "utilized" and will never be removed.

Requests that are too high waste nodes, and requests that are too low pack nodes so tightly that you get throttling and OOM kills no autoscaler will fix. That is why the Vertical Pod Autoscaler and Goldilocks lessons come first: measure usage, set requests, and only then let a node autoscaler act on them.

Karpenter: Provisioning From Pod Requirements

Karpenter, originally from AWS and now a Kubernetes SIG Autoscaling project, solves the same problem without node groups. Instead of picking which existing group to grow, it reads the Pending Pods' requirements directly (resources, node selectors, affinity, topology spread, tolerations) and asks the cloud for a machine that satisfies them. It can choose any instance type the provider offers, so one Pending Pod might get a small arm64 spot instance and the next a large x86 on-demand one.

Karpenter's API (v1) has two main resources:

1. NodePool: what kind of nodes may be created. It holds requirements (instance families, architectures, capacity type such as spot or on-demand, zones), the labels and taints new nodes get, overall limits such as total CPU, and the disruption policy.
2. NodeClass: provider-specific settings such as the machine image, subnets and security groups. On AWS it is called EC2NodeClass, on Azure AKSNodeClass.

A NodePool looks like this. It is shown for reference only and is not applied in this lesson:

Instead of a utilization threshold, Karpenter uses consolidation: it keeps asking whether the same Pods could run on fewer or cheaper nodes. Disruption budgets cap how many nodes it may disrupt at once, optionally only in cron-scheduled windows, and expireAfter recycles nodes after a maximum lifetime.

Cluster Autoscaler fits a few well-known node shapes on almost any infrastructure. Karpenter fits when the cluster should choose from many instance types, use spot capacity and keep packing nodes tighter.

Overprovisioning: Paying for Headroom

On a real cloud, a new node takes one to four minutes to boot, join the cluster and pull images. During that time new Pods are Pending, which means requests are queuing or failing. Overprovisioning hides this delay by keeping some spare capacity warm:

1. Create a PriorityClass with a negative value, lower than every real workload.
2. Run a Deployment of placeholder Pods (the pause image) in that class with large requests. CA adds nodes for them like for any other Pending Pod.
3. When real Pods arrive and do not fit, the scheduler preempts the placeholders, because they have lower priority. The real Pods start immediately on capacity that already exists.
4. The evicted placeholders become Pending and trigger a scale-up in the background, so the headroom is restored.

The size of the placeholder Deployment is how much headroom you pay for. Teams often scale it with a CronJob (more during business hours) or with the cluster-proportional-autoscaler. One detail matters: CA ignores Pods whose priority is below --expendable-pods-priority-cutoff (default -10). Such Pods never trigger a scale-up. Keep the placeholder priority at -10 or higher, for example -5.

KWOK and the kwok Cloud Provider

KWOK (Kubernetes WithOut Kubelet) is a SIG Scheduling toolkit that simulates nodes and Pods. Its controller keeps Node objects annotated kwok.x-k8s.io/node: fake alive with heartbeats and moves Pods bound to them through their lifecycle. No container is ever started.

Cluster Autoscaler ships a kwok cloud provider that uses KWOK as its "cloud". Node groups come from template nodes in a ConfigMap, scale-up creates fake Node objects from them, and scale-down deletes them. The loop, the scheduling simulation, the expanders, the scale-down rules and the events are the real Cluster Autoscaler. Keep the limits in mind:

1. Pods on KWOK nodes are simulated. They report Running, but kubectl logs and kubectl exec fail and the container status has no imageID.
2. New nodes appear within one CA loop, so the one-to-four-minute boot time of a real cloud is missing.
3. The provider taints its nodes with kwok-provider=true:NoSchedule, so only Pods that tolerate it land there.

That is why the real .NET API in this lesson runs on the real kind worker, and a second Deployment of the same image, which tolerates the taint, plays the burst that forces the cluster to grow.

Managed Clouds

On all of them the controller runs inside the provider's control plane, so you do not deploy or upgrade it yourself. The concepts in this lesson still apply: Pending Pods trigger scale-up, requests decide everything, and PDBs and annotations control scale-down.

Hands-On: Kubernetes Commands

The commands below assume KUBECONFIG points at the lab cluster's kubeconfig file, as set in Step 1.

Find Pods that are waiting for capacity

Pending Pods are the only scale-up trigger, so this is the first thing to check.

See why a Pod is Pending and what the autoscaler did about it

The scheduler writes FailedScheduling. Cluster Autoscaler adds TriggeredScaleUp or NotTriggerScaleUp.

Read the autoscaler's own status report

Cluster Autoscaler keeps a ConfigMap with the health, size and last activity of every node group.

Follow its decisions in the log

At verbosity 4 the log shows waste calculations, the chosen group and every scale-down simulation.

Check how full a node is in terms of requests

This is the number Cluster Autoscaler compares with the utilization threshold.

Protect a node or a Pod from scale-down

For Pods, put cluster-autoscaler.kubernetes.io/safe-to-evict: "false" in the Pod template's annotations, not on a running Pod, so it survives restarts.

Step-by-Step Example

The Scenario

You run a .NET 10 checkout-api on a cluster with one control-plane node and one worker. The real API Pods stay on the real worker and serve traffic. A second Deployment of the same image, checkout-burst, represents a spike in demand that only fits on autoscaled capacity. Cluster Autoscaler, with the kwok provider, can add two kinds of simulated nodes:

You will watch it scale up, choose between groups, hit its limits, scale down, respect blockers and keep warm headroom with placeholder Pods. The timers are shortened to one minute so the whole lesson runs in about half an hour.

Step 1 — Create the Lab Cluster

Save the cluster definition as kind-ca-lab.yaml. Pinning the node image keeps the cluster on Kubernetes v1.36, which matches Cluster Autoscaler v1.36. kind v0.33.0 would otherwise default to v1.37.

Create it with its own kubeconfig file and point this shell at it:

Your Docker Desktop cluster and your default kubeconfig are untouched. Every command from here on uses the KUBECONFIG you just exported.

Step 2 — Build the .NET 10 API

Create the project:

Replace Program.cs with a small API that reports which Pod and node served the request:

The project file stays as the template created it, targeting net10.0 with no extra packages. Next to the project folder, add a multi-stage Dockerfile that publishes into a chiseled runtime image:

Build it and load it into the kind nodes. A cluster created with the kind CLI cannot see images in your local Docker image store until you load them:

Step 3 — Deploy the Real API on the Real Worker

1-namespace-and-api.yaml pins the API to ca-lab-worker and gives it proper requests. These Pods actually run, so this is the part of the system that serves traffic.

Step 4 — Install KWOK

KWOK v0.8.0 consists of a controller and a set of stages, the rules that move simulated nodes and Pods through their lifecycle. stage-fast.yaml makes everything happen immediately.

Step 5 — Install Cluster Autoscaler With the kwok Provider

The official Helm chart (cluster-autoscaler 9.59.0) supports cloudProvider: kwok. Its default app version is 1.35, so the values below set the image to v1.36.1 to match the cluster. The scale-down timers drop from 10 minutes to 1 minute for the lab, and v: 4 makes the log show the decisions. replicaCount: 0 installs everything but keeps the autoscaler stopped until you have replaced the chart's sample node templates with your own.

The chart creates two ConfigMaps: kwok-provider-config (how the provider behaves) and kwok-provider-templates (the node shapes). Replace the provider config with one that groups template nodes by the kwok-nodegroup label. Set install: false, because KWOK is already installed and the provider would otherwise download and apply the latest KWOK release itself:

Now define the two node groups in node-templates.yaml, a List of Node objects. The kwok-nodegroup label names the group, the annotations set its minimum and maximum size, node-pool: burst is the label your workloads select, and status.allocatable is what the scheduling simulation fits Pods into. The second item, burst-large, is identical except for its name and kwok-nodegroup label, instance-type: kwok-8c-16g, max-count: "2", and cpu: "8" and memory: 16Gi in both allocatable and capacity:

Swap the ConfigMaps and start the autoscaler:

The two real nodes belong to no node group, so the autoscaler never touches them, just like a non-autoscaled system node pool.

Step 6 — Create Demand That Does Not Fit

checkout-burst runs the same image, but it selects node-pool: burst and tolerates the taint the kwok provider puts on its nodes. Each replica requests 900m CPU and 1 Gi of memory, so two fit on a 2-CPU node.

Apply it and scale to three replicas:

No node has the node-pool: burst label yet, so the scheduler rejects all three Pods. The next autoscaler loop picks them up. Its log shows the whole decision, including the waste each group would cause:

Each Pod also gets a TriggeredScaleUp event with the same plan, [{burst-small 0->2 (max: 4)}], right after its FailedScheduling event. About ten seconds after the scale, the cluster has two new nodes and all Pods are bound:

Look closely at the new nodes. Their version is fake, each carries the kwok-provider=true:NoSchedule taint, and the provider ID is kwok:<node-name>. The "Running" Pods on them are simulated. Nothing pulled the image, and there is no kubelet to serve logs:

That is the trade-off of this lab. The scheduling, the autoscaling decisions and the Kubernetes objects are all real. The machines and containers are not.

Step 7 — Watch the Expander Choose and Hit the Limits

Scale the burst to twelve replicas. That is 10.8 CPU of requests, more than the small group can hold at its maximum of four nodes (eight Pods).

In the same loop, the autoscaler grew the small group to its maximum and then used the large group for the Pods that were left. The result is four Pods on the large node and two on each small node:

Now push past both maximums with thirty replicas:

Four small nodes hold 8 Pods and two large nodes hold 16, so 6 Pods stay Pending. The event names the reason: both groups are at their maximum. In production, this is the event to alert on. It means your maximums, or your cloud quota, are too low for the demand.

Step 8 — Scale Down and Read the Simulation

Drop to four replicas. The Deployment controller picks which Pods to delete, and the remaining four end up on three of the small nodes:

That leaves three nodes empty: burst-small-mp7cl and both large nodes. One minute later (scale-down-unneeded-time), the autoscaler acts. The log shows every removal:

Two empty nodes went in bulk. burst-small-h4lbq was not empty. It ran one 900m Pod plus the 100m kindnet DaemonSet Pod, which is 50% of 2 CPUs and not above the threshold. Its Pod could move to the empty large node burst-large-dq6px, so the autoscaler drained it and kept dq6px instead.

Each of the three nodes also gets a ScaleDown event, marked the node as toBeDeleted/unschedulable. The decisions on the nodes that still have Pods are the instructive ones. The log at verbosity 4 shows the simulation for each:

Three different outcomes are visible here:

1. burst-small-p952j, at 50% like h4lbq before it, could move its Pod to the large node. After a minute of being unneeded, it was drained and deleted at 12:11:30.
2. burst-large-dq6px was far below the threshold, but the simulation found no other node for its Pods, because the only other burst nodes were full or about to be removed. Low utilization alone is never enough.
3. burst-small-lkzp6 stayed because of utilization:

The final state is two nodes holding four Pods: two on burst-small-lkzp6 and two on burst-large-dq6px. The scale-down did not repack Pods onto the smallest possible set of nodes. It removed nodes only where the simulation proved the Pods could move. That conservative, one-node-at-a-time behaviour is the main practical difference from Karpenter's consolidation.

Step 9 — Block Scale-Down With an Annotation and a PodDisruptionBudget

Add a small reporting Deployment whose Pod must not be interrupted. 4-checkout-report.yaml is a copy of checkout-burst with three changes: the name and label checkout-report, one replica, and requests of 200m CPU and 256Mi memory with no memory limit. The important part is the annotation in the Pod template:

The report Pod landed on burst-large-dq6px. With the burst gone, the small node becomes empty and is removed. The large node, which now uses only 300m of 8 CPUs, stays:

Now swap the annotation for a PodDisruptionBudget, the more common way to protect a workload. With one replica and minAvailable: 1, no voluntary disruption is allowed:

At the scheduled re-check, the reason changes:

A single replica with minAvailable: 1 pins its node forever, usually by accident. Give such workloads two replicas or maxUnavailable: 1.

Remove both and time how long the empty node lingers:

The Deployment was deleted at 12:22:29, but the node went at 12:27:21:

The PDB check at 12:21:21 had marked the node unremovable again, and Cluster Autoscaler does not look at an unremovable node until --unremovable-node-recheck-timeout (5 minutes) has passed. Add the one-minute unneeded time and you get 12:27:21. When a node lingers after you have fixed its blocker, this timer is usually why.

Step 10 — Keep Warm Headroom With Placeholder Pods

The burst pool is empty again. Create a negative-priority class and two placeholder Pods that each reserve almost a whole small node:

preemptionPolicy: Never means the placeholders never push anything else out, they only wait. The value -5 is above the default --expendable-pods-priority-cutoff of -10, so they still trigger a scale-up:

Two nodes now exist that hold no real work, which is the headroom you pay for. Send a real burst of two replicas and check four seconds later:

The real Pods did not wait for a node. The scheduler preempted a placeholder and put both burst Pods on its node right away. The placeholder's replacement is the one that is now Pending, and it triggers the scale-up in the background:

On KWOK this saves seconds. On a real cloud it saves the minutes a VM needs to boot.

The node that now runs the burst shows what the autoscaler sees. Two 900m Pods plus the kindnet DaemonSet make 1900m, which is 95% in request terms, whatever the Pods actually use:

Step 11 — Exempt a Node From Scale-Down

Sometimes a node must stay, for example while you investigate a problem on it. Annotate one of the three burst nodes, then remove all burst demand:

A minute later the other two nodes are gone, and the log explains the survivor:

After kubectl annotate node burst-small-9tg59 cluster-autoscaler.kubernetes.io/scale-down-disabled- it was removed about a minute later, leaving only ca-lab-control-plane and ca-lab-worker. Through all of this the real API never moved. A final curl http://checkout-api/cart from inside the cluster was still answered by checkout-api-8499f4c6b6-df9gv on the real worker.

Step 12 — Clean Up

The whole lab lives in one disposable cluster, so one command removes it, including KWOK, Cluster Autoscaler and every fake node:

Summary

1. Pod autoscalers create demand, node autoscalers create supply. The only scale-up trigger is a Pending Pod that new nodes would let the scheduler place.
2. Cluster Autoscaler grows pre-defined node groups within their min and max. It simulates the scheduler against each group's template, lets an expander such as least-waste choose, and explains every refusal with a NotTriggerScaleUp event.
3. Scale-down needs low request utilization (below 0.5 by default), Pods that can move, no blockers, and scale-down-unneeded-time to pass. Unremovable nodes are re-checked only after 5 minutes.
4. safe-to-evict: "false", PodDisruptionBudgets with zero allowed disruptions and scale-down-disabled keep nodes alive. A single replica with minAvailable: 1 pins its node forever.
5. Every decision is based on requests, never on real usage, so right-size requests with VPA or Goldilocks first.
6. Karpenter launches nodes that match Pending Pods' requirements through NodePools and NodeClasses, and shrinks the cluster with consolidation limited by disruption budgets.
7. Overprovisioning with negative-priority placeholder Pods gives real Pods instant capacity while the slow node scale-up happens in the background.
8. KWOK with the kwok provider runs the real Cluster Autoscaler on a laptop, with simulated nodes and Pods.