Vertical Pod Autoscaler and In-Place Pod Resize

In the previous article the HorizontalPodAutoscaler added Pods when CPU utilization rose. Everything depended on a number you chose by hand: the container's CPU request. Choose it too small and the HPA panics at normal load. Choose it too large and you pay for capacity that sits idle, because the scheduler reserves every requested millicore whether you use it or not.

Most teams guess that number once and never revisit it. The Vertical Pod Autoscaler (VPA) replaces the guess with a measurement. It watches what your containers actually use and recommends, or applies, better requests and limits. Since Kubernetes 1.35 it can apply them to running Pods without restarting them, thanks to in-place Pod resize.

We will cover the following topics:

1. What the VPA's three components do
2. The update modes, from read-only Off to InPlaceOrRecreate
3. Reading a recommendation: target, lower bound, upper bound
4. In-place Pod resize, the resize subresource and resizePolicy
5. What happens when a resize does not fit on the node
6. Why VPA and HPA must not both scale on CPU or memory

Technical Requirements

1. A local multi-node cluster. The examples use Docker Desktop's Kubernetes with the kind provisioner (one control-plane node and three workers), Kubernetes v1.36. In-place resize needs Kubernetes 1.33 or later, and it is stable from 1.35.
2. metrics-server, installed as shown in the previous article.
3. git and OpenSSL 3 or later to install the VPA. On macOS, read the note in Step 1 first.
4. The scaling-api:1.0 image from the previous article.

Core Concepts

Three Components

The VPA is not built into Kubernetes. It is a separate project (kubernetes/autoscaler, version 1.8.0 at the time of writing) made of three Deployments:

The VPA object itself is small. It points at a workload with targetRef and says how much the VPA may do.

Update Modes

spec.updatePolicy.updateMode decides how far the VPA goes:

Off is where every team should start. It costs nothing and turns guessing into data. InPlaceOrRecreate no longer needs a feature gate: it became default-on in VPA 1.5 and the gate was removed in 1.7.

Reading a Recommendation

The recommender publishes four numbers per container:

The updater does not chase every small movement of the target. It acts on a Pod whose requests fall outside the lower and upper bounds. For a Pod that has been running more than 12 hours, it also acts when the requests differ from the target by more than 10%. The bounds are wide when there is little history and narrow as data accumulates over days.

The recommender also has floors: by default it never recommends less than 25m CPU or 250Mi memory per Pod. On a small .NET API you will see memory jump to 250Mi even if the container uses 25Mi. That floor is configurable on the recommender with --pod-recommendation-min-memory-mb and --pod-recommendation-min-cpu-millicores.

Limits Follow Requests

With controlledValues: RequestsAndLimits (the default), the VPA keeps the ratio between limit and request. A container with cpu: 50m request and 100m limit has a 1:2 ratio. If the VPA raises the request to 500m, the limit becomes 1000m. Set controlledValues: RequestsOnly if you want limits left exactly as written.

In-Place Pod Resize

Before Kubernetes 1.33, a Pod's resources were immutable. Changing them meant deleting the Pod and creating a new one. In-place resize changes that:

1. spec.containers[*].resources for CPU and memory is now mutable, but only through the Pod's resize subresource. A plain kubectl edit of the Pod still refuses.
2. The kubelet applies the change through the container runtime by updating the container's cgroup. The process keeps running, with the same Pod name, IP and restart count.
3. status.containerStatuses[*].resources shows what is actually applied. While the two differ, the resize is in progress.

The feature was alpha in 1.27, beta and on by default in 1.33, and stable in 1.35.

Each container can say how it reacts to a resize with resizePolicy:

RestartContainer is useful for memory with runtimes that size themselves at startup. A .NET process reads its memory limit when it starts and sizes the GC heap from it. If you raise the memory limit live, the GC will not use the new headroom until the process restarts. For CPU, NotRequired is almost always right.

When the Resize Does Not Fit

A resize asks the node for more resources. The node may not have them:

A resize never moves a Pod to another node. That is what the VPA's InPlaceOrRecreate fallback is for: if the Pod cannot grow where it is, the updater evicts it and the scheduler places the new Pod somewhere with room.

VPA and HPA Together

Do not let the VPA and the HPA act on the same resource. If the HPA scales on CPU utilization and the VPA raises the CPU request, utilization drops, so the HPA removes Pods. The remaining Pods get busier, so the VPA raises the request again. The two controllers keep reacting to each other.

Safe combinations:

1. VPA in Off mode alongside any HPA. Use it as a sizing report.
2. HPA on a custom or external metric (requests per second, queue length) while the VPA manages CPU and memory.
3. VPA controlling memory only (controlledResources: ["memory"]) while the HPA scales on CPU.

If you would rather browse recommendations than run kubectl describe, Goldilocks creates a VPA in Off mode for every workload in a namespace and shows the results in a dashboard. It is covered later in this section.

Hands-On: Kubernetes Commands

List VPAs with their current recommendation

The CPU and MEM columns are the target values.

Read the full recommendation

Shows target, lower bound, upper bound and uncapped target per container.

Compare desired and actual resources of a Pod

spec is what you asked for. status is what the kubelet has applied.

Resize a running Pod in place

Patching the resize subresource is the only way to change a running Pod's CPU and memory.

See resize progress and problems

Resize conditions appear only while something is pending or in progress.

See what the VPA components are doing

The updater logs every decision it makes.

Step-by-Step Example

The Scenario

You deploy scaling-api with requests you guessed: 50m CPU and 64Mi memory. A steady background load hits it. First you ask the VPA for an opinion without letting it change anything. Then you resize a Pod by hand to see in-place resize at work. Finally you let the VPA resize the Pods itself.

Step 1 — Install the VPA

The project installs with a script from its repository. Pin the release tag:

The script installs the CRDs, RBAC and the three Deployments into kube-system, and generates a self-signed certificate for the admission webhook with openssl.

macOS note. The openssl that ships with macOS is LibreSSL. With it, the script produces a CA certificate without the CA:TRUE basic constraint. The API server then refuses to call the webhook. Because the webhook's failure policy is Ignore, nothing visibly breaks: new Pods are silently created without the recommended values, and the admission controller log fills with this:

Run the script with OpenSSL 3 or later first on the PATH, for example the Homebrew build:

If you already installed with LibreSSL, regenerate the certificate and restart the admission controller:

Check that all three components are running:

Step 2 — Deploy the Application With Guessed Requests

Save as 2-scaling-api.yaml. The resizePolicy allows both CPU and memory to change without a restart, so you can watch the restart count stay at zero.

Step 3 — Ask for a Recommendation With Mode Off

Save as 3-vpa-off.yaml. minAllowed and maxAllowed are guard rails: whatever the data says, the VPA stays inside them.

Within a minute or two the recommender has enough samples for a first opinion:

Three things stand out:

1. The CPU target of 109m is more than double the 50m guess. Under this load, the guess was too low.
2. Memory is 250Mi, which is the recommender's built-in floor, not real usage. kubectl top pods shows the containers using about 25Mi.
3. The bounds are wide (25m to 1 CPU) because the recommender has only a minute of history. With mode Off, nothing changes, and the Pods still run with 50m.

Step 4 — Resize a Pod by Hand

Before handing control to the VPA, do what it will do, by hand. Pick one Pod:

Triple its CPU through the resize subresource:

Read desired and actual resources and the restart count again:

Spec and status agree, and the restart count is still 0. The same process, with the same Pod name and IP, now has three times the CPU. The events record it:

Step 5 — Ask for Too Much

First ask for more CPU than the node has in total. Each worker node in this cluster has 8 CPUs:

The API server rejects it outright, so nothing changes. Now ask for 7.8 CPUs, which fits the node on paper but not next to the Pods already running there:

The patch was accepted, but the kubelet parks it as Deferred and keeps running the Pod at its old size. It will apply the change later if other Pods leave the node. Undo it by patching back to the previous values. The PodResizePending condition disappears:

Step 6 — Let the VPA Resize In Place

Switch the mode. Everything else stays the same:

The updater runs once a minute. Within two loops both Pods change:

Same Pod names, zero restarts, new sizes. The limits kept their original ratios: CPU 1:2 (511m to 1022m) and memory 1:4 (250Mi to 1000Mi). The events say who did it:

The updater log shows its reasoning, including a Pod it chose to leave alone:

Notice that the CPU target kept rising, from 109m to 511m and later past 900m. The load generator is a closed loop: the faster the API answers, the sooner the next request arrives, so more CPU creates more demand. Real traffic rarely behaves like this, but the scenario shows why maxAllowed matters. Without the 1 CPU ceiling the VPA would keep chasing the load.

Step 7 — New Pods Start With the Recommendation

Scale the Deployment up:

The new Pod never runs with the 50m written in the Deployment. The admission controller rewrote it at creation and left an annotation explaining what it changed:

The Deployment's own template still says 50m. The VPA changes Pods, never the Deployment. If you remove the VPA, every new Pod goes back to your original numbers. Once the recommendation has stabilized over a few days, copy it into the manifest.

Step 8 — Clean Up

To remove the VPA itself, run the matching script from the same directory:

Summary

1. The VPA measures real usage and recommends, or applies, requests and limits. It has three parts: the recommender, the updater and the admission controller.
2. Start with updateMode: "Off". It is free and turns a guess into data. Initial, Recreate and InPlaceOrRecreate act progressively more. Auto is deprecated.
3. Recommendations include a target and bounds. The updater acts only when a Pod is outside the bounds. The recommender has floors (25m CPU, 250Mi memory by default), and minAllowed and maxAllowed keep it within limits you trust.
4. In-place Pod resize is stable since Kubernetes 1.35. Change a running Pod through the resize subresource. status.containerStatuses[*].resources shows what is applied, and resizePolicy decides whether a container restarts.
5. A resize larger than the node is rejected. One that does not fit right now is Deferred. It never moves the Pod. InPlaceOrRecreate falls back to eviction for that case.
6. Never let the VPA and the HPA scale on the same resource. Combine VPA Off with any HPA, or let the HPA use custom or external metrics.
7. On macOS, install the VPA with OpenSSL 3+. LibreSSL produces a webhook CA that the API server rejects without any visible error.

CPU and memory are not the only reasons to scale. In the next article we scale a .NET worker on the length of a RabbitMQ queue, all the way down to zero, with KEDA.