PriorityClass, Preemption and PodDisruptionBudgets

The previous two articles were about where a Pod should run. This one is about two harder questions. What happens when there is no room left, and which Pod has to give way? And what happens when you deliberately take a node away for maintenance, and how do you stop that from taking your API down?

The answers are three objects that work closely together. A PriorityClass ranks Pods. Preemption lets the scheduler evict lower-ranked Pods to make room for a higher-ranked one. A PodDisruptionBudget limits how many Pods of an application voluntary operations such as kubectl drain may take away at the same time.

We will cover the following topics:

1. What PriorityClass objects are, and the built-in system-cluster-critical and system-node-critical classes
2. How preemption works, and how to see it happen without overloading a real cluster
3. preemptionPolicy: Never for workloads that should jump the queue but never evict anyone
4. PodDisruptionBudgets with minAvailable and maxUnavailable
5. The difference between evicting a Pod and deleting it
6. Running kubectl drain safely, and why it sometimes hangs
7. unhealthyPodEvictionPolicy for budgets that block maintenance forever

Technical Requirements

1. A multi-node cluster. The examples use Docker Desktop's Kubernetes with the kind provisioner (desktop-control-plane plus desktop-worker, desktop-worker2, desktop-worker3), Kubernetes v1.36. A plain kind cluster with three workers behaves the same with kind-* node names.
2. kubectl with cluster-admin rights: PriorityClasses are cluster-scoped, and the preemption demo patches a node's status.
3. The node-info-api:1.0 image from the first article in this section, or any small image that keeps running.

Core Concepts

PriorityClass

A PriorityClass is a cluster-scoped object that maps a name to an integer. Pods reference it by name in spec.priorityClassName, and the admission controller copies the integer into spec.priority.

Pods with no class and no global default get priority 0.

Every cluster ships with two classes reserved for system components:

Do not give these to application Pods. They exist so that DNS and networking win every contest for resources, because nothing else works without them.

Priority has two effects. In the scheduling queue, Pods with higher priority are tried first. And when no node has room, a high-priority Pod may preempt lower-priority Pods.

How Preemption Works

When a Pod fails to schedule, the scheduler asks: "If I removed some lower-priority Pods from one node, would this Pod fit there?" If the answer is yes for at least one node, it:

1. Picks the node where the fewest and lowest-priority victims are needed.
2. Sets the pending Pod's status.nominatedNodeName to that node.
3. Evicts the victims gracefully, honouring their terminationGracePeriodSeconds.
4. Schedules the pending Pod once the room is free.

A few rules matter in practice:

1. Only lower-priority Pods can be victims. Equal priority is never preempted.
2. The scheduler tries to respect PodDisruptionBudgets of victims, but it may violate them if there is no other option. Preemption is not a "voluntary disruption" in the strict sense.
3. Preemption does not help when the problem is not resources. A Pod that is Pending because of a node selector or a taint will see "Preemption is not helpful for scheduling" in its events, as you saw in the previous articles.
4. The victims' controllers (Deployments, ReplicaSets) recreate them, and the new Pods go back into the queue with their original, lower priority.

preemptionPolicy: Never

Some workloads should go to the front of the queue but must never kill anyone, for example a data-science job that is important but not urgent. A class with preemptionPolicy: Never gives the Pod its high priority in the queue, so it is placed before lower-priority Pods when room appears, but the scheduler will not evict anything for it.

Voluntary and Involuntary Disruptions

Kubernetes separates two kinds of disruption:

A PodDisruptionBudget can only protect against voluntary disruptions, and only those that go through the Eviction API. It cannot stop hardware from failing.

PodDisruptionBudget

A PDB selects Pods with a label selector and states how many must stay available, or how many may be unavailable, during voluntary disruptions:

Set only one of minAvailable and maxUnavailable. "Healthy" means the Pod's Ready condition is true, so a good readiness probe is a prerequisite for a meaningful budget.

The controller keeps four numbers in the PDB's status: expectedPods, currentHealthy, desiredHealthy and disruptionsAllowed. The last one is what the Eviction API checks.

Eviction Versus Deletion

This is the most misunderstood part of PDBs.

An eviction that would push the application below its budget is rejected with HTTP 429 Too Many Requests. The caller is expected to retry later. A deletion is never rejected by a PDB. So a PDB protects you from well-behaved automation, not from someone running kubectl delete pod by hand. Deployment rollouts use maxUnavailable/maxSurge on the Deployment for their own safety; they do not consult the PDB.

unhealthyPodEvictionPolicy

Imagine an API whose Pods are all running but none is Ready, perhaps because a dependency is down. A budget of minAvailable: 2 now has currentHealthy: 0. With the default policy, IfHealthyBudget, a not-Ready Pod can only be evicted if the application is currently meeting its budget. It is not, so nothing can be evicted, and kubectl drain hangs on that node indefinitely, even though evicting broken Pods would hurt no one.

Setting unhealthyPodEvictionPolicy: AlwaysAllow tells the Eviction API that Pods which are not Ready may always be evicted. Healthy Pods are still protected by the budget. For most stateless APIs this is the better setting; the Kubernetes documentation recommends it unless your application needs the stricter behaviour, for example a database that must not lose a not-yet-ready replica that is still catching up.

Hands-On: Kubernetes Commands

List PriorityClasses

Shows value, global default, and preemption policy.

Show the priority each Pod received

spec.priority is the integer the admission controller filled in.

Find preemption events

The scheduler emits a Preempted event on every victim.

List PodDisruptionBudgets with allowed disruptions

ALLOWED DISRUPTIONS is the number that matters during maintenance.

Read the full status of a budget

Includes currentHealthy, desiredHealthy and the DisruptionAllowed condition with a reason.

Drain only your own Pods from a node

--pod-selector limits eviction to matching Pods; the node is still cordoned. Always uncordon afterwards.

Call the Eviction API directly

Useful to see the raw 429 response a PDB produces.

Step-by-Step Example

The Scenario

Your shop has a few nodes with expensive accelerators used both by a nightly report generator and by a real-time fraud-scoring service. When accelerators are scarce, fraud scoring must win. Separately, the platform team needs to drain worker nodes for upgrades without ever taking the cart API below two healthy replicas.

Filling real CPU or memory to force preemption would disturb everything else on a shared cluster, so the example uses a fake extended resource instead. You will tell Kubernetes that desktop-worker3 has two units of dotnetacademy.dev/gpu. Only Pods that request this resource compete for it, so nothing else on the cluster is affected.

Step 1 — Look at the Built-in Classes

There is no global default, so every Pod in the cluster that does not ask for a class has priority 0.

Step 2 — Advertise a Fake Extended Resource

Extended resources are added to a node's status.capacity with a JSON patch on the status subresource. The / in the resource name is written as ~1 in JSON Pointer syntax.

The kubelet copies it into allocatable on its next status update, usually within a few seconds:

This is the same mechanism a real device plugin uses to advertise GPUs; only the name is invented.

Step 3 — Create the Namespace and PriorityClasses

Step 4 — Fill the Accelerators with Low-Priority Work

The report generator runs two replicas, each needing one accelerator. Extended resources must be requested as whole integers and, if a limit is given, it must equal the request.

Both replicas went to desktop-worker3, the only node with the resource, and they use both units. The accelerators are full.

Step 5 — A High-Priority Pod That May Not Preempt

Start with the polite version. It has the same high value but preemptionPolicy: Never:

It waits. The message says why in plain words: preemption was considered and refused because of the policy. Note also "3 Insufficient": the two workers without the resource count as having zero.

Step 6 — A High-Priority Pod That Preempts

Now the real fraud-scoring Pod, identical except for its name and the class lesson-online-high:

One report Pod is gone and its replacement, fbn8n, is Pending. The events tell the story:

The sequence is exactly the algorithm from Core Concepts: the fraud Pod failed to schedule, the scheduler picked a victim with lower priority, evicted it gracefully, and placed the fraud Pod in the freed slot. The recreated report Pod cannot preempt anything, because everything left on the accelerators has equal or higher priority.

Step 7 — Priority Also Orders the Queue

Two Pods are now waiting for an accelerator: polite-scoring (100000) and the recreated report Pod (1000). Free one slot:

polite-scoring got the slot even though the report Pod had been waiting just as long. It never evicted anyone, yet its priority still put it first in line. That is precisely what preemptionPolicy: Never is for.

Clean up this part:

Step 8 — A PodDisruptionBudget for the Cart API

To make the drain behaviour easy to see, pin all three cart API replicas to desktop-worker with a node selector. In production you would spread them, but a single node makes every eviction decision visible.

Three healthy Pods, one may be disrupted.

Step 9 — Drain the Node

Drain desktop-worker, limited to the cart API Pods so nothing else on the node is touched, with a timeout so the command cannot hang forever:

Drain asked to evict all three at once. The Eviction API allowed exactly one, 629lh, and rejected the others. Drain retries every five seconds. In a real cluster the evicted Pod's replacement would start on another node, become Ready, disruptionsAllowed would go back to 1, and the next eviction would succeed. Here the replacement can only run on desktop-worker because of the node selector, and that node is cordoned, so the budget never recovers and drain gives up at the timeout.

After uncordon, the replacement Pod starts and the budget will return to one allowed disruption once it is Ready. This is the everyday lesson of PDBs: a drain that "hangs" is usually a budget doing its job, and the fix is to make sure replacements can start somewhere else, not to delete the budget.

Never forget the uncordon. A cordoned node silently stops accepting new Pods.

Step 10 — When Unhealthy Pods Block Maintenance

Now an inventory API whose readiness probe points at an endpoint that does not exist, so its Pods run but never become Ready, protected by minAvailable: 2:

Try to drain:

Nothing could be evicted. The application is already broken, yet the budget protects the broken Pods and blocks the maintenance. The status explains it:

Step 11 — Deletion Ignores the Budget, AlwaysAllow Fixes Eviction

First, prove that a plain delete is not subject to the budget at all:

It succeeds immediately, while disruptionsAllowed is 0. That is why PDBs protect against automation and careful operators, not against a careless kubectl delete.

Now the real fix: let not-Ready Pods be evicted.

The drain completes. Healthy Pods would still have been protected; only the not-Ready ones were let go. For stateless .NET APIs, put unhealthyPodEvictionPolicy: AlwaysAllow in your PDB template from day one.

Step 12 — Seeing the 429 Directly

kubectl drain hides the HTTP details. Switch the policy back and call the Eviction API yourself:

Create eviction.json with the name of one of the inventory Pods:

TooManyRequests is HTTP 429. Any tool that evicts Pods, including your own operators written with the Kubernetes .NET client, should treat 429 as "try again later", which is exactly what drain does.

Step 13 — Clean Up

Remove the namespace, the cluster-scoped PriorityClasses, and the fake resource. Removing a capacity entry uses the same JSON Pointer path with remove.

The last command should show <none> for every node, confirming nothing was left cordoned.

Summary

1. A PriorityClass maps a name to an integer. Higher-priority Pods are scheduled first and may preempt lower-priority Pods when there is no room. Leave system-cluster-critical and system-node-critical to cluster components.
2. Preemption evicts only lower-priority Pods, picks the node needing the fewest victims, and tries but does not guarantee to respect PDBs. It never helps with selector or taint problems.
3. preemptionPolicy: Never keeps the queue advantage without evicting anyone.
4. A fake extended resource on a node status is a safe way to create scarcity for experiments.
5. A PodDisruptionBudget limits voluntary disruptions through the Eviction API. Prefer maxUnavailable, and make sure replacements can start elsewhere, or drains will stall.
6. Deletion bypasses PDBs; only eviction respects them. A rejected eviction returns HTTP 429.
7. kubectl drain --pod-selector limits a drain to specific Pods; always uncordon afterwards.
8. unhealthyPodEvictionPolicy: AlwaysAllow lets broken, not-Ready Pods be evicted so they cannot block node maintenance.

That completes the section on advanced scheduling. You can now attract Pods to nodes, keep them away, spread them precisely, rank them when resources run out, and protect them while nodes come and go.