Taints, Tolerations and Topology Spread Constraints

In the previous article you attracted Pods to nodes with selectors and affinity, and you pushed replicas apart with pod anti-affinity. All of those rules live on the Pod. The node had no say.

This article covers two tools that complete the picture. Taints let a node push Pods away unless they explicitly tolerate it, which is how you reserve nodes for special workloads. Topology spread constraints let you say exactly how uneven a distribution of replicas is allowed to be, which is what pod anti-affinity was trying to do with a much blunter instrument. Along the way you will hit a real gotcha that makes a perfectly reasonable spread rule leave Pods Pending on almost every kind cluster.

We will cover the following topics:

1. What a taint and a toleration are, and why a toleration is not an invitation
2. The three taint effects: NoSchedule, PreferNoSchedule and NoExecute, including tolerationSeconds
3. The taint every control-plane node carries, and the tolerations Kubernetes adds to every Pod
4. The dedicated nodes pattern: taint, label, toleration and affinity together
5. Topology spread constraints: maxSkew, topologyKey, whenUnsatisfiable, minDomains, nodeTaintsPolicy and matchLabelKeys
6. When to choose spread constraints over pod anti-affinity

Technical Requirements

1. A multi-node cluster. The examples use Docker Desktop's Kubernetes with the kind provisioner: desktop-control-plane plus the workers desktop-worker, desktop-worker2 and desktop-worker3, Kubernetes v1.36. A plain kind cluster with one control-plane and three workers behaves the same with kind-* node names.
2. kubectl configured for that cluster.
3. The node-info-api:1.0 image from the previous article. Any small image that keeps running works too.
4. Permission to add labels and taints to nodes. Every change in this article is removed again in the last step.

Core Concepts

Taints Repel, Tolerations Permit

A taint is a property of a node with three parts: key=value:effect. It says: "Pods may not come here unless they tolerate this."

A toleration is a property of a Pod that matches a taint. It says: "I accept that taint."

The important word is accept. A toleration does not attract a Pod to the tainted node. It only removes one obstacle. A Pod that tolerates a taint can still be scheduled on any untainted node, and usually will be. To send Pods to a node you still need a selector or node affinity from the previous article. You will see this clearly in the step-by-step example.

The Three Effects

NoSchedule is the hard, everyday choice. PreferNoSchedule is a soft version, useful for "please move off this node soon" during maintenance planning.

NoExecute is different from the other two: it acts on Pods that are already running. When you add a NoExecute taint, every Pod on that node without a matching toleration is evicted immediately. That makes it a powerful but dangerous tool on a shared cluster, which is why this article explains it but does not apply it. A Pod can soften NoExecute with tolerationSeconds:

This means: "if the node gets this taint, let me stay for 300 more seconds, then evict me". Without tolerationSeconds a matching toleration lets the Pod stay forever.

Taints Kubernetes Adds for You

Two kinds of taints exist on most clusters before you add any.

The control-plane taint, node-role.kubernetes.io/control-plane:NoSchedule, keeps application Pods off the nodes that run the API server, etcd and the scheduler. That is why the scheduler messages in the previous article always said "1 node(s) had untolerated taint(s)".

The node lifecycle taints are added automatically by the node controller when something goes wrong with a node. The most common ones are:

The first two use NoExecute. To avoid evicting everything the moment a node blips, Kubernetes adds a toleration for both, with tolerationSeconds: 300, to every Pod that does not declare its own. That is where the familiar "Pods take about five minutes to move off a dead node" behaviour comes from. You can shorten it per workload by declaring those tolerations yourself with a smaller value.

Matching Rules for Tolerations

DaemonSets that must run on every node, such as log collectors and CNI agents, typically carry a toleration with operator: Exists and no key, which tolerates everything.

The Dedicated Nodes Pattern

To reserve a node for one kind of workload, you need both directions:

1. A taint on the node keeps everyone else off.
2. A label on the node identifies it.
3. A toleration on the special workload lets it in.
4. Node affinity (or a nodeSelector) on the special workload makes it go there and nowhere else.

Skip step 4 and your special workload spreads across the whole cluster. Skip step 1 and ordinary workloads fill up your special node. You will try both halves below.

Topology Spread Constraints

Pod anti-affinity can only say "at most one per domain" (required) or "please avoid" (preferred). Topology spread constraints let you specify how much imbalance you allow.

The skew calculation is simple. Count matching Pods in each eligible domain. Find the smallest count, the global minimum. A domain may receive a new Pod only if, after adding it, its count minus the global minimum is not more than maxSkew.

Nodes that do not have the topologyKey label at all are ignored. A Pod can have several constraints; all DoNotSchedule ones must hold at the same time.

Spread Constraints vs Pod Anti-Affinity

For "spread my replicas evenly", topology spread constraints are the modern answer. Keep required pod anti-affinity for the narrow case where two copies on one node are genuinely forbidden.

Hands-On: Kubernetes Commands

Show the taints of every node

custom-columns turns the taints array into readable columns.

Add a taint

The format is key=value:effect.

Remove a taint

Same text with a trailing -.

See the tolerations Kubernetes added to a Pod

Run this on any Pod you did not give tolerations to.

Count replicas per node

The quickest way to check a spread.

Read the documentation for a spread field

Shows the default and whether the field is beta in your cluster version.

Step-by-Step Example

The Scenario

Your shop runs nightly invoice generation, a CPU-heavy batch job that should have a worker to itself so it never slows down customer-facing APIs. You will reserve desktop-worker3 for it. Then you will make the checkout and payments APIs spread evenly across your three workers, using the same pretend zones as in the previous article: desktop-worker is zone-a, desktop-worker2 is zone-b, desktop-worker3 is zone-c.

To keep the manifests short, every Deployment below uses the same container block; only the spec.template.spec scheduling fields change:

Step 1 — Look at the Existing Taints and Default Tolerations

Run a throwaway Pod without any tolerations and look at what it got anyway:

Those two lines are the five-minute grace period described earlier, added by an admission plugin.

Step 2 — Taint and Label the Batch Node

Using the same key for the taint and the label is a common convention. It makes the pairing obvious to anyone reading the node.

Step 3 — Three Workloads, Three Behaviours

First an ordinary API with no tolerations, web-api, six replicas:

Second, report-worker, which tolerates the taint but has no affinity:

Third, invoice-batch, with the toleration and required node affinity to the label:

Apply all three and count where the replicas went:

This one output summarises the whole pattern:

1. web-api has no toleration, so the taint keeps it off desktop-worker3.
2. report-worker tolerates the taint, so desktop-worker3 is allowed, but nothing pulls it there. It spreads over all three workers. A toleration alone does not make a dedicated node.
3. invoice-batch tolerates the taint and requires the label, so all four replicas land on desktop-worker3 and nowhere else.

Step 4 — PreferNoSchedule Is a Nudge

Pretend desktop-worker2 is scheduled for maintenance tomorrow. You want new Pods to avoid it, but not at any cost.

Deploy a fresh six-replica API, storefront-api, identical to web-api apart from its name:

desktop-worker3 is still hard-tainted for batch, and desktop-worker2 is softly tainted, so everything went to desktop-worker. If desktop-worker had been full, the scheduler would have used desktop-worker2 rather than leave Pods Pending.

Remove the maintenance taint and the batch reservation:

Step 5 — What NoExecute Would Do

This step is reading only. On a shared cluster, a NoExecute taint evicts every Pod on the node that lacks a matching toleration, including other people's workloads, so do not run it on a cluster you share.

Every Pod on desktop-worker2 without a toleration for that taint would be evicted at once. A Pod that should survive briefly, for example to finish in-flight requests, declares:

It would stay for 60 seconds and then be evicted. For planned maintenance, kubectl drain is usually the better tool because it respects PodDisruptionBudgets, which the next article covers. NoExecute taints do not.

Step 6 — Spread Across Zones with maxSkew

Label the workers with the pretend zones:

Now the checkout API, seven replicas, with a hard spread rule:

Seven replicas in three zones: 3/2/2, skew exactly 1. The control-plane node has no zone label, so it is not a domain at all. Pod anti-affinity could not have produced this: required anti-affinity stops at three, and preferred anti-affinity gives no guarantee.

Step 7 — The Control-Plane Gotcha

Zone labels are not always available, so it is tempting to spread across nodes with kubernetes.io/hostname instead. Try it with the payments API, six replicas:

Three Pods run, one per worker, and three are stuck, although each worker has plenty of room.

Here is why. Every node has a kubernetes.io/hostname label, including the control-plane node. The default nodeTaintsPolicy is Ignore, so the tainted control-plane node still counts as a domain. It has zero payments Pods, which makes the global minimum 0. With maxSkew: 1, no worker may hold more than one replica. The scheduler cannot put a Pod on the control-plane node because of its taint, so the minimum never rises and the remaining replicas wait forever.

The fix is one line: tell the constraint to honour taints, so only nodes the Pod could actually use count as domains.

The same trap appears on any cluster with tainted nodes that carry the topology label: GPU pools, dedicated batch nodes like the one from Step 2, or nodes that are cordoned. If a hostname-based spread leaves Pods Pending while nodes look empty, check nodeTaintsPolicy first.

Step 8 — minDomains: Refuse to Pile Up When Domains Are Missing

Suppose your production cluster normally has four zones and you never want more than one ledger replica per zone above the others. If a zone disappears, you would rather have Pods wait than squeeze extra replicas into the surviving zones. minDomains expresses that.

Only three zones exist, fewer than minDomains: 4, so the scheduler treats the global minimum as 0. With maxSkew: 1 each zone may hold one replica, and the other two wait until a fourth zone appears. Without minDomains you would get 2/2/1 instead. Which behaviour is right depends on whether you prefer availability now or a guaranteed spread later. minDomains only applies with DoNotSchedule.

Step 9 — Zones and Nodes Together, Safe for Rolling Updates

In real clusters you usually want two constraints: a hard spread across zones and a soft spread across nodes inside each zone. You also want each rollout to spread its own Pods evenly, which is what matchLabelKeys does.

pod-template-hash is the label a Deployment puts on Pods of each ReplicaSet. Without matchLabelKeys, during a rolling update the scheduler counts old and new Pods together, so the new ReplicaSet can end up lopsided once the old Pods are gone. With it, only Pods from the same revision are counted.

After the rollout the new revision is balanced 2/2/2. You can see how the API server applied matchLabelKeys by reading the stored constraint on a Pod:

The hash value was merged into the selector when the Pod was created. On Kubernetes v1.36, kubectl explain still describes matchLabelKeys as a beta field behind the MatchLabelKeysInPodTopologySpread feature gate, which is enabled by default.

Step 10 — Clean Up

The last command should show only the control-plane taint.

Summary

1. A taint on a node repels Pods; a toleration on a Pod permits it onto that node but never attracts it there.
2. NoSchedule blocks new Pods, PreferNoSchedule only discourages them, and NoExecute also evicts running Pods, optionally after tolerationSeconds.
3. Every Pod gets default NoExecute tolerations for not-ready and unreachable with 300 seconds, which is why Pods take five minutes to leave a failed node.
4. A dedicated node needs four pieces: taint and label on the node, toleration and node affinity on the workload.
5. Topology spread constraints limit imbalance with maxSkew over domains defined by topologyKey. DoNotSchedule is a filter, ScheduleAnyway is a score.
6. With the default nodeTaintsPolicy: Ignore, tainted nodes such as the control plane still count as domains and can leave Pods Pending. Set nodeTaintsPolicy: Honor for hostname spreads.
7. minDomains keeps replicas waiting rather than piling up when fewer domains exist than expected.
8. matchLabelKeys with pod-template-hash keeps each rollout balanced on its own.
9. Prefer spread constraints over pod anti-affinity for "spread evenly"; keep required anti-affinity for "never together".

In the next article we look at what happens when the cluster is full: PriorityClasses decide which Pods win, preemption decides who has to leave, and PodDisruptionBudgets protect your API during node maintenance.