Node Selectors, Node Affinity and Pod Affinity/Anti-Affinity

Up to this point in the Kubernetes series, you have let the scheduler decide where every Pod runs. You wrote a Deployment, applied it, and the Pods appeared on whichever worker node had room. For most stateless .NET APIs that is exactly what you want. The scheduler is good at its job, and the less you tell it, the more freedom it has to keep the cluster balanced.

Real clusters are rarely that uniform, though. Some nodes have fast SSDs and some do not. Some sit in one availability zone and some in another. Some workloads must never share a node with a second copy of themselves, and some work best when they sit right next to a specific neighbour, such as an API and its local cache. This article shows how to express those wishes to the scheduler, from the bluntest tool to the most precise.

We will cover the following topics:

1. How the scheduler picks a node: filtering and scoring
2. nodeSelector, the simplest way to target labelled nodes
3. nodeName, and why you should almost never use it
4. Node affinity, with required and preferred rules, operators and weights
5. What IgnoredDuringExecution really means
6. Pod affinity and pod anti-affinity, and the role of topologyKey
7. Spreading replicas of a .NET API across workers, and co-locating a Redis cache with it

Technical Requirements

1. A multi-node cluster. The examples use Docker Desktop's built-in Kubernetes with the kind provisioner: one control-plane node (desktop-control-plane) and three workers (desktop-worker, desktop-worker2, desktop-worker3), Kubernetes v1.36. A plain kind create cluster with a config file that declares one control-plane and three workers behaves the same; only the node names change to kind-control-plane, kind-worker, and so on.
2. kubectl configured for that cluster.
3. Docker and the .NET 10 SDK to build the small sample API.
4. A single-node cluster such as minikube can run the manifests, but you will not see any of the placement effects this article is about.

Core Concepts

How the Scheduler Chooses a Node

When a new Pod has no node yet, the scheduler runs it through two phases.

Every feature in this article plugs into one of those two phases. A hard rule such as nodeSelector or a required affinity term is a filter: a node that does not match is simply not a candidate. A soft rule such as a preferred affinity term only changes the score: a node that does not match can still win if nothing better is available.

Keep that distinction in mind. It explains almost every surprise you will meet, including Pods that stay Pending forever and Pods that land somewhere you "told" them not to go.

Labels Are the Raw Material

All of these features work by matching labels on nodes or labels on Pods. Kubernetes adds some node labels for you:

On a managed cloud cluster the zone and instance-type labels are already there. On a local kind cluster they are not, so in the hands-on part we will add our own labels under a prefix we own, lesson.dotnetacademy.dev/. Using your own prefix for custom labels is good practice in real clusters too: it makes it obvious which labels your team manages, and it never collides with labels a cloud provider or controller might add later.

nodeSelector

nodeSelector is a map of labels that a node must all have. It is the oldest and simplest placement tool.

It is a pure filter. If no node carries every listed label, the Pod stays Pending. It cannot express "ssd or nvme", "anything except hdd", or "prefer ssd if possible". For those you need node affinity.

nodeName — Bypassing the Scheduler

Setting spec.nodeName does not ask the scheduler anything. It skips the scheduler entirely and hands the Pod straight to the kubelet on that node.

That sounds convenient, but it has sharp edges:

1. Taints are not checked. In the hands-on section you will see a Pod land on the control-plane node, which every normal workload is kept off.
2. Resources are only checked by the kubelet at admission time, not by the scheduler's view of the cluster. If the node is full, the Pod fails with an error such as OutOfcpu instead of waiting politely.
3. If the node does not exist, the Pod sits there unassigned. Nothing will move it elsewhere.
4. A Deployment with a hard-coded node name defeats the point of a Deployment. If that node dies, the replacement Pods are pinned to the dead node.

nodeName exists for tools that implement their own scheduling, and it is handy for a one-off debugging Pod. For application workloads, use a selector or affinity instead.

Node Affinity

Node affinity is nodeSelector with a richer language. It lives under spec.affinity.nodeAffinity and comes in two flavours with deliberately long names:

A required rule is a list of nodeSelectorTerms. The terms are ORed: the node needs to match at least one of them. Inside a term, the matchExpressions are ANDed: the node must satisfy all of them.

Each expression has a key, an operator and, for most operators, values:

NotIn and DoesNotExist give you node anti-affinity: "keep me off nodes like these".

A preferred rule is a list of {weight, preference} pairs, where weight is 1–100. For every node that passed filtering, the scheduler adds up the weights of the preferences that node matches. That sum becomes one of several scoring inputs; it is not the only factor, so a preference is a strong hint rather than a promise.

What IgnoredDuringExecution Means

Both names end in IgnoredDuringExecution. This tells you that affinity is evaluated only when the Pod is scheduled. If you later change a node's labels so that a running Pod no longer matches, Kubernetes does nothing to that Pod. It keeps running where it is. Only Pods scheduled from that moment on see the new labels.

This is intentional. Evicting running Pods every time someone edits a label would be dangerous. If you want running Pods to move, you replace them, for example with kubectl rollout restart.

Pod Affinity and Pod Anti-Affinity

Node affinity looks at node labels. Pod affinity and anti-affinity look at the labels of other Pods already running, and they need one more ingredient: a topologyKey.

The topology key is the name of a node label. Nodes that share the same value for that label form one topology domain. The rule then reads like this:

1. Pod affinity: schedule me into a domain that already runs a Pod matching this labelSelector.
2. Pod anti-affinity: schedule me into a domain that does not run a Pod matching this labelSelector.

Both come in the same required and preferred flavours as node affinity. Pod anti-affinity with kubernetes.io/hostname is the classic way to make sure two replicas of the same API never share a node, so that losing one node never takes out both.

Inter-Pod rules are more expensive for the scheduler than node rules, because it has to look at the Pods on every candidate node. On clusters with many thousands of nodes the Kubernetes documentation recommends using them with care. For the cluster sizes most teams run, they are fine.

Choosing the Right Tool

Hands-On: Kubernetes Commands

Show nodes with chosen labels as columns

-L adds one column per label key, which is the fastest way to see your topology.

Show every label on a node

Useful when you are looking for the exact key a cloud provider uses.

Add and remove a node label

A trailing - removes the label. --overwrite is required to change an existing value.

See which node each Pod landed on

-o wide adds the NODE column.

Count Pods per node for one app

Handy when you have many replicas and only care about the distribution.

Find out why a Pod is Pending

The last event from default-scheduler tells you which filter rejected each node.

Read the affinity field documentation from the cluster

kubectl explain always matches your cluster version.

Step-by-Step Example

The Scenario

You run several .NET 10 APIs for an online shop on a cluster with three workers. To make the examples realistic, you will pretend the workers are spread across three availability zones and have different disks:

Then you will place a catalog API only on SSD nodes, keep a reports API inside two zones, nudge a search API towards SSDs, spread a basket API one replica per node, and pin a Redis cache next to each basket API replica.

Step 1 — Build a Small API That Reports Its Node

Every Pod in this article runs the same tiny API. It returns its own Pod name and the node it runs on, read from the Downward API.

Replace Program.cs:

Add a Dockerfile:

Build it:

Docker Desktop's kind cluster can use images from your local Docker image store directly, so imagePullPolicy: IfNotPresent is enough. With a plain kind cluster, copy the image into the nodes first with kind load docker-image node-info-api:1.0.

Step 2 — Namespace and Node Labels

The control-plane node has no zone or disk label. That matters later: any rule that requires these labels automatically excludes it.

Step 3 — nodeSelector: SSD Nodes Only

The catalog API reads a large product index from local disk, so it should only run on SSD nodes.

All four replicas are on the two SSD nodes. desktop-worker was filtered out before scoring even started.

Step 4 — What Happens When Nothing Matches

Ask for a disk type that no node has:

Read the message as a tally of the filter phase. The control-plane node was rejected by its taint (the subject of the next article) and the three workers were rejected by the selector. The Pod will stay Pending until a node with that label appears. Note that nodeSelector failures are reported as "node affinity/selector": internally the scheduler treats them as the same filter.

Step 5 — nodeName Skips Every Check

Now pin a Pod to the control-plane node by name:

Two things stand out. The Pod is running on the control-plane node even though that node's taint keeps normal Pods away. And there is no Scheduled event from default-scheduler at all: the first event comes from the kubelet. The scheduler never saw this Pod. That is exactly why nodeName does not belong in application manifests.

Clean up before moving on:

Step 6 — Required Node Affinity: Two Zones, Any Disk

The reports API must stay inside zone-a and zone-b for data-residency reasons, and it must only run on nodes that declare a disk type at all.

The two expressions are in the same term, so a node must satisfy both. nodeSelector cannot express the In list.

Nothing on desktop-worker3, which is in zone-c.

Step 7 — Proving IgnoredDuringExecution

Remove the zone label from desktop-worker, which is running two reports Pods:

The two Pods on desktop-worker keep running. Kubernetes does not re-check affinity for running Pods. Now scale up:

The two new Pods both went to desktop-worker2, the only node that still matches. The old Pods on desktop-worker are untouched. In a real cluster this is how a label change "drains" a node slowly: new Pods avoid it while old ones finish their life naturally.

Put the label back:

Step 8 — Preferred Node Affinity with Weights

The search API can run anywhere in our three zones, but it is noticeably faster on SSD and slightly better in zone-a, where most users are.

The required part keeps it on labelled workers. The preferred part gives desktop-worker2 and desktop-worker3 80 points each for SSD, and desktop-worker 20 points for zone-a.

80 beats 20, so all six went to the SSD nodes. If both SSD nodes were full, the scheduler would still place replicas on desktop-worker instead of leaving them Pending. That is the difference between required and preferred.

Clean up:

Step 9 — Required Pod Anti-Affinity: One Replica per Node

The basket API holds in-memory session state for a few seconds during checkout. Losing two replicas at once is painful, so no two replicas may share a node.

The rule says: do not schedule me on a node (kubernetes.io/hostname) that already runs a Pod labelled app: basket-api. The Pod's own label matches its own selector, which is what makes the replicas repel each other.

One per worker. Now ask for a fourth:

This is the price of a required rule: replicas can never exceed the number of domains. With three workers you get at most three replicas. During a rolling update this can also block progress, because the new Pod needs a free node before an old one goes away. Use required anti-affinity when "two on one node" is genuinely unacceptable, and size the Deployment strategy accordingly (for example maxSurge: 0, maxUnavailable: 1).

Scale back to three:

Step 10 — Pod Affinity: A Cache Next to Every API Replica

Each basket API replica talks to a Redis instance for short-lived data. Network hops between nodes add latency, so you want one Redis Pod on every node that runs a basket API, and never two Redis Pods on the same node.

Two rules work together. Pod affinity attracts each cache Pod to a node that runs app: basket-api. Pod anti-affinity against its own label keeps two caches apart.

Sorted by node, the pairs are obvious: every worker runs exactly one API and one cache. If you scaled the cache to four, the fourth would be Pending for the same reason as the fourth basket API in the previous step.

Remember that the affinity is one-directional. The cache follows the API; the API knows nothing about the cache. If the API Pods move, existing cache Pods stay where they are (IgnoredDuringExecution again), so in production you would usually restart both together or run them as two containers of one Pod when they are truly inseparable.

Step 11 — Preferred Anti-Affinity Is Only a Hint

Replace the basket API's hard rule with a soft one and ask for five replicas, more than there are nodes:

Note the extra podAffinityTerm wrapper that preferred terms need.

All five run, spread 1-2-2. Nothing is Pending, which is the whole point of the preferred form.

Be aware that a preference is one scoring input among several. While testing this article on a busy cluster, with Pods from a previous run still terminating, one attempt put all five replicas on a single node. The rule was obeyed in the sense that it was considered; it simply lost to other scores. If an even spread really matters, use a topology spread constraint, covered in the next article, which gives you a hard, measurable limit on imbalance.

Step 12 — Clean Up

Delete the namespace and remove every label you added. The trailing - removes a label.

Summary

1. The scheduler filters nodes with hard rules, then scores the survivors with soft rules. Required rules filter; preferred rules score.
2. nodeSelector is a simple AND of node labels. If nothing matches, the Pod stays Pending.
3. nodeName bypasses the scheduler, ignores taints, and pins Pods to a single machine. Keep it out of application manifests.
4. Node affinity adds In, NotIn, Exists, DoesNotExist, Gt and Lt. Terms are ORed, expressions inside a term are ANDed, and preferred terms carry weights from 1 to 100.
5. IgnoredDuringExecution means label changes never move running Pods. Only newly scheduled Pods see them.
6. Pod affinity and anti-affinity match labels of other Pods inside a topology domain chosen by topologyKey. kubernetes.io/hostname means "per node"; a zone label means "per zone".
7. Required anti-affinity caps replicas at the number of domains. Preferred anti-affinity never blocks scheduling but does not guarantee an even spread.
8. Use your own label prefix for custom node labels, and on cloud clusters prefer the well-known topology.kubernetes.io/zone label.

In the next article we turn the problem around with taints and tolerations, which let a node keep Pods away, and with topology spread constraints, which give you precise control over how evenly replicas are distributed.