Kubernetes Architecture — Control Plane, Worker Nodes and the Life of a Pod

You can use Kubernetes for a long time by treating it as a black box: you hand it YAML, Pods appear. That works until something goes wrong. A Pod stays Pending, a node goes NotReady, a Service returns nothing, and suddenly you need to know which part of the cluster is responsible.

This article opens the box. We look at every component of a cluster, what it owns, and how they cooperate. Then we follow one kubectl apply of a Deployment through the whole system on a real four-node cluster, using the cluster's own events, its etcd keys and its container runtime as evidence.

We will cover the following topics:

1. The control plane and the worker nodes, and why they are split
2. The API server, etcd, the scheduler and the controller manager
3. The kubelet, the container runtime, kube-proxy and the CNI plugin
4. The declarative model and reconciliation loops
5. CoreDNS and add-ons
6. The life of a Pod, traced step by step on a live cluster

Technical Requirements

1. A multi-node cluster. The examples use Docker Desktop's Kubernetes with the kind provisioner: one control-plane node and three workers, Kubernetes v1.36. The next article shows how to create exactly this cluster. A plain kind cluster with the same shape works the same way.
2. kubectl configured for the cluster. The context in the examples is docker-desktop.
3. Docker, because the kind nodes are containers and we will open a shell inside them.

Core Concepts

Two Halves: Control Plane and Worker Nodes

A Kubernetes cluster is a set of machines split into two roles.

The control plane makes decisions. It stores the desired state of the cluster, decides where Pods should run, and notices when reality drifts away from what you asked for.

The worker nodes do the work. They run your containers, report their health, and wire up networking so that Pods can reach each other.

In managed services such as AKS, EKS and GKE you never see the control plane machines. The provider runs them. On a local kind cluster you can see everything, which makes it the best place to learn.

Here is the cluster used in this article:

Four nodes, one of them with the control-plane role. Every node, including the control plane, runs containerd as its container runtime.

kube-apiserver: The Front Door

The API server is the only component that talks to etcd, and the only one every other component talks to. kubectl, the scheduler, the controllers, the kubelets: all of them are clients of the API server.

It does three jobs on every request:

1. Authentication: who are you? (client certificates, tokens, OIDC)
2. Authorization: are you allowed to do this? (RBAC)
3. Admission: should this object be changed or rejected before it is stored? (mutating and validating admission, including policies)

Only after these three steps does the object land in etcd. The API server is stateless, so production clusters run several copies behind a load balancer.

The other job of the API server is just as important: watches. Components do not poll. They open a long-lived watch and the API server streams every change to the objects they care about. This is what makes the whole system react within milliseconds.

etcd: The Single Source of Truth

etcd is a consistent, distributed key-value store. Every Kubernetes object you create is stored there as a key under /registry. If etcd is lost and you have no backup, the cluster's state is gone, even if every container is still running.

Two things to remember about etcd:

1. Only the API server reads and writes it. Nothing else touches it directly.
2. It uses the Raft consensus algorithm, so production clusters run an odd number of members (3 or 5) to survive failures.

kube-scheduler: Choosing a Node

The scheduler watches for Pods that have no node assigned. For each one it does two things:

1. Filtering: drop nodes that cannot run the Pod. Not enough CPU or memory, a taint the Pod does not tolerate, a node selector that does not match.
2. Scoring: rank the remaining nodes. Spread replicas, prefer nodes that already have the image, balance resource usage.

The winner is written into the Pod's spec.nodeName through the API server. That is all the scheduler does. It never starts a container itself.

kube-controller-manager: The Reconciliation Engine

The controller manager is one binary that runs dozens of controllers: the Deployment controller, the ReplicaSet controller, the Node controller, the Job controller, the EndpointSlice controller and many more.

Every controller follows the same loop:

1. Watch the objects it owns.
2. Compare the desired state (the spec) with the actual state (the status and the related objects).
3. Act to close the gap.
4. Repeat, forever.

This is the declarative model. You never tell Kubernetes "start three Pods". You tell it "there should be three Pods", and a controller keeps making that true. If a Pod dies, the gap reappears and the controller closes it again.

cloud-controller-manager

The cloud-controller-manager contains the controllers that need to talk to a cloud provider: creating load balancers for type: LoadBalancer Services, attaching routes, labelling nodes with their zone. On AKS, EKS or GKE it is always there. On bare metal or kind there is no cloud, so either nothing fills this role, or a small replacement does. Docker Desktop's kind mode ships a component called desktop-cloud-provider-kind, which is why LoadBalancer Services get an external address there.

kubelet: The Node Agent

The kubelet runs on every node as a normal system service, not as a Pod. It watches the API server for Pods assigned to its node, and then:

1. asks the container runtime to pull images and start containers,
2. mounts volumes and injects Secrets and ConfigMaps,
3. runs liveness, readiness and startup probes,
4. reports Pod and node status back to the API server.

The kubelet also runs static Pods: Pods defined by files in a directory on the node, usually /etc/kubernetes/manifests. This is how the control plane components themselves run on a kubeadm or kind cluster.

The Container Runtime and CRI

The kubelet does not run containers itself. It talks to a container runtime through the Container Runtime Interface (CRI), a gRPC API. containerd and CRI-O are the common runtimes. Docker Engine is no longer used directly by the kubelet; the dockershim was removed in Kubernetes 1.24.

crictl is the debugging client for the CRI. It is to the runtime what kubectl is to the API server, and it is already installed inside every kind node.

kube-proxy and the CNI Plugin

Two different components handle networking, and people often confuse them.

The CNI plugin is called by the container runtime whenever a Pod sandbox is created. Popular plugins are Calico, Cilium, Flannel and Azure CNI. kind ships its own small plugin, kindnet.

kube-proxy runs on every node and watches Services and EndpointSlices. In iptables mode it writes NAT rules so that traffic to a ClusterIP is redirected to one of the backing Pods. Some CNI plugins, such as Cilium, can replace kube-proxy entirely.

CoreDNS and Other Add-ons

CoreDNS is not part of the control plane, but almost nothing works without it. It runs as an ordinary Deployment in kube-system and answers queries such as hello-api.arch-demo.svc.cluster.local. Other common add-ons are metrics-server, a storage provisioner (kind uses local-path-provisioner) and an ingress or gateway controller.

Leader Election

The scheduler and the controller manager can run several replicas for high availability, but only one copy may act at a time, otherwise two schedulers would place the same Pod twice. They agree on a leader by competing for a Lease object through the API server:

Hands-On: Kubernetes Commands

Check the API server's health

The /readyz endpoint runs every internal readiness check of the API server.

Find the control plane endpoint

Shows the address of the API server and of the CoreDNS proxy.

List the static Pod manifests

On a kind node the control plane runs as static Pods. The node is a container, so docker exec gets you inside it.

See containers through the CRI

crictl ps shows what the container runtime is running, independently of the API server.

Check which mode kube-proxy uses

The mode is part of the kube-proxy ConfigMap.

Read events in time order

kubectl events is the newer, sorted alternative to kubectl get events.

Check that kubelet and containerd are running on a node

Both are system services on the node, not Pods.

Step-by-Step Example

The Scenario

We deploy a small .NET sample web app with three replicas and follow the request through every component: from kubectl to the API server, into etcd, through the Deployment and ReplicaSet controllers, to the scheduler, to the kubelet and containerd on each node, and finally through kube-proxy and CoreDNS when we add a Service.

Step 1 — Look at the Control Plane

The four control plane components are static Pods on the control-plane node.

Open one of them. It is an ordinary Pod definition, which the kubelet starts directly from the file without asking the API server.

Notice --kubeconfig: the scheduler is just another API client with its own credentials, and --leader-elect=true turns on the Lease mechanism described above.

Now look at the same components through the container runtime:

The output is trimmed to the important columns. Every component we discussed is here: the API server, etcd, the scheduler, the controller manager, plus kube-proxy, the kindnet CNI agent and CoreDNS.

Step 2 — Confirm the Node Agents

The kubelet and containerd are not in that list, because they are not containers. They are system services of the node:

If the kubelet on a node stops, the Node controller marks the node NotReady after a grace period of a few tens of seconds, and eventually evicts its Pods.

Step 3 — Apply a Deployment

Save this as arch-deploy.yaml:

kubectl sent two requests to the API server. Each was authenticated, authorized, passed admission, and was stored in etcd. At this moment no Pod exists yet. The API server has simply recorded your intent.

Step 4 — Follow the Events

Every component that acts writes an Event. Reading them in order shows the whole chain:

Events created in the same second are not strictly ordered, so read them by reason. The real sequence is:

Notice that no component called another directly. Each one watched the API server, saw an object it owns, and wrote a new object back. That is the entire architecture in one table.

Also notice that the scheduler spread the three replicas over three different workers and skipped the control-plane node. The control-plane node has the taint node-role.kubernetes.io/control-plane:NoSchedule, so the filtering step removed it.

Step 5 — See the Objects in etcd

Every object you just watched being created is a key in etcd. The etcd Pod contains etcdctl, so you can list the keys under /registry that mention the namespace. This is a read-only command. Never write to etcd by hand.

The Deployment, the ReplicaSet, the three Pods and even the Events are there. You also see two objects you never created: the default ServiceAccount and the kube-root-ca.crt ConfigMap. Two more controllers added them the moment the namespace appeared.

Step 6 — See the Containers on the Node

Check where the Pods landed:

Each worker has its own Pod address range: 10.244.2.x on desktop-worker, 10.244.1.x on desktop-worker2, 10.244.4.x on desktop-worker3. The kindnet CNI plugin assigned these addresses and routes them between nodes.

Now ask containerd on desktop-worker directly, bypassing the API server completely:

A Pod in containerd is a sandbox: a set of shared Linux namespaces. The application container runs inside it. This is the level the kubelet works at.

Step 7 — Watch Reconciliation Fix a Failure

Delete one Pod and list the Pods immediately:

Within four seconds a replacement Pod (nppvt) is running. You did not ask for it. The ReplicaSet controller noticed that the actual count (2) no longer matched the desired count (3) and closed the gap. The scheduler placed the new Pod, and a kubelet started it. This loop is why Kubernetes is described as self-healing.

Step 8 — Add a Service: kube-proxy and CoreDNS

Two more controllers just worked. The Service controller allocated the ClusterIP, and the EndpointSlice controller collected the addresses of the three ready Pods.

kube-proxy turned that into iptables rules on every node:

Any packet to 10.96.205.50:80 jumps to a chain that picks one of the three Pod IPs at random. The ClusterIP is not bound to any network interface. It exists only as this rule.

Finally CoreDNS gives the Service a name:

Step 9 — Clean Up

Deleting the namespace deletes everything inside it. The garbage collector, yet another controller, removes the ReplicaSet and the Pods that the Deployment owned.

Summary

1. A cluster is split into the control plane, which decides, and the worker nodes, which run containers.
2. The API server is the only component that talks to etcd. Every other component is an API client that watches for changes.
3. The scheduler only writes spec.nodeName. The kubelet on that node does the actual work, through the container runtime via the CRI.
4. Controllers in the controller manager run reconciliation loops: compare desired and actual state, act, repeat. Deleting a Pod proved it: a replacement appeared in four seconds.
5. The CNI plugin gives Pods their IPs. kube-proxy turns Service IPs into forwarding rules. CoreDNS gives Services names.
6. On a kind cluster the control plane runs as static Pods, and docker exec plus crictl let you inspect every layer.

In the next article we build exactly this kind of local multi-node cluster ourselves, with Docker Desktop and with the kind CLI.