Lesson 11 of 66 · 16 min read · Updated

Native Sidecar Containers and Multi-Container Pod Patterns

In the previous article init containers prepared a Pod and then disappeared. Many helpers should not disappear. A log shipper must keep reading log files for as long as the application writes them. A proxy must keep forwarding requests. A metrics converter must answer every scrape. These helpers are sidecars: containers that run next to the application, in the same Pod, for its whole life.

For years Kubernetes had no real concept of a sidecar. People simply added a second container to containers and lived with the consequences: no start order, sidecars that died before the application during shutdown, and Jobs that never finished. Native sidecar containers, stable since Kubernetes 1.33, fix all three. This article shows how they work and builds the three classic multi-container patterns on top of them, around a .NET 10 API.

We will cover the following topics:

  1. Why a Pod has more than one container at all
  2. Native sidecars: initContainers with restartPolicy: Always
  3. Start-up and shutdown order, measured on a real cluster
  4. Why Jobs with classic sidecars never complete, and how native sidecars fix it
  5. The sidecar, ambassador and adapter patterns
  6. Handling SIGTERM in sidecars

Technical Requirements

  1. Kubernetes 1.33 or later. The examples run on Docker Desktop's kind-based cluster, Kubernetes v1.36, with three worker nodes.
  2. kubectl, Docker and the .NET 10 SDK.
  3. On plain kind, load the image you build with kind load docker-image.

Core Concepts

Why Several Containers in One Pod

Containers in the same Pod share three things:

  1. the network namespace: they reach each other on localhost, and share one IP address,
  2. volumes that both mount,
  3. the Pod's life cycle and node: they are scheduled, started and deleted together.

That makes a Pod the right unit for a process and its tightly coupled helpers. The rule of thumb: put containers in one Pod only if they must run on the same machine and scale together. Two independent services belong in two Deployments.

Classic Sidecars and Their Problems

Before native sidecars, a helper was just a second entry in containers. That caused three well-known problems:

ProblemWhy it happens
No start orderAll containers start in parallel, so the app may call a proxy that is not listening yet
Wrong shutdown orderAll containers get SIGTERM at the same time, so a log shipper may stop before the app writes its last lines
Jobs never finishA Job completes when all containers have exited; a sidecar that runs forever keeps the Pod alive

Native Sidecars

A native sidecar is declared in initContainers with one extra field:

initContainers:
- name: log-shipper
image: busybox:1.37
restartPolicy: Always

That single line changes its life cycle:

Regular init containerNative sidecarApplication container
Declared ininitContainersinitContainers + restartPolicy: Alwayscontainers
StartIn order, must exitIn order, keeps runningAfter all init containers
Blocks the next container untilIt exits with 0It has started (or its startupProbe passed)
ProbesNoYes: startup, liveness, readinessYes
Restarted if it exitsDepends on Pod policyAlways, without restarting the appDepends on Pod policy
ShutdownAlready goneAfter all application containers have stopped, in reverse orderFirst
Counts toward Job completionNoYes

The feature appeared in 1.28, was enabled by default from 1.29, and became stable (GA) in 1.33. On 1.33 and later the feature gate is locked on.

Start-up Order and startupProbe

The kubelet starts init containers in order. For a native sidecar it waits until the container has started before moving on. If the sidecar has a startupProbe, it waits until the probe succeeds. This is how you guarantee that a proxy is actually listening before the application starts.

Shutdown Order

When a Pod is deleted, the kubelet first sends SIGTERM to the application containers and waits for them to exit. Only then does it stop the sidecars, in the reverse order of their declaration. A log shipper therefore sees the application's last log lines.

All of this happens inside the Pod's terminationGracePeriodSeconds (30 seconds by default). If the sidecar ignores SIGTERM, the kubelet waits until the grace period runs out and then kills it. The step-by-step example measures both cases.

The Three Classic Patterns

PatternWhat the helper doesExample
SidecarExtends the app without changing itShips log files, refreshes certificates, syncs files from Git
AmbassadorProxies the app's outgoing connectionsThe app calls localhost:9000; the ambassador forwards to the real backend, adds TLS, retries or auth
AdapterConverts the app's output into a standard formatTurns the app's JSON stats into Prometheus metrics

All three are "sidecars" in the broad sense. The names describe the direction of the data.

Hands-On: Kubernetes Commands

See how many containers are ready

READY counts native sidecars too: 4/4 means three sidecars plus the app.

kubectl get pods -n mc-sidecar

Read the logs of a sidecar

Native sidecars are addressed by name with -c, like any container.

kubectl logs -n mc-sidecar deploy/orders-api -c log-shipper

Read all containers' logs at once

--all-containers includes init containers and sidecars.

kubectl logs -n mc-sidecar deploy/orders-api --all-containers --prefix

Check which init containers are sidecars

The container-level restartPolicy field marks them.

kubectl get pod -n mc-sidecar <pod> -o jsonpath='{range .spec.initContainers[*]}{.name}{"\t"}{.restartPolicy}{"\n"}{end}'

Show start times and restart counts per container

Useful to verify ordering and independent restarts.

kubectl get pod -n mc-sidecar <pod> -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{" started="}{.state.running.startedAt}{" restarts="}{.restartCount}{"\n"}{end}{range .status.containerStatuses[*]}{.name}{" started="}{.state.running.startedAt}{" restarts="}{.restartCount}{"\n"}{end}'

Step-by-Step Example

The Scenario

An orders-api written in .NET 10 writes its log to a file, exposes its own JSON statistics and calls an inventory service. We add:

  1. a log-shipper sidecar that streams the log file to stdout,
  2. an inventory-ambassador that the app reaches on localhost:9000,
  3. a metrics-adapter that serves the app's statistics in Prometheus format.

Along the way we measure start-up and shutdown order, and compare Jobs with classic and native sidecars.

Step 1 — The Application

dotnet new web -n OrdersApi

Program.cs:

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

string logFile = Environment.GetEnvironmentVariable("ORDER_LOG") ?? "/var/log/orders/orders.log";
Directory.CreateDirectory(Path.GetDirectoryName(logFile)!);
int ordersPlaced = 0;
var uptime = System.Diagnostics.Stopwatch.StartNew();

void Log(string message) =>
File.AppendAllText(logFile, $"{DateTime.UtcNow:O} {message}{Environment.NewLine}");

Log("orders-api started");

app.MapPost("/orders", (OrderRequest order) =>
{
int id = Interlocked.Increment(ref ordersPlaced);
Log($"order {id} placed for {order.Product} x{order.Quantity}");
return Results.Created($"/orders/{id}", new { id, order.Product, order.Quantity });
});

// JSON stats in the app's own format; an adapter turns this into Prometheus text.
app.MapGet("/stats", () => Results.Ok(new { ordersPlaced, uptimeSeconds = (int)uptime.Elapsed.TotalSeconds }));

// Calls a backend through the ambassador on localhost.
app.MapGet("/inventory", async () =>
{
using var http = new HttpClient { BaseAddress = new Uri("http://localhost:9000") };
return Results.Text(await http.GetStringAsync("/"));
});

app.Lifetime.ApplicationStopping.Register(() => Log("orders-api stopping"));
app.Run();

record OrderRequest(string Product, int Quantity);

The app knows nothing about Kubernetes, log collectors or Prometheus. It writes a file, serves JSON and calls localhost:9000. That is the point: the helpers adapt it to the platform.

The Dockerfile:

FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY OrdersApi/ .
RUN dotnet publish OrdersApi.csproj -c Release -o /app

FROM mcr.microsoft.com/dotnet/aspnet:10.0
WORKDIR /app
COPY --from=build /app .
ENV ASPNETCORE_HTTP_PORTS=8080
EXPOSE 8080
ENTRYPOINT ["dotnet", "OrdersApi.dll"]
docker build -t shop-orders-api:1.0 .

Step 2 — A Native Log-Shipper Sidecar

Start with just the log shipper. The app and the sidecar share an emptyDir mounted at /var/log/orders:

apiVersion: v1
kind: Namespace
metadata:
name: mc-sidecar
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-api
namespace: mc-sidecar
spec:
replicas: 1
selector:
matchLabels:
app: orders-api
template:
metadata:
labels:
app: orders-api
spec:
initContainers:
- name: log-shipper
image: busybox:1.37
restartPolicy: Always # this one line makes it a native sidecar
command: ["sh", "-c", "trap 'exit 0' TERM; touch /var/log/orders/orders.log; tail -n +1 -F /var/log/orders/orders.log & wait"]
volumeMounts:
- name: logs
mountPath: /var/log/orders
containers:
- name: orders-api
image: shop-orders-api:1.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
volumeMounts:
- name: logs
mountPath: /var/log/orders
volumes:
- name: logs
emptyDir: {}

The trap 'exit 0' TERM ... & wait wrapper is explained in Step 5. In a real cluster the log shipper would be Fluent Bit or the OpenTelemetry Collector. busybox tail keeps the example small.

kubectl apply -f 1-orders-with-log-sidecar.yaml
kubectl rollout status deploy/orders-api -n mc-sidecar
kubectl get pods -n mc-sidecar
NAME READY STATUS RESTARTS AGE
orders-api-65ff9549f5-zgr8w 2/2 Running 0 3m19s

Check the start order:

kubectl get pod -n mc-sidecar <pod> -o jsonpath='...'
log-shipper ready=true started=2026-10-21T08:35:26Z
orders-api ready=true started=2026-10-21T08:38:42Z

The sidecar started first. In this run the app started three minutes later only because the node was busy pulling its image. The order is what matters: the sidecar was already running when the first log line was written.

Place two orders and read the sidecar's output:

kubectl port-forward -n mc-sidecar deploy/orders-api 8080:8080
curl -X POST localhost:8080/orders -H 'Content-Type: application/json' -d '{"product":"Keyboard","quantity":2}'
{"id":1,"product":"Keyboard","quantity":2}
curl -X POST localhost:8080/orders -H 'Content-Type: application/json' -d '{"product":"Mouse","quantity":1}'
{"id":2,"product":"Mouse","quantity":1}

kubectl logs -n mc-sidecar deploy/orders-api -c log-shipper
2026-10-21T08:38:42.9232855Z orders-api started
2026-10-21T08:38:55.5444250Z order 1 placed for Keyboard x2
2026-10-21T08:38:55.5627344Z order 2 placed for Mouse x1

A file-based log has become a container log stream that any cluster log collector can pick up.

Step 3 — Measure the Shutdown Order

To see the order precisely, use a Pod where each container prints a timestamp when it receives SIGTERM. The app takes 5 seconds to drain:

apiVersion: v1
kind: Pod
metadata:
name: shutdown-order
namespace: mc-sidecar
spec:
initContainers:
- name: sidecar
image: busybox:1.37
restartPolicy: Always
command: ["sh", "-c", "trap 'echo \"$(date +%T) sidecar: SIGTERM received\"; exit 0' TERM; echo \"$(date +%T) sidecar: started\"; while true; do sleep 1; done"]
containers:
- name: app
image: busybox:1.37
command: ["sh", "-c", "trap 'echo \"$(date +%T) app: SIGTERM received, draining for 5s\"; sleep 5; echo \"$(date +%T) app: exiting\"; exit 0' TERM; echo \"$(date +%T) app: started\"; while true; do sleep 1; done"]

Follow both logs in the background, then delete the Pod:

kubectl apply -f 2-shutdown-order.yaml
kubectl wait --for=condition=Ready pod/shutdown-order -n mc-sidecar
kubectl logs -f shutdown-order -n mc-sidecar -c sidecar > sidecar.log &
kubectl logs -f shutdown-order -n mc-sidecar -c app > app.log &
kubectl delete pod shutdown-order -n mc-sidecar
sort app.log sidecar.log
08:40:06 app: started
08:40:06 sidecar: started
08:40:09 app: SIGTERM received, draining for 5s
08:40:14 app: exiting
08:40:15 sidecar: SIGTERM received

The app received SIGTERM first and drained for five seconds. The sidecar got its SIGTERM only after the app had exited. With a classic sidecar in containers, both would have received SIGTERM at 08:40:09.

The same thing happened to the real log shipper from Step 2. When its Pod was replaced, the sidecar was still running when the app wrote its final line:

2026-10-21T08:38:55.5627344Z order 2 placed for Mouse x1
2026-10-21T08:39:28.9997650Z orders-api stopping

Step 4 — Jobs: Classic versus Native Sidecars

Run the same five-second report job twice. First with the helper in containers:

apiVersion: batch/v1
kind: Job
metadata:
name: report-classic
namespace: mc-sidecar
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
containers:
- name: report
image: busybox:1.37
command: ["sh", "-c", "echo generating report; sleep 5; echo report done"]
- name: log-shipper
image: busybox:1.37
command: ["sh", "-c", "while true; do sleep 1; done"]

Then with the helper as a native sidecar:

apiVersion: batch/v1
kind: Job
metadata:
name: report-native
namespace: mc-sidecar
spec:
backoffLimit: 0
template:
spec:
restartPolicy: Never
initContainers:
- name: log-shipper
image: busybox:1.37
restartPolicy: Always
command: ["sh", "-c", "while true; do sleep 1; done"]
containers:
- name: report
image: busybox:1.37
command: ["sh", "-c", "echo generating report; sleep 5; echo report done"]
kubectl apply -f 3-job-classic-sidecar.yaml -f 4-job-native-sidecar.yaml
kubectl wait --for=condition=Complete job/report-native -n mc-sidecar
kubectl get jobs,pods -n mc-sidecar
NAME STATUS COMPLETIONS DURATION AGE
job.batch/report-classic Running 0/1 50s 50s
job.batch/report-native Complete 1/1 40s 50s

NAME READY STATUS RESTARTS AGE
pod/report-classic-8jvkv 1/2 NotReady 0 50s
pod/report-native-89g6d 0/2 Completed 0 50s

The report finished in both Pods. The classic Job is still Running with 1/2 containers, and it will stay like that forever because its log shipper never exits. The native Job is Complete: once the main container finished, the kubelet stopped the sidecar. Before native sidecars, teams had to make the main container signal the sidecar through a shared file or a special endpoint to get this behaviour.

Delete both Jobs:

kubectl delete job report-classic report-native -n mc-sidecar

Step 5 — Why the trap Matters

The log shipper in Step 2 wraps tail in trap 'exit 0' TERM ... & wait. Without it, the shell runs tail as PID 1 of the container, PID 1 has no SIGTERM handler, and the signal is ignored. The kubelet then waits for the full grace period before killing the container.

We measured both versions by deleting a Pod and timing kubectl delete:

tail-plain deleted in 32s
tail-trap deleted in 1s

32 seconds is the default 30-second grace period plus overhead. Every rollout, scale-down and node drain pays that cost for every Pod.

One slow sidecar is enough to cause it. In the full Deployment of Step 6 we first wrapped only the log shipper, and deleting the Pod still took 31 seconds. The culprit was the metrics adapter: busybox httpd -f running as PID 1 ignores SIGTERM too. After wrapping it the same way, the Pod was gone in 1 second. Make sure every sidecar, whether it is a shell script, Fluent Bit or a .NET process, exits promptly on SIGTERM.

Step 6 — The Ambassador and the Adapter

Add a small inventory backend that the app will reach through the ambassador:

apiVersion: v1
kind: ConfigMap
metadata:
name: inventory-content
namespace: mc-sidecar
data:
index.html: |
{"Keyboard": 12, "Mouse": 40, "Monitor": 3}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: inventory-backend
namespace: mc-sidecar
spec:
replicas: 1
selector:
matchLabels:
app: inventory-backend
template:
metadata:
labels:
app: inventory-backend
spec:
containers:
- name: web
image: busybox:1.37
command: ["sh", "-c", "trap 'exit 0' TERM; httpd -f -p 8080 -h /www & wait"]
ports:
- containerPort: 8080
volumeMounts:
- name: content
mountPath: /www
volumes:
- name: content
configMap:
name: inventory-content
---
apiVersion: v1
kind: Service
metadata:
name: inventory-backend
namespace: mc-sidecar
spec:
selector:
app: inventory-backend
ports:
- port: 80
targetPort: 8080

The adapter is a CGI script served by busybox httpd, wrapped in the same SIGTERM trap as the log shipper. It reads the app's JSON and prints Prometheus text format:

apiVersion: v1
kind: ConfigMap
metadata:
name: metrics-adapter
namespace: mc-sidecar
data:
metrics: |
#!/bin/sh
# Adapter: read the app's JSON stats and print Prometheus text format.
stats=$(wget -qO- http://localhost:8080/stats)
orders=$(echo "$stats" | sed -n 's/.*"ordersPlaced":\([0-9]*\).*/\1/p')
uptime=$(echo "$stats" | sed -n 's/.*"uptimeSeconds":\([0-9]*\).*/\1/p')
echo "Content-Type: text/plain; version=0.0.4"
echo ""
echo "# HELP orders_placed_total Orders placed since the process started."
echo "# TYPE orders_placed_total counter"
echo "orders_placed_total ${orders:-0}"
echo "# HELP orders_api_uptime_seconds Seconds since the process started."
echo "# TYPE orders_api_uptime_seconds gauge"
echo "orders_api_uptime_seconds ${uptime:-0}"

Now the full Deployment with all three helpers as native sidecars:

apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-api
namespace: mc-sidecar
spec:
replicas: 1
selector:
matchLabels:
app: orders-api
template:
metadata:
labels:
app: orders-api
spec:
initContainers:
- name: log-shipper # sidecar pattern
image: busybox:1.37
restartPolicy: Always
command: ["sh", "-c", "trap 'exit 0' TERM; touch /var/log/orders/orders.log; tail -n +1 -F /var/log/orders/orders.log & wait"]
volumeMounts:
- name: logs
mountPath: /var/log/orders
- name: inventory-ambassador # ambassador pattern
image: alpine/socat:1.8.0.3
restartPolicy: Always
args: ["TCP-LISTEN:9000,fork,reuseaddr", "TCP:inventory-backend.mc-sidecar.svc.cluster.local:80"]
startupProbe:
tcpSocket:
port: 9000
periodSeconds: 1
failureThreshold: 30
- name: metrics-adapter # adapter pattern
image: busybox:1.37
restartPolicy: Always
command: ["sh", "-c", "trap 'exit 0' TERM; httpd -f -p 9100 -h /www & wait"]
ports:
- name: metrics
containerPort: 9100
volumeMounts:
- name: adapter-script
mountPath: /www/cgi-bin
containers:
- name: orders-api
image: shop-orders-api:1.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
volumeMounts:
- name: logs
mountPath: /var/log/orders
volumes:
- name: logs
emptyDir: {}
- name: adapter-script
configMap:
name: metrics-adapter
defaultMode: 0755

The ambassador has a startupProbe on port 9000. Because it is a native sidecar, the kubelet does not start the next container until that probe passes, so the app can never call localhost:9000 before socat is listening.

kubectl apply -f 5-inventory-backend.yaml -f 6-orders-patterns.yaml
kubectl rollout status deploy/orders-api -n mc-sidecar
kubectl get pods -n mc-sidecar -l app=orders-api
NAME READY STATUS RESTARTS AGE
orders-api-9d8f945b4-z9nzh 4/4 Running 0 17s

Four of four: three sidecars and the app. The start times follow the declaration order:

log-shipper started=2026-10-21T08:41:46Z ready=true
inventory-ambassador started=2026-10-21T08:41:50Z ready=true
metrics-adapter started=2026-10-21T08:41:51Z ready=true
orders-api started=2026-10-21T08:41:51Z

Step 7 — Exercise All Three Patterns

Forward the app port and the adapter port:

kubectl port-forward -n mc-sidecar deploy/orders-api 8080:8080 9100:9100

Place an order, call the inventory through the ambassador, and scrape the adapter:

curl -X POST localhost:8080/orders -H 'Content-Type: application/json' -d '{"product":"Monitor","quantity":1}'
{"id":1,"product":"Monitor","quantity":1}

curl localhost:8080/inventory
{"Keyboard": 12, "Mouse": 40, "Monitor": 3}

curl localhost:9100/cgi-bin/metrics
# HELP orders_placed_total Orders placed since the process started.
# TYPE orders_placed_total counter
orders_placed_total 1
# HELP orders_api_uptime_seconds Seconds since the process started.
# TYPE orders_api_uptime_seconds gauge
orders_api_uptime_seconds 11

The app called localhost:9000 and got the inventory from another Service. It never learned that Service's name. The adapter turned {"ordersPlaced":1,...} into metrics that Prometheus understands, without a single change to the .NET code. In a real cluster you would point a Prometheus scrape config or a ServiceMonitor at port 9100.

Step 8 — Sidecars Restart Independently

Stop the log-shipper container through the container runtime on its node. On kind, the node is a Docker container, so crictl is one docker exec away:

N=$(kubectl get pod -n mc-sidecar <pod> -o jsonpath='{.spec.nodeName}')
C=$(docker exec $N crictl ps --name log-shipper --pod $(docker exec $N crictl pods --name <pod> -q) -q)
docker exec $N crictl stop --timeout 0 $C
log-shipper restarts=1
inventory-ambassador restarts=0
metrics-adapter restarts=0
orders-api restarts=0

The kubelet restarted only the sidecar. The app kept serving. A native sidecar is always restarted, regardless of the Pod's own restartPolicy.

Step 9 — Clean Up

kubectl delete namespace mc-sidecar

Summary

  1. Containers in one Pod share localhost, volumes and life cycle; use that only for tightly coupled helpers.
  2. A native sidecar is an init container with restartPolicy: Always. Stable since Kubernetes 1.33.
  3. Native sidecars start before the app (after their startupProbe passes, if they have one), stop after it, and restart on their own.
  4. Jobs complete normally with native sidecars; with classic sidecars they hang in Running.
  5. The sidecar pattern extends the app, the ambassador proxies its outgoing calls, the adapter converts its output. The .NET code stayed unaware of all three.
  6. Every sidecar must exit on SIGTERM. A shell sidecar without a trap cost 32 seconds per Pod deletion in our test, against 1 second with one.

This completes the multi-container patterns. The next topics build on them: namespaces and quotas to organise workloads, and later, service meshes and observability agents that are delivered as exactly these kinds of sidecars.