Lesson 35 of 66 · 16 min read · Updated

Canary and Blue/Green Releases with HTTPRoute Weights

In the previous article you learned how HTTPRoute picks a rule by path, header, query and method, and how filters change a request on the way through. One part of a rule was always trivial: backendRefs had a single entry, so every matched request went to one Service.

A rule can list several backends, each with a weight. The Gateway then splits requests between them in proportion to those weights. That one field is enough to run a canary release, a blue/green switch and an instant rollback, all without touching a Deployment.

You have seen canary and blue/green releases before in this course, in the articles Kubernetes Canary Deployments and Kubernetes Blue/Green Deployments. There, the traffic split came from Pod counts and Service selectors. This article does the same at the Gateway and shows why the result is more precise.

We will cover the following topics:

  1. How backendRefs weights split traffic, and what weight 0 means
  2. Why a weighted split is different from a replica-ratio canary
  3. A progressive canary: 90/10, then 50/50, then 0/100
  4. Header-based canaries, so testers reach the new version before anyone else
  5. Blue/green: two full environments and one weight flip
  6. Measuring the real split with a curl loop
  7. Rolling back in one kubectl apply

Technical Requirements

  1. The minikube cluster with Envoy Gateway v1.9.2 and the storefront-gateway Gateway (GatewayClass eg, listener http on port 80) in the storefront namespace.
  2. The kiosk-api:1.0 image from the Services series.
  3. Docker and the .NET 10 SDK to build kiosk-api:2.0.
  4. kubectl, curl, and a shell with sort and uniq for counting responses.

Core Concepts

Weighted backendRefs

backendRefs:
- name: kiosk-api-v1
port: 80
weight: 90
- name: kiosk-api-v2
port: 80
weight: 10

The rules for weight are short:

RuleConsequence
Weights are relative, not percentages9 and 1 split exactly like 90 and 10. Using numbers that add up to 100 just makes them easier to read
The default weight is 1Two backends without weights split 50/50
weight: 0 sends no trafficThe backend stays in the route, ready to receive traffic again with a one-number change
Weight support is CoreEvery conformant implementation must honour it

The split is decided per request. Each new request draws again, so over a few hundred requests you see roughly the configured ratio, and over a handful you may not.

Replica-Ratio Canary vs Gateway Weights

In the Kubernetes Canary Deployments article, two Deployments shared one label, and one Service selected both. With 9 stable Pods and 1 canary Pod, about 10% of traffic hit the canary. It works, but the traffic share is a side effect of the Pod count.

Replica-ratio canaryHTTPRoute weights
Smallest step1 / total Pods. 1% needs 100 PodsAny ratio, with 2 Pods
Scaling the canaryChanges the traffic shareDoes not change the traffic share
Target specific usersNot possibleHeader, cookie or query matches
Where the split is decidedkube-proxy, per connectionThe Gateway, per request
RollbackScale the canary to 0 and wait for Pods to terminateSet its weight to 0; effective in seconds
Works for traffic between Pods inside the clusterYesOnly for traffic that goes through the Gateway

The last row is the honest limit. A weighted HTTPRoute on a Gateway controls north-south traffic, from outside the cluster in. A Pod calling kiosk-api-v1 directly by its Service name bypasses the Gateway and the split. Splitting east-west traffic is the job of a service mesh, which can use the same HTTPRoute API through the GAMMA initiative.

Progressive Canary

A canary release moves traffic in stages and checks health between them:

Stagev1 weightv2 weightWhat you check before moving on
Start1000v2 Pods are Ready
Canary9010Error rate and latency of v2 match v1
Half5050v2 handles real load; resource usage is acceptable
Done0100Nothing; v1 can be removed later

Both Deployments run at full size during the whole release. That costs more than a replica-ratio canary, but it means any stage can take 100% of traffic immediately, in either direction.

Header-Based Canary

Weights are random. Sometimes you want specific people, such as your QA team, to reach v2 before any real user does. That is a separate rule with a header match:

- matches:
- path:
type: PathPrefix
value: /
headers:
- name: x-canary
value: "true"
backendRefs:
- name: kiosk-api-v2
port: 80

From the previous article you know why this works: for the same path, a match with a header condition ranks higher than a match without one. Requests carrying x-canary: true always go to v2, whatever the weights of the default rule are.

Blue/Green

Blue/green is the canary with only two stages: 100/0 and 0/100. The blue environment serves everyone. The green environment is deployed in full beside it, tested privately through the header rule, and then all traffic flips at once. Blue keeps running until you are confident, so the rollback is the same flip in reverse.

In the Kubernetes Blue/Green Deployments article the flip was a change to a Service selector. Here it is a change of two weights, with the same effect and two advantages: the header route lets you test green through the real Gateway first, and the change does not touch any Service that other workloads may use.

What Weights Do Not Give You

  1. Stickiness. A user may hit v1, then v2, then v1 again. If v2 changes the shape of a response or a cookie, design for both versions coexisting. Gateway API has an experimental sessionPersistence field, and Envoy Gateway offers consistent hashing through its own BackendTrafficPolicy, but neither is part of the Standard channel.
  2. Automation. Something still has to change the weights and watch the metrics. Tools such as Argo Rollouts (with its Gateway API plugin) and Flagger automate exactly this loop by editing the HTTPRoute for you. Doing it by hand first, as in this article, is the best way to understand what they do.

Hands-On: Kubernetes Commands

See the current weights

jsonpath prints each backend with its weight. A missing weight means 1.

kubectl get httproute kiosk-api -n storefront -o jsonpath='{range .spec.rules[*].backendRefs[*]}{.name}={.weight}{"\n"}{end}'

Change weights without editing a file

A JSON patch on the two weight fields of the default rule. Handy in a pipeline.

kubectl patch httproute kiosk-api -n storefront --type=json -p='[{"op":"replace","path":"/spec/rules/1/backendRefs/0/weight","value":50},{"op":"replace","path":"/spec/rules/1/backendRefs/1/weight","value":50}]'

Count which version served 200 requests

Each response contains the Pod name, and the Pod name starts with the Deployment name.

for i in $(seq 1 200); do curl -s -H "Host: storefront.local" http://localhost:8888/; echo; done | grep -o 'kiosk-api-v[12]' | sort | uniq -c

Confirm the route was accepted after a change

A typo in a Service name shows up as ResolvedRefs=False with reason BackendNotFound.

kubectl get httproute kiosk-api -n storefront -o jsonpath='{range .status.parents[*].conditions[*]}{.type}={.status} ({.reason}){"\n"}{end}'

Check both versions are ready before shifting traffic

Never give weight to a Deployment that is not fully available.

kubectl get deploy -n storefront -l app=kiosk-api

Watch Pods of both versions

The version label is added as a column.

kubectl get pods -n storefront -l app=kiosk-api -L version

Step-by-Step Example

The Scenario

The storefront team has built kiosk-api 2.0. It adds a version field to every response. The release plan:

  1. Deploy v2 next to v1 without sending it any real traffic.
  2. Let the QA team test v2 through the real Gateway with an x-canary: true header.
  3. Send 10% of real traffic to v2, then 50%, then 100%.
  4. Practise a rollback.
  5. Run the next release as a blue/green flip.

Step 1 — Build kiosk-api 2.0

Open the KioskApi project from the Services series and add one field. APP_VERSION comes from the Deployment, so the image is the only thing that changes between releases.

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

string podName = Environment.GetEnvironmentVariable("POD_NAME") ?? Environment.MachineName;
string podIp = Environment.GetEnvironmentVariable("POD_IP") ?? "unknown";
string nodeName = Environment.GetEnvironmentVariable("NODE_NAME") ?? "unknown";
string version = Environment.GetEnvironmentVariable("APP_VERSION") ?? "2.0";

app.MapGet("/healthz", () => Results.Ok("healthy"));

app.MapGet("/", (HttpContext context) => Results.Ok(new
{
version,
servedByPod = podName,
podIp,
node = nodeName,
seenClientAddress = context.Connection.RemoteIpAddress?.ToString() ?? "unknown"
}));

app.Run();

Build it against minikube's Docker daemon, as before:

eval $(minikube docker-env)
docker build -t kiosk-api:2.0 .

On Docker Desktop's built-in kind cluster a plain docker build is enough; on a plain kind cluster add kind load docker-image kiosk-api:2.0. A new tag is exactly right here: had you rebuilt kiosk-api:1.0 under the old tag, nodes that already cached it would keep running the old image.

seenClientAddress will show an IPv4-mapped IPv6 address such as ::ffff:10.244.0.18. That is the Envoy Pod's address as Kestrel reports it, not your machine's: the Gateway opens a new connection to the backend.

Step 2 — Two Deployments, Two Services

Both Deployments carry app: kiosk-api, and a second label, version, tells them apart. Each Service selects exactly one version. This is the key difference from the replica-ratio approach: the Gateway, not a shared selector, decides who gets traffic.

The original kiosk-api Deployment from earlier articles is replaced by kiosk-api-v1. Scale it down so its Pods do not confuse the counts:

kubectl scale deployment kiosk-api -n storefront --replicas=0
apiVersion: apps/v1
kind: Deployment
metadata:
name: kiosk-api-v1
namespace: storefront
spec:
replicas: 3
selector:
matchLabels:
app: kiosk-api
version: v1
template:
metadata:
labels:
app: kiosk-api
version: v1
spec:
containers:
- name: kiosk-api
image: kiosk-api:1.0
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
readinessProbe:
httpGet:
path: /healthz
port: http
periodSeconds: 5
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kiosk-api-v2
namespace: storefront
spec:
replicas: 3
selector:
matchLabels:
app: kiosk-api
version: v2
template:
metadata:
labels:
app: kiosk-api
version: v2
spec:
containers:
- name: kiosk-api
image: kiosk-api:2.0
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: APP_VERSION
value: "2.0"
readinessProbe:
httpGet:
path: /healthz
port: http
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: kiosk-api-v1
namespace: storefront
spec:
selector:
app: kiosk-api
version: v1
ports:
- name: http
port: 80
targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: kiosk-api-v2
namespace: storefront
spec:
selector:
app: kiosk-api
version: v2
ports:
- name: http
port: 80
targetPort: 8080
kubectl apply -f 1-kiosk-api-v1-v2.yaml
kubectl get pods -n storefront -l app=kiosk-api -L version
NAME READY STATUS RESTARTS AGE VERSION
kiosk-api-v1-7d6f5c8b9-2gk4p 1/1 Running 0 20s v1
kiosk-api-v1-7d6f5c8b9-9xw7m 1/1 Running 0 20s v1
kiosk-api-v1-7d6f5c8b9-tq3zc 1/1 Running 0 20s v1
kiosk-api-v2-5b8c7d6f4-4hj8n 1/1 Running 0 20s v2
kiosk-api-v2-5b8c7d6f4-kp2rw 1/1 Running 0 20s v2
kiosk-api-v2-5b8c7d6f4-z9m5d 1/1 Running 0 20s v2

Before going on, look at what the old kiosk-api Service now does. Its selector is only app: kiosk-api, so it selects all six Pods, v1 and v2 alike. That is precisely the replica-ratio canary from the earlier article, at 50/50, and nobody chose that ratio. It is a good reminder of why each version gets its own Service here. Nothing in this article routes to the old Service.

Step 3 — The Route: v2 Deployed, Dark

Remove any routes left from earlier articles, then apply a route with two rules. Rule 0 is the header canary for testers. Rule 1 is the default for everyone else, with v2 at weight 0.

kubectl delete httproute --all -n storefront
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: kiosk-api
namespace: storefront
spec:
parentRefs:
- name: storefront-gateway
hostnames:
- storefront.local
rules:
# Rule 0: testers who send x-canary: true always reach v2.
- matches:
- path:
type: PathPrefix
value: /
headers:
- name: x-canary
value: "true"
backendRefs:
- name: kiosk-api-v2
port: 80

# Rule 1: everyone else, split by weight.
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: kiosk-api-v1
port: 80
weight: 100
- name: kiosk-api-v2
port: 80
weight: 0
kubectl apply -f 2-kiosk-api-route.yaml
kubectl get httproute kiosk-api -n storefront -o jsonpath='{range .status.parents[*].conditions[*]}{.type}={.status} ({.reason}){"\n"}{end}'
Accepted=True (Accepted)
ResolvedRefs=True (ResolvedRefs)

Start the port-forward in a second terminal if it is not running:

export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system \
--selector=gateway.envoyproxy.io/owning-gateway-namespace=storefront,gateway.envoyproxy.io/owning-gateway-name=storefront-gateway \
-o jsonpath='{.items[0].metadata.name}')
kubectl -n envoy-gateway-system port-forward service/$ENVOY_SERVICE 8888:80

Count 200 requests:

for i in $(seq 1 200); do curl -s -H "Host: storefront.local" http://localhost:8888/; echo; done | grep -o 'kiosk-api-v[12]' | sort | uniq -c
200 kiosk-api-v1

v2 is running, Ready and attached to the route, and receives nothing. That is what "deployed dark" means.

Step 4 — Testers Reach v2 First

curl -s -H "Host: storefront.local" -H "x-canary: true" http://localhost:8888/
{"version":"2.0","servedByPod":"kiosk-api-v2-5b8c7d6f4-kp2rw","podIp":"10.244.0.52","node":"minikube","seenClientAddress":"::ffff:10.244.0.18"}

Every request with the header goes to v2, every request without it to v1. QA tests the real image, through the real Gateway, on the real host name, and no customer is involved. If v2 is broken, you delete it and nobody outside the team ever knew it existed.

Step 5 — Canary: 90/10

Change the two weights in rule 1:

- name: kiosk-api-v1
port: 80
weight: 90
- name: kiosk-api-v2
port: 80
weight: 10
kubectl apply -f 2-kiosk-api-route.yaml
for i in $(seq 1 200); do curl -s -H "Host: storefront.local" http://localhost:8888/; echo; done | grep -o 'kiosk-api-v[12]' | sort | uniq -c
181 kiosk-api-v1
19 kiosk-api-v2

Close to 90/10, not exact, because each request is an independent draw. Run it again and the numbers move by a few either way.

Now compare with the replica-ratio approach. You have 3 Pods of each version, yet traffic is 90/10. To get 10% with Pod counts you would need 9 v1 Pods for every v2 Pod. Here the Pod count and the traffic share are independent: you can scale v2 for capacity without changing who sees it.

In a real release this is where you stop and watch. Compare v2's error rate and latency with v1's in your monitoring for long enough to see real traffic patterns before moving on.

Step 6 — Half and Half

The JSON patch from the Hands-On section does the same job without editing the file. Rule 1 is index 1 in spec.rules, and its backends are indexes 0 and 1.

kubectl patch httproute kiosk-api -n storefront --type=json -p='[{"op":"replace","path":"/spec/rules/1/backendRefs/0/weight","value":50},{"op":"replace","path":"/spec/rules/1/backendRefs/1/weight","value":50}]'
for i in $(seq 1 200); do curl -s -H "Host: storefront.local" http://localhost:8888/; echo; done | grep -o 'kiosk-api-v[12]' | sort | uniq -c
103 kiosk-api-v1
97 kiosk-api-v2

If you patch, remember that the file in Git no longer matches the cluster. In a GitOps setup, change the file and let the pipeline apply it.

Step 7 — Roll Back

Suppose the dashboards show v2's error rate climbing at 50%. One patch sends everything back to v1:

kubectl patch httproute kiosk-api -n storefront --type=json -p='[{"op":"replace","path":"/spec/rules/1/backendRefs/0/weight","value":100},{"op":"replace","path":"/spec/rules/1/backendRefs/1/weight","value":0}]'
for i in $(seq 1 200); do curl -s -H "Host: storefront.local" http://localhost:8888/; echo; done | grep -o 'kiosk-api-v[12]' | sort | uniq -c
200 kiosk-api-v1

Nothing was scaled, restarted or rescheduled. Envoy Gateway pushed new configuration to Envoy, and the next request already went to v1. The v2 Pods are still running, so the team can debug them with the header route while customers are safe on v1. Compare that with the replica-ratio rollback, where you scale the canary to zero and wait for Pods to terminate.

Step 8 — Complete the Release: 0/100

The fix is in and v2 has passed the canary stages again. Finish:

kubectl patch httproute kiosk-api -n storefront --type=json -p='[{"op":"replace","path":"/spec/rules/1/backendRefs/0/weight","value":0},{"op":"replace","path":"/spec/rules/1/backendRefs/1/weight","value":100}]'
for i in $(seq 1 200); do curl -s -H "Host: storefront.local" http://localhost:8888/; echo; done | grep -o 'kiosk-api-v[12]' | sort | uniq -c
200 kiosk-api-v2

Leave v1 running for a while at weight 0. It is your instant rollback until you are confident. Then scale it down or delete it.

Step 9 — The Next Release as Blue/Green

For the next release, the team prefers a single switch. The names change to describe the roles: blue is what serves customers now, green is the new release. In this cluster, blue is kiosk-api-v2 and green would be a new kiosk-api-v3 Deployment and Service built the same way as in Step 2.

The route has the same shape as before. The header rule points at green, and the default rule has two backends at 100 and 0:

rules:
- matches:
- path:
type: PathPrefix
value: /
headers:
- name: x-canary
value: "true"
backendRefs:
- name: kiosk-api-v3 # green: testers only
port: 80

- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: kiosk-api-v2 # blue: everyone
port: 80
weight: 100
- name: kiosk-api-v3 # green: nobody yet
port: 80
weight: 0

Testing works exactly as in Step 4. The switch is one patch that swaps the two numbers, 100/0 to 0/100, and the rollback is the same patch in reverse. There are no intermediate stages, which is the whole difference between blue/green and canary. Blue/green is simpler to reason about; a canary limits how many users a bad release can hurt. Both use the same field.

You do not need to build v3 to finish this article. You have already seen the flip itself in Steps 7 and 8, which were a blue/green switch in both directions between v1 and v2.

Step 10 — Clean Up

Remove the route and both versioned Deployments and Services, and bring the original kiosk-api Deployment back for the next article.

kubectl delete httproute kiosk-api -n storefront
kubectl delete -f 1-kiosk-api-v1-v2.yaml
kubectl scale deployment kiosk-api -n storefront --replicas=3
kubectl get pods -n storefront -l app=kiosk-api

Summary

  1. A rule can list several backendRefs, each with a weight. Weights are relative, default to 1, and 0 means no traffic. Weight support is Core.
  2. The split is decided per request at the Gateway, so the traffic share no longer depends on Pod counts. 10% with 2 Pods is fine.
  3. Give each version its own Deployment and its own Service with a version label in the selector. A shared selector silently turns back into a replica-ratio split.
  4. A canary moves through stages such as 100/0, 90/10, 50/50 and 0/100, with a health check between them.
  5. A second rule with a header match, such as x-canary: true, lets testers reach the new version through the real Gateway before any customer does. It wins because a header match outranks the same path without one.
  6. Blue/green is the same mechanism with only two states. The switch and the rollback are one weight change each, effective within seconds, with no Pod restarts.
  7. Weights only control traffic that passes through the Gateway, and they give no session stickiness. Tools like Argo Rollouts and Flagger automate the weight changes for you.

In the next article we move beyond HTTP and expose .NET gRPC services through the same Gateway with GRPCRoute.