Lesson 51 of 66 · 17 min read · Updated
Event-Driven Autoscaling with KEDA
On this page
The first two articles in this section scaled on CPU and memory. That works for an API, because a busy API burns CPU. It does not work for a background worker that reads from a queue. A worker that is waiting for messages uses no CPU at all, and a worker with 10,000 messages behind it may still look calm if each message mostly waits on I/O. The number that matters is the backlog, and the HPA cannot see it.
KEDA (Kubernetes Event-Driven Autoscaling) fills that gap. It reads metrics from more than 70 sources (queues, streams, databases, cloud services, cron schedules) and feeds them to the HPA. It also does something the HPA cannot do on its own: it scales a workload down to zero when there is nothing to do and back up when work arrives.
In this article a .NET 10 worker consumes orders from RabbitMQ. With an empty queue it runs zero Pods. Publish 300 messages and it grows to ten, drains the queue, and disappears again. If RabbitMQ itself is new to you, the site's RabbitMQ category covers queues, exchanges and acknowledgements in depth.
We will cover the following topics:
- How KEDA extends the HPA instead of replacing it
ScaledObject,TriggerAuthenticationandScaledJob- Activation versus scaling, and how scale to zero works
- A RabbitMQ
QueueLengthtrigger for a .NET worker built on RabbitMQ.Client 7 - Pausing autoscaling and reading KEDA's metrics directly
Technical Requirements
- A local multi-node cluster. The examples use Docker Desktop's Kubernetes with the kind provisioner (one control-plane node and three workers), Kubernetes v1.36.
- Helm 3 or 4.
- The .NET 10 SDK and Docker to build the worker image.
- metrics-server is not required for this article. KEDA serves its own metrics.
Core Concepts
KEDA Is an HPA Feeder
KEDA does not have its own scaling algorithm for the range from one Pod to many. It installs three components:
| Component | Job |
|---|---|
Operator (keda-operator) | Watches ScaledObject resources, creates a matching HPA for each, polls the event sources, and handles the move between 0 and 1 replica itself. |
Metrics server (keda-operator-metrics-apiserver) | Registers itself as the cluster's external.metrics.k8s.io API, so the HPA can ask it for values like "messages in the orders queue". |
| Admission webhooks | Validate KEDA resources, for example rejecting two ScaledObjects that target the same Deployment. |
So when you create a ScaledObject, you get an ordinary HPA named keda-hpa-<name>. Everything from the previous articles about the HPA formula, behavior and stabilization still applies between 1 and the maximum. KEDA adds the 0 to 1 and 1 to 0 steps, which the HPA cannot take, because the HPA has no Pods to measure at zero.
Because KEDA occupies the external.metrics.k8s.io API, only one component can serve it per cluster. If another adapter already does, the two conflict.
ScaledObject
A ScaledObject connects a workload to one or more triggers:
| Field | Meaning |
|---|---|
scaleTargetRef | The Deployment (or StatefulSet, or any resource with a /scale subresource) to scale. |
minReplicaCount | Lowest replica count. 0 enables scale to zero. The default is 0. |
maxReplicaCount | Highest replica count. The default is 100. |
pollingInterval | How often KEDA checks the triggers, in seconds. The default is 30. |
cooldownPeriod | How long all triggers must be inactive before scaling from 1 to 0. The default is 300. |
triggers | The event sources, each with a type and type-specific metadata. |
advanced.horizontalPodAutoscalerConfig | Lets you set behavior on the generated HPA. |
Do not create your own HPA for a workload that has a ScaledObject. KEDA owns that HPA.
Activation and Scaling
Every trigger has two thresholds, and they do different jobs:
- activationValue decides whether the workload should exist at all. KEDA scales from 0 to 1 when the metric goes above it, and back to 0 when it drops to or below it for longer than
cooldownPeriod. The default is 0. - value is the per-Pod target that the HPA uses between 1 and the maximum. With
value: "10"and 95 messages waiting, the HPA wantsceil(95 / 10) = 10Pods.
With activationValue: "0", a single message wakes the worker. With activationValue: "50", small backlogs wait until a bigger batch is worth starting a Pod for.
The RabbitMQ Trigger
The rabbitmq scaler reads queue statistics. Its main settings:
| Metadata | Meaning |
|---|---|
queueName | The queue to watch. |
mode | QueueLength (messages waiting) or MessageRate (publish rate per second). Other modes exist for specific needs. |
value | Target per Pod, in the unit of the mode. |
activationValue | The 0-to-1 threshold described above. |
protocol | amqp, http, or auto, which picks from the host URL. |
The connection string, including the password, should not sit in the ScaledObject. It goes into a Secret, and a TriggerAuthentication maps a Secret key to the trigger's host parameter. A ClusterTriggerAuthentication does the same for every namespace.
ScaledJob
A ScaledObject keeps long-running Pods and changes how many there are. A ScaledJob creates Kubernetes Jobs instead: one Job per unit of work, each running to completion and exiting.
| Use | Choose |
|---|---|
| A worker that loops forever and handles many messages | ScaledObject on a Deployment |
| Long or heavy tasks (video encoding, report generation) that should not be interrupted by scale-down | ScaledJob |
Scaling down a Deployment kills Pods that might be halfway through a message. A Job is never killed by KEDA scaling down. It finishes and exits. That is the main reason to choose ScaledJob for long tasks.
Hands-On: Kubernetes Commands
List ScaledObjects and their state
READY means the trigger configuration works. ACTIVE means the activation threshold is currently exceeded.
See the HPA that KEDA created
Its target is an External metric named after the trigger.
Read KEDA's events
Scaler errors (unreachable host, missing queue, bad credentials) show up here first.
Query the external metric directly
This is exactly what the HPA receives.
Pause autoscaling at a fixed replica count
Useful during maintenance. Remove the annotation to resume.
Check the queue inside RabbitMQ
Messages waiting and consumers attached, straight from the broker.
Step-by-Step Example
The Scenario
An online shop publishes orders to a durable RabbitMQ queue called orders. A .NET worker takes one order at a time and spends about a second on each. Most of the day the queue is empty, and paying for idle workers makes no sense. During a campaign, hundreds of orders arrive within seconds. You want zero workers when idle, up to ten during a burst, and each worker handling a share of about ten messages.
Step 1 — Install KEDA
KEDA 2.21 is the current release. Install it from the official chart into its own namespace:
Wait until the external metrics API reports True. It can take a minute after the Pods start:
Step 2 — Build the Worker
The same small program is both the consumer and, with the argument publish <count>, the producer you will use to create a burst. Create it:
The current RabbitMQ.Client is version 7, and its API is fully asynchronous: CreateConnectionAsync, CreateChannelAsync, BasicConsumeAsync, and an AsyncEventingBasicConsumer with a ReceivedAsync event. Code written for version 6 (CreateModel, EventingBasicConsumer) does not compile against it.
Replace Program.cs:
Two choices in this code matter for autoscaling:
prefetchCount: 1stops RabbitMQ from pushing a large batch of messages to the first Pod that connects. Without it, one Pod would hoard the backlog and the new Pods would sit idle.autoAck: falsewith an explicitBasicAckAsyncafter the work is done means that a Pod killed during scale-down loses nothing. Its unacknowledged message goes back to the queue for another Pod.
A Dockerfile using the smaller runtime image, since this is not a web app:
Docker Desktop's kind cluster can use local images directly. With a standalone kind cluster, run kind load docker-image order-worker:1.0 first.
Step 3 — Run RabbitMQ
Save as 2-rabbitmq.yaml. A single broker is enough for the demo. The Secret holds the broker credentials and the AMQP URI that KEDA will use.
For production, run RabbitMQ with the RabbitMQ Cluster Operator and quorum queues, not a single Deployment.
Step 4 — Deploy the Worker at Zero Replicas
Save as 3-order-worker.yaml. It starts with replicas: 0. KEDA decides from here on.
Step 5 — Create the TriggerAuthentication and ScaledObject
Save as 4-scaledobject.yaml. The short pollingInterval and cooldownPeriod make the demo quick to watch. In production the defaults of 30 and 300 seconds are more sensible.
KEDA has created keda-hpa-order-worker. Its MINPODS is 1, not 0, because an HPA cannot go to zero. KEDA handles that step itself. The HPA's metric is an external one:
The s0- prefix is the trigger's index, which keeps metric names unique when a ScaledObject has several triggers.
Step 6 — Create a Burst
In two extra terminals, watch the Deployment and the HPA:
Save as 5-publish-job.yaml. It runs the same image in publish mode and puts 300 orders in the queue:
Step 7 — Watch Zero Become Ten
The Deployment watch, trimmed to the lines where something changed:
And the HPA watch:
Read the two together:
- KEDA saw messages above the activation value and moved the Deployment from 0 to 1 on its own. That is the
KEDAScaleTargetActivatedevent below. - From there the HPA took over. With 295 messages and one Pod, the average was 295 against a target of 10, so it wanted 30 Pods. The default scale-up policy (double the Pods or add four, whichever is more, every 15 seconds) took it to 4 and then 8, and
maxReplicaCountcapped it at 10. 58750mis Kubernetes notation for 58.75. That is 235 messages divided across 4 Pods.
Each worker logs the orders it handles:
Midway through, the broker shows the backlog shrinking with all ten consumers attached:
Step 8 — Watch Ten Become Zero
The queue empties within seconds. The HPA alone would now spend five minutes stepping down, because of its default 300-second scale-down window. KEDA does not wait for that. Once the trigger has been inactive for the 30-second cooldownPeriod, it sets the Deployment straight to zero:
The ScaledObject's events tell the whole story, including one detail worth knowing:
The NOT_FOUND - no queue 'orders' warnings appeared because the ScaledObject was created before anything had declared the queue. The worker declares it, but the worker was at zero Pods. Only the publisher's QueueDeclareAsync created it. This chicken-and-egg problem is common with scale to zero. In production, declare queues ahead of time, through the broker's definitions, an operator, or a migration step, so that the scaler never looks at a queue that does not exist.
Step 9 — Pause and Inspect
Look at the metric the HPA is reading now that the queue is empty:
Pin the worker at two replicas, for example while you debug it:
Two Pods start even though the queue is empty, and PAUSED is True. Remove the annotation and KEDA takes them back to zero:
Step 10 — The Same Trigger as a ScaledJob
If each message were a ten-minute report instead of a one-second order, you would not want scale-down to kill a Pod halfway through. The ScaledJob version reuses the same TriggerAuthentication and starts one Job per message:
The container for a ScaledJob must take one message (or a fixed batch), process it, acknowledge it and exit. A consumer that loops forever, like order-worker, would turn every Job into a permanent Pod.
Step 11 — Clean Up
Summary
- CPU is the wrong signal for queue workers. KEDA scales on the event source itself: queue length, stream lag, a cron schedule, and 70+ other scalers.
- KEDA does not replace the HPA. A
ScaledObjectcreates an HPA (keda-hpa-<name>) that handles 1 to N, and the KEDA operator handles 0 to 1 and 1 to 0 itself. activationValuedecides whether the workload runs at all.valueis the per-Pod target used by the HPA.cooldownPeriodis how long the trigger must be quiet before KEDA scales to zero.- Keep credentials in a Secret and connect them with a
TriggerAuthentication. - For RabbitMQ consumers, set
prefetchCountlow and acknowledge manually, so new Pods share the backlog and scale-down never loses a message. RabbitMQ.Client 7 is async-only. - Make sure the queue exists before the ScaledObject looks at it. Otherwise the scaler logs
NOT_FOUNDuntil a producer creates it. - Use a ScaledJob for long tasks. Each unit of work gets its own Job, which scale-down never interrupts.
That completes the scaling triad: the HPA for how many Pods, the VPA for how big they are, and KEDA for scaling on events, down to zero. The next lesson in this section shows Goldilocks, which turns VPA recommendations into a dashboard for every workload in a namespace.