Kubernetes Jobs and CronJobs

So far you have run long-lived workloads: web APIs, databases and background services that should stay up until you replace them. Deployments, ReplicaSets and DaemonSets manage these and keep their Pods running forever.

Many tasks are different. Applying a database migration, generating a monthly report, compressing old logs or sending a batch of emails should run once, finish, and stop. If you ran them in a normal Pod, Kubernetes would restart them every time they exit, and your migration or report would run in an endless loop.

A Job is the Kubernetes object for this kind of work. Think of it as a to-do item: Kubernetes keeps working on it until it is ticked off, then leaves it alone. A CronJob is the recurring reminder in your calendar: it creates a new Job on a schedule, like the Unix cron daemon.

What you will learn

1. Why run-to-completion work needs a Job instead of a Deployment
2. The three Job patterns (one shot, parallel fixed completions and work queue) and how completions and parallelism control them
3. How restart policies, backoffLimit and deadlines handle failures
4. How to schedule recurring work with a CronJob and keep old Jobs from piling up
5. How to run, watch, debug and clean up each pattern with kubectl

Jobs versus regular Pods

Pods managed by a Deployment use restartPolicy: Always. The kubelet restarts the container every time it exits, whatever the exit code. That is perfect for a web server, and wrong for a migration.

How a Job works

A Job is a controller. It creates Pods from its spec.template, counts how many finish successfully, and keeps going until it reaches the number you asked for.

1. You submit a Job manifest to the API server.
2. The Job controller reads the Pod template and creates one or more Pods.
3. Each Pod is scheduled onto a node and runs its containers.
4. When a Pod exits with code 0, the controller records a completion. When it fails, the container is restarted or the Pod is replaced, depending on restartPolicy.
5. When the successful completions reach spec.completions, the Job is marked Complete.
6. The finished Job and its Pods stay in the cluster so you can read their logs. They are not deleted unless you ask for it.

The controller also labels every Pod it creates with batch.kubernetes.io/job-name and batch.kubernetes.io/controller-uid (plus the older short forms job-name and controller-uid). The unique ID keeps a Job's Pods separate from any other Pods, and the name label lets you list them with -l job-name=<name>.

Job patterns

Two fields shape a Job: completions (how much work: the number of Pods that must succeed) and parallelism (how many workers: the number of Pods that may run at once). Together they give three patterns:

Picture a restaurant kitchen. A one-shot Job is one chef cooking one special dish; when it is done, the kitchen closes. A fixed completions Job is four chefs who must turn out eight dishes between them. A work queue Job is four chefs sharing one stack of orders; they keep cooking until the stack is empty.

completions and parallelism in detail

1. spec.completions: the number of Pods that must exit successfully. Defaults to 1. With completions: 10 the Job keeps creating Pods until 10 have exited with code 0.
2. spec.parallelism: the maximum number of Pods running at the same time. Defaults to 1. With parallelism: 5 and completions: 10, up to 5 Pods run at once, and a new one starts each time one finishes, until 10 have succeeded.

A worked example: you need to generate 800 invoices, and each Pod generates 100. That is completions: 8. Your cluster has room for 4 Pods at once, so parallelism: 4. Kubernetes starts 4 Pods, replaces each one as it finishes, and stops when 8 have succeeded.

Restart policies and failure handling

A Job's Pod template must use restartPolicy: OnFailure or restartPolicy: Never. Always is rejected, because it would contradict run-to-completion.

Retries are not immediate. With OnFailure, the kubelet waits 10 s, 20 s, 40 s and so on (capped at five minutes) between container restarts; the Pod shows CrashLoopBackOff while it waits. With Never, the Job controller applies a similar growing delay, capped at six minutes, before it creates each new Pod. Either way, a crashing task cannot hammer your nodes in a tight loop.

Three fields put limits on a Job:

1. spec.backoffLimit (default 6): how many retries are allowed before the Job is marked Failed. A migration that either works or does not might use 2 to 4; a task that calls a flaky network service might use more.
2. spec.activeDeadlineSeconds: a hard time limit for the whole Job. When it passes, all running Pods are stopped and the Job fails, whatever backoffLimit says.
3. Liveness probes: they work in Job Pods just as in Deployment Pods. If a worker hangs (a deadlock, or a loop that makes no progress), the probe fails and the container is restarted.

CronJobs: scheduling recurring Jobs

A CronJob creates a new Job on a schedule. The schedule field uses standard five-field cron syntax:

Common schedules:

By default the schedule follows the time zone of the cluster's controller manager, which is often UTC. Set spec.timeZone (for example "Europe/Istanbul" or "Etc/UTC") to make it explicit.

Two fields control how many old Jobs a CronJob keeps:

1. successfulJobsHistoryLimit (default 3): how many successful Jobs to keep. Older ones are deleted automatically.
2. failedJobsHistoryLimit (default 1): how many failed Jobs to keep for debugging.

concurrencyPolicy decides what happens when a new run is due but the previous one is still running:

1. Allow (default): the runs overlap.
2. Forbid: the new run is skipped.
3. Replace: the running Job is cancelled and a new one starts.

Finally, suspend: true pauses a CronJob: it stops creating new Jobs until you set it back to false. This is handy during maintenance windows.

Cleaning up finished Jobs

Finished Jobs and their Pods stay in the cluster. That is useful for reading logs, but over months it can leave thousands of old objects behind. You have three tools:

1. TTL after finished: set spec.ttlSecondsAfterFinished on the Job. With ttlSecondsAfterFinished: 3600, the Job and its Pods are deleted one hour after it completes or fails. This is the recommended way for standalone Jobs.
2. History limits: for CronJobs, successfulJobsHistoryLimit and failedJobsHistoryLimit do the cleanup for you.
3. Manual cleanup: kubectl delete job <name> removes a Job and all its Pods.

Command reference

You will use these commands throughout the hands-on section:

Hands-on: run every Job pattern

You will run a one-shot database migration, a parallel invoice batch, a work queue backed by Redis, and a scheduled cleanup CronJob. Every object carries the label chapter: jobs, so you can remove everything with one command at the end.

Step 1 — Run a one-shot migration Job

The simplest pattern: one Pod, one task, run until it succeeds. This Job simulates Entity Framework Core migrations for a .NET ShippingApi. Save it as schema-migrate-job.yaml:

Apply it:

Key points in this manifest:

1. restartPolicy: OnFailure restarts the container inside the same Pod, on the same node, if it exits with a non-zero code.
2. backoffLimit: 4 allows up to 4 retries before the Job is marked Failed.
3. Neither completions nor parallelism is set, so both default to 1: the classic one-shot pattern.

The SDK image is large, so the first run may spend a minute pulling it.

Step 2 — Inspect the Job and its Pod

List Jobs to see the completion count:

Expected output once it has finished:

For more detail, describe it:

The output includes lines like these:

Read the migration output from the Pod's logs:

Expected output:

The Pod ran once and stopped, and Kubernetes will not restart it. It stays in the Completed state:

Expected output:

Step 3 — See how failures are handled

To try this yourself, change exit 0 to exit 1 in the manifest. A Job's Pod template cannot be changed after creation, so delete the Job and apply it again:

With restartPolicy: OnFailure, the same Pod is restarted. The RESTARTS count grows and the status shows CrashLoopBackOff between attempts:

This keeps the cluster tidy: one Pod, retrying in place. When the retry count reaches the backoffLimit of 4, the Job is marked Failed and the Pod is terminated.

Now change restartPolicy to Never, delete and apply again. The kubelet no longer restarts the container; each failure marks the Pod as failed and the Job controller creates a new one:

The extra Pods are clutter, but each one keeps the logs of its own attempt (kubectl logs schema-migrate-abc12). When you are done, set the manifest back to exit 0 and OnFailure, then delete and re-apply it so the Job completes.

Step 4 — Run a parallel batch Job

Now scale up. The billing system needs 8 invoice batches, with at most 4 running at once: the parallel fixed completions pattern. Save this as invoice-batch-job.yaml:

Apply it:

The Job immediately starts 4 Pods, the parallelism limit. Each time one completes, a new one starts, until 8 have succeeded.

Step 5 — Watch the parallel Job progress

Use --watch to see Pods come and go in real time (press Ctrl+C to stop):

You will see something like:

No more than 4 Pods run at any moment. As one completes, the next is scheduled, and once 8 have succeeded no new Pods are created. Check the final status:

The relevant lines:

Clean up before the next step:

Step 6 — Start a Redis work queue

In the work queue pattern, Pods do not get a fixed share of work. They all pull items from one shared queue until it is empty. You will use Redis as the queue.

Run Redis under a ReplicaSet rather than as a bare Pod, so it is recreated if its node fails. Save this as task-queue-rs.yaml:

Then give it a Service so workers can reach it by the DNS name task-queue. Save this as task-queue-service.yaml:

Apply both and check that Redis is running:

Expected output:

Step 7 — Fill the queue

Push ten work items onto a Redis list called work-items. kubectl exec rs/task-queue runs the command in the ReplicaSet's Pod, so you do not need its generated name:

Expected output (the new length of the list):

Step 8 — Run the queue workers

This Job starts 5 workers in parallel. Each one pops items from the list with LPOP and stops when the list is empty. Save it as queue-processor-job.yaml:

The worker reuses the redis:7-alpine image only because it already contains redis-cli. A real .NET worker would use a Redis client library in the same loop.

Apply it and watch the workers:

You will see something like:

Read the logs of one worker. The items it got depend on timing, so yours will differ:

Example output:

Confirm the queue is empty:

Expected output:

Step 9 — Schedule a CronJob

Finally, a cleanup task that runs every day at 02:00 UTC, for example to purge old reports or compact database indexes. Save it as report-cleanup-cronjob.yaml:

A few things to notice:

1. There are two templates. jobTemplate describes each Job the CronJob creates, and the template inside it describes that Job's Pods.
2. concurrencyPolicy: Forbid skips a run if yesterday's cleanup is somehow still going.
3. successfulJobsHistoryLimit: 3 keeps the last three successful runs; failedJobsHistoryLimit: 1 keeps the latest failed run for debugging.
4. The chapter: jobs label on jobTemplate.metadata is copied to every Job it creates, so the final cleanup finds them too.

Apply it and check it:

Expected output:

Step 10 — Trigger, pause and resume the CronJob

You do not want to wait until 02:00 to test it. Create a one-off Job from the CronJob's template:

Read its logs once it has run:

Expected output (with the current date):

Pause the schedule, for example during a maintenance window, and resume it afterwards:

While it is suspended, kubectl get cronjobs shows True in the SUSPEND column.

Step 11 — Clean up

Every object in this walkthrough carries chapter: jobs, including the manually triggered Job, so one command removes them all:

Common mistakes

1. Using restartPolicy: Always. The API server rejects the Job. Use OnFailure or Never.
2. Editing a Job and re-applying it. A Job's Pod template is immutable, so kubectl apply fails. Delete the Job and create it again.
3. A task that is not safe to run twice. A Pod can be retried after a node failure or a timeout, so the same work may run more than once. Make migrations and batch steps idempotent.
4. A work queue worker that exits 0 too early. The first successful exit stops new Pods from starting. Only exit 0 when the queue is really empty; exit non-zero on errors.
5. Old Jobs piling up. Set ttlSecondsAfterFinished on standalone Jobs and sensible history limits on CronJobs.
6. A CronJob that fires at the wrong hour. Without timeZone the schedule uses the controller manager's time zone. Set it explicitly.

Summary

A Job runs Pods until a task has finished successfully, then stops; a CronJob creates such Jobs on a schedule. Two fields, completions and parallelism, decide which pattern a Job follows.

1. One shot (both 1, the defaults): one-time tasks such as database migrations.
2. Parallel fixed completions: several Pods at once until a set number have succeeded, for example invoice batches.
3. Work queue (completions unset, parallelism above 1): workers drain a shared queue, and the Job completes once a worker has exited successfully and the rest have finished.
4. Use restartPolicy: OnFailure for most Jobs and Never while debugging; backoffLimit caps retries and activeDeadlineSeconds caps run time.
5. CronJobs take a five-field cron schedule, an optional timeZone, a concurrencyPolicy and history limits; suspend pauses them.
6. Clean up with ttlSecondsAfterFinished, CronJob history limits, or kubectl delete job.

With Jobs and CronJobs you can run migrations, batch work and scheduled maintenance with the same tools you already use for long-running services.