Lesson 60 of 66 · 18 min read · Updated
GitOps with Argo CD
On this page
In the previous two articles you packaged the Catalog API with Helm and with Kustomize. In both cases a person, or a CI job, still ran helm upgrade or kubectl apply -k against the cluster. That has three weaknesses. The CI system needs cluster credentials. Nobody notices when someone runs kubectl edit in production at midnight. And the question "what is running right now?" has no reliable answer, because the cluster may have drifted from anything written down.
GitOps turns this around. Git becomes the single source of truth for what should run, and an agent inside the cluster continuously compares the cluster to Git and fixes any difference. Nobody pushes to the cluster; the cluster pulls from Git. Argo CD is the most widely used GitOps agent for Kubernetes.
In this article we install Argo CD, point it at a Git repository containing the Kustomize overlays and the Helm chart from the previous articles, and watch it deploy, detect drift, heal it, prune deleted resources, and generate one Application per environment automatically. To keep everything on your machine, the Git repository is served by a small Gitea instance inside the cluster. With GitHub or Azure Repos, only the repository URL changes.
We will cover the following topics:
- The GitOps model and how Argo CD implements it
- Installing Argo CD and the
argocdCLI - The Application resource: source, destination and sync policy
- Automated sync,
pruneandselfHeal, and what drift looks like - Deploying Kustomize overlays and Helm charts from Git
- How Argo CD runs Helm hooks
- ApplicationSets for one Application per environment, and the app-of-apps pattern
- Rolling back the GitOps way
Technical Requirements
- A Kubernetes cluster and
kubectl. The examples run on Docker Desktop's built-in Kubernetes with the kind provisioner (1 control-plane node and 3 workers, Kubernetes v1.36). A plainkindcluster with the same layout behaves the same. - Argo CD v3.5.6 and the matching
argocdCLI. git, and thecatalog-api:1.0.0andcatalog-api:1.1.0images from the Helm article. On a plain kind cluster, load them withkind load docker-image.- The Kustomize folder from the previous article and the Helm chart from the one before it.
- About 1 GB of free memory in the cluster for Argo CD and Gitea.
Core Concepts
Push versus Pull Deployments
Push (CI runs kubectl/helm) | Pull (GitOps agent) | |
|---|---|---|
| Who changes the cluster | The CI pipeline | An agent running inside the cluster |
| Cluster credentials | Stored in CI | Never leave the cluster |
| Drift (manual changes) | Unnoticed until the next deployment | Detected within minutes, optionally reverted |
| Source of truth | Whatever was applied last | The Git repository |
| Rollback | Re-run an old pipeline | git revert |
| Audit trail | Pipeline logs | Git history: who, what, when, reviewed by whom |
Your CI pipeline still builds and tests the .NET application and pushes the image. What changes is the last step: instead of deploying, CI (or a person) commits the new image tag to the deployment repository, and Argo CD does the rest.
Argo CD's Components
| Component | Job |
|---|---|
argocd-server | API and web UI, used by the CLI and the browser |
argocd-repo-server | Clones Git repositories and renders manifests: plain YAML, Kustomize, Helm |
argocd-application-controller | Compares rendered manifests with live objects, reports status, runs syncs |
argocd-applicationset-controller | Generates Applications from templates |
argocd-redis | Cache |
argocd-dex-server | Optional single sign-on |
The repo-server renders Helm charts with helm template. Argo CD does not run helm install, so helm list shows nothing for charts that Argo CD manages. The release history lives in Git and in Argo CD, not in Helm Secrets.
The Application Resource
An Application is a custom resource in the argocd namespace that connects one source to one destination.
| Field | Meaning |
|---|---|
spec.source.repoURL | Git repository (or Helm/OCI repository) |
spec.source.targetRevision | Branch, tag or commit to follow, for example main |
spec.source.path | Folder inside the repository. Argo CD detects Kustomize or Helm automatically. |
spec.destination | Which cluster and namespace to deploy to |
spec.project | An AppProject that limits which repositories, clusters and namespaces are allowed. default allows everything. |
spec.syncPolicy | Manual or automated sync, plus options |
Two statuses describe every Application and every resource in it.
| Status | Values | Question it answers |
|---|---|---|
| Sync status | Synced, OutOfSync | Does the live object match Git? |
| Health status | Healthy, Progressing, Degraded, Missing, Suspended | Is the object actually working? A Deployment is Healthy when its rollout completes. |
Sync Policies
| Setting | Effect |
|---|---|
No automated block | Argo CD only reports OutOfSync. A person clicks Sync or runs argocd app sync. |
automated: {} | A new commit in Git is applied automatically. |
automated.prune: true | Resources deleted from Git are deleted from the cluster. Off by default, as a safety measure. |
automated.selfHeal: true | Changes made directly in the cluster are reverted to match Git. |
syncOptions: [CreateNamespace=true] | Creates the destination namespace if needed. It is not deleted with the Application. |
Argo CD checks Git for new commits every 3 minutes by default (120 seconds plus up to 60 seconds of jitter). A webhook from GitHub or GitLab to argocd-server makes it near-instant. Drift in the cluster is noticed much faster, because the controller watches live objects.
ApplicationSets and App of Apps
Writing one Application per environment, per service, by hand does not scale. Two patterns help:
- ApplicationSet: a template plus a generator. The Git directory generator, for example, creates one Application for each folder that matches
catalog/overlays/*. Add a folder, get an Application. Delete the folder, the Application and its resources go away. Other generators read a list, the registered clusters, pull requests, or combine several. - App of apps: one parent Application whose source folder contains other Application manifests. Syncing the parent creates the children. It is the simplest way to bootstrap a whole cluster from one repository.
Hands-On: Kubernetes Commands
List Applications with their status
Show one Application, resource by resource
Ask Argo CD to check Git right now
Instead of waiting for the next poll.
Show the difference between Git and the cluster
Exits with code 1 when there is a difference.
Sync manually
Show the sync history
Applications are Kubernetes objects too
Step-by-Step Example
The Scenario
A deployment repository, catalog-deploy, holds everything needed to run the Catalog API: the Kustomize base, the dev and prod overlays, the PDB component, and the Helm chart. We install Argo CD, register the repository, and let Argo CD own the production namespace. Then we try to break it.
Step 1 — Install Argo CD
Pin the version instead of using the moving stable tag, so you know what you installed. The --server-side flag is required, because some Argo CD CRDs exceed the 262 KB annotation limit of client-side apply.
Download the CLI for your platform from the release page, for example on an Apple Silicon Mac:
On macOS brew install argocd works too, and on Windows download argocd-windows-amd64.exe.
Step 2 — Log In
Expose the API server and UI on your machine. Leave this running in its own terminal:
Get the generated admin password and log in. --insecure is needed because the local server uses a self-signed certificate:
Open https://localhost:8080 in a browser with the same credentials to see the UI. After you set a real password with argocd account update-password, delete the argocd-initial-admin-secret Secret.
Step 3 — A Git Server Inside the Cluster
If you already have a GitHub repository, skip this step and push the files there. To stay fully local, run Gitea in its own namespace. Its data lives in an emptyDir, which is fine for a lab and wrong for anything else.
Create a user and an empty public repository through Gitea's API, using a second port-forward:
Step 4 — Push the Deployment Repository
Lay out the repository with the Kustomize folder from the previous article under catalog/, and the Helm chart under charts/:
Check that the overlays set namespace: catalog-dev and namespace: catalog-prod, then commit and push:
Argo CD runs inside the cluster, so it uses the Service's cluster DNS name, not localhost. Register the repository to confirm Argo CD can reach it. Public repositories work without this step, but it gives you a quick connectivity check, and private repositories need it to store credentials.
Step 5 — The Production Application
Write the Application as YAML and keep it in Git too. Creating it with argocd app create works, but then the definition only lives in the cluster.
The finalizer makes deleting the Application also delete the resources it deployed. Without it, deleting the Application leaves everything running. Kubernetes 1.36 prints a warning that it prefers finalizer names containing a /; the name above is the one Argo CD documents, and the warning is harmless.
Argo CD rendered the Kustomize overlay itself, including the hashed ConfigMap name. Nobody ran kubectl apply.
Step 6 — Drift and Self-Healing
Someone "temporarily" scales production down by hand:
Watch the generation, the desired replicas and the ready replicas:
Within about two seconds the Deployment was back at three replicas. The events on the Application tell you what happened:
Now turn self-healing off to see drift without the automatic fix. This edits the live Application; in a real setup you would change the YAML in Git.
The Application is OutOfSync but still Healthy: one replica is running fine, it just is not what Git says. Sync by hand, then switch self-healing back on:
Without self-healing, drift is reported. With it, drift is reverted. Many teams start production with reporting only, and turn self-healing on once they trust the setup.
Step 7 — Deploy by Committing
Change the production greeting, and decide that production no longer needs the PodDisruptionBudget component. Both are edits to catalog/overlays/prod/kustomization.yaml:
and the components: block is removed. Commit and push:
Instead of waiting up to three minutes for the next poll, ask for a refresh:
A few seconds later the automated sync has run:
Read the first two lines carefully. Because prune: true is set, Argo CD deleted the PodDisruptionBudget that is no longer in Git, and the old hashed ConfigMap that kubectl apply -k left behind in the previous article. The leftover-ConfigMap problem is solved without ApplySets.
Every deployment maps to a commit, and the commit tells you who changed what and why.
Step 8 — A Helm Chart as the Source
Point an Application at the chart folder. Argo CD recognises Chart.yaml and renders it with Helm. valueFiles paths are relative to the chart folder and may point elsewhere in the same repository; valuesObject adds inline values on top. Here we turn the migration hook on for dev, to see how Argo CD handles it.
The Helm pre-install,pre-upgrade hook became an Argo CD PreSync hook: it runs before every sync, and the Deployment is only applied after it succeeds. And as promised, Helm itself knows nothing about this release:
Argo CD also has its own hook annotations, such as argocd.argoproj.io/hook: PreSync, and sync waves (argocd.argoproj.io/sync-wave: "-1") that order resources within a sync. Use them for plain YAML and Kustomize sources, where Helm hooks are not available.
Delete both Applications before the next step. Thanks to the finalizer, their resources go with them:
The namespaces themselves remain. CreateNamespace=true creates a namespace but never deletes it.
Step 9 — One Application per Environment with an ApplicationSet
Instead of one hand-written Application per overlay, let the Git directory generator find the overlays. {{.path.basename}} is the folder name, dev or prod.
Now add an environment the GitOps way. Copy the prod overlay to staging, change its namespace to catalog-staging, its label to staging and its greeting, and push:
The ApplicationSet controller also polls Git every three minutes. In our test the new Application appeared three minutes after the push, created its namespace, and was Synced and Healthy:
Nobody wrote an Application for staging. The folder is the declaration.
Step 10 — Roll Back with git revert
In GitOps, a rollback is a commit. Undo the staging change:
At the next poll the generator no longer finds the staging folder, so the ApplicationSet deletes the catalog-staging Application. Generated Applications carry the resources finalizer, so the Deployment, Service, ConfigMap and Secret go with it:
The same works for a bad release: git revert the commit that bumped the image tag, and Argo CD deploys the previous tag. Argo CD also has argocd app rollback, but it only works when automated sync is off, because otherwise the next sync would immediately re-apply what Git says.
Step 11 — Clean Up
Delete the ApplicationSet; its Applications and their resources follow:
Uninstall Argo CD with the same manifest you installed it with, which also removes its CRDs and cluster roles, then delete the namespaces:
Summary
- GitOps makes Git the source of truth. An agent in the cluster pulls changes and keeps the cluster matching Git; CI no longer needs cluster credentials.
- Install Argo CD with a pinned version and
kubectl apply --server-side. Argo CD v3.5.6 runs an API server, a repo-server that renders manifests, an application controller, and an ApplicationSet controller. - An Application connects a repository path to a destination namespace. Argo CD detects plain YAML, Kustomize and Helm automatically, and renders Helm with
helm template, sohelm listshows nothing. - Sync status says whether the cluster matches Git; health status says whether it works.
automateddeploys new commits,prunedeletes what was removed from Git (including stale hashed ConfigMaps), andselfHealreverts manual changes within seconds.- Git is polled every 3 minutes by default; use
--refreshor a webhook for faster feedback. - Helm hooks become Argo CD PreSync hooks; sync waves order resources for non-Helm sources.
- An ApplicationSet with a Git directory generator creates one Application per folder. Adding or deleting a folder adds or removes an environment.
- Roll back with git revert. The resources finalizer makes deleting an Application delete its resources, but namespaces created with
CreateNamespace=truestay.
That completes the Packaging and GitOps section: Helm to package your services, Kustomize to describe your environments, and Argo CD to make Git the only way anything reaches the cluster.