GitOps Delivery with Argo CD: Sync, Drift & Self-Heal

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd rollout status deploy/argocd-server --timeout=300s

Get the initial password and port-forward the UI — the CLI works too, but the diff view is worth seeing at least once:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo
kubectl -n argocd port-forward svc/argocd-server 8080:443

Argo CD is a controller. It has no knowledge of your pipeline and no webhook into it; it watches a Git path and a cluster, and works to make the second look like the first.

Create the Application. This is the whole configuration — everything else is consequence:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ivolve-app
  namespace: argocd            # MUST be Argo CD's own namespace
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/<you>/CloudDevOpsProject.git
    targetRevision: HEAD
    path: 04-Kubernetes/manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: ivolve
  syncPolicy:
    automated:
      prune: true              # objects removed from Git are removed from the cluster
      selfHeal: true           # changes made to the cluster are reverted
      allowEmpty: false
    syncOptions:
      - CreateNamespace=true

Two details that cause silent failures:

• metadata.namespace must be argocd. An Application created anywhere else is ignored — no error, no event, it simply never syncs.
• The finalizer makes deletion cascade. Without it, kubectl delete application removes the record and orphans everything it created.

kubectl apply -f argocd-application.yaml

kubectl -n argocd get application ivolve-app

Wait for SYNC STATUS: Synced and HEALTH STATUS: Healthy. Those answer two different questions — does the cluster match Git, and are the workloads actually working — and you will see them disagree later.

Watch the first sync happen in order:

kubectl -n ivolve get pods -w

The database comes up first, then the two backends, then the frontend, then the ingress. That is not luck. It is argocd.argoproj.io/sync-wave on the manifests — 1 for the database, 2 for the backends, 3 for the frontend, 4 for the ingress. Without it everything applies at once and the services that need the database crash-loop until it happens to be ready.

This is the part that replaces every kubectl apply you have run so far.

In 04-Kubernetes/manifests/kustomization.yaml, find the images: block — this is exactly what stage 8 of the Jenkins pipeline rewrites:

images:
  - name: ivolve-frontend
    newName: <account>.dkr.ecr.us-east-1.amazonaws.com/ivolve-frontend
    newTag: 9-c1713b9          # <- change this to another tag you have pushed

Change newTag to a different tag that exists in ECR, then:

git commit -am "deploy frontend 10-abc1234"
git push

Now do nothing. Argo CD polls roughly every three minutes:

kubectl -n argocd get application ivolve-app -w

Synced → OutOfSync → Synced. You did not touch the cluster.

The manifest says what you asked for. The Pod says what happened:

kubectl -n ivolve get pods -l app.kubernetes.io/name=ivolve-frontend \
  -o jsonpath='{.items[*].spec.containers[*].image}'; echo

That must show the tag you committed. Checking the manifest instead would only prove you can read your own commit.

kubectl -n argocd get application ivolve-app \
  -o jsonpath='{.status.history[-1].revision}'; echo

The sync history records which commit produced the running state. This is the deployment record — there is no separate one, and it cannot drift from reality because it is the thing that caused reality.

Everything so far shows the happy path. The interesting behaviour is what happens when the cluster and Git disagree.

kubectl -n ivolve scale deployment ivolve-frontend --replicas=1
kubectl -n ivolve get deploy ivolve-frontend -w

Watch the replica count. It goes to 1 — and then comes back.

Git                          Cluster
replicas: 3                  replicas: 3
     |                            |
     |                     kubectl scale --replicas=1
     |                            v
replicas: 3        <-- drift -->  replicas: 1
     |
     |  Argo CD compares, finds a difference,
     |  and applies Git's version
     v
replicas: 3                  replicas: 3

That is selfHeal: true. Your change was not rejected — the API server accepted it happily. It was reverted, because a controller is continuously making the cluster match Git, and your edit was simply the next difference it found.

Look at what it recorded:

kubectl -n argocd get application ivolve-app -o jsonpath='{.status.operationState.phase}'; echo

This is the lesson of the whole phase. In a push-based world, that kubectl scale would have stayed — and six weeks later nobody would know why production had one replica while the repository said three. Here the repository is not documentation of the system. It is the system.

Self-heal has a cost, and pretending otherwise is how people get surprised at 3am:

• You cannot hotfix by hand. An emergency kubectl edit is reverted within minutes. The fix is a commit, and your ability to recover is now bounded by how fast you can merge.
• A bad commit deploys itself. Automation applies what Git says, including a mistake. The protection is branch review, not the cluster.
• prune: true deletes. Remove a manifest from Git and the object is removed from the cluster. That is the point, and it is also how a bad rebase deletes a database.

Rolling back is therefore git revert, and it goes through the same path as everything else:

git revert --no-edit HEAD
git push