App-of-apps & ApplicationSets

Manage many apps/clusters.

An apartment building has one superintendent. When every unit needs a new smoke detector, nobody expects you to knock on forty doors and fit each one yourself. You hand the super a single list, and the whole building gets kitted out. Argo CD's app-of-apps pattern is that superintendent for your platform.

First, the piece it is built on. Argo CD (a tool that continuously makes a Kubernetes cluster match configuration stored in Git) manages workloads through an object it calls an Application (a Custom Resource, meaning a new object type Argo CD taught Kubernetes about, that says 'take this path in this Git repo and make the cluster look like it'). One Application, one deployable thing. That is fine for five. It falls apart at fifty, because now you are hand-editing fifty files and hoping none of them drift.

App-of-apps fixes the counting problem with one move. You write a single parent Application whose Git source is not a chart or a pile of plain manifests, but a directory full of other Application manifests. Sync the parent, and Argo CD lays down every child Application in that directory. Your whole platform (ingress, certificates, logging, metrics) becomes one thing you point at and turn on.

App-of-apps in one file

Here is a root Application. The only unusual part is what it points at: a folder of Applications, not workloads.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: platform-root
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://git.acme.internal/platform/apps.git
    path: applications        # a directory full of Application YAMLs
    targetRevision: v3.4.0    # a tag, not a moving branch
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Apply it once.

kubectl apply -n argocd -f platform-root.yaml

application.argoproj.io/platform-root created

The parent syncs, reads the folder, and creates a child Application for each file it finds. Ask Argo CD what the root is managing and you see the children as its resources. Not Pods and Services, but Applications.

argocd app get platform-root

Name:               argocd/platform-root
Project:            platform
Server:             https://kubernetes.default.svc
Namespace:          argocd
Source:
- Repo:             https://git.acme.internal/platform/apps.git
  Target:           v3.4.0
  Path:             applications
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to v3.4.0 (a1b2c3d)
Health Status:      Healthy
 
GROUP        KIND         NAMESPACE  NAME           STATUS  HEALTH   HOOK  MESSAGE
argoproj.io  Application  argocd     ingress-nginx  Synced  Healthy        application.argoproj.io/ingress-nginx created
argoproj.io  Application  argocd     cert-manager   Synced  Healthy        application.argoproj.io/cert-manager created
argoproj.io  Application  argocd     prometheus     Synced  Healthy        application.argoproj.io/prometheus created
argoproj.io  Application  argocd     loki           Synced  Healthy        application.argoproj.io/loki created

Two flags in that root do the heavy lifting. prune: true means that if you delete a file from the folder, Argo CD deletes the matching child Application (and if that child also prunes, its workloads go too). selfHeal: true means that if someone hand-edits a child live in the cluster, Argo CD stomps the manual change back to whatever Git says. Respect what that buys you: a delete in Git is a delete in production.

Where a plain app-of-apps stops helping

App-of-apps still expects one file per app. Run the same monitoring stack on twenty clusters and you copy that Application twenty times, changing one line (the target cluster) in each. Twenty near-identical files. Twenty chances to fat-finger a namespace (Kubernetes' way of carving one cluster into separate, walled-off areas). The pattern scaled the bootstrap, not the authoring.

ApplicationSets: mail-merge for Applications

Mail merge takes one letter template and a spreadsheet of names, and prints one personalized letter per row. An ApplicationSet is that for Argo CD. You write one Application template with blanks in it, and a generator (the spreadsheet) fills the blanks and stamps out one real Application per row.

The generator is where the fan-out comes from, and Argo CD ships several. The list generator makes one app per hand-written entry. The cluster generator makes one per cluster registered with Argo CD. The Git generator makes one per directory or per file in a repository, so a monorepo (a single repository holding many teams' code) becomes a fleet of apps. The matrix and merge generators combine two generators, for example 'every app in the repo, on every prod cluster.' There are also pull-request and SCM-provider (source control management, the system that hosts your Git repositories) generators that call your Git host's API (application programming interface, the machine endpoint a service exposes) to discover repositories and branches on their own.

Here is the twenty-cluster problem solved once. Note goTemplate: true, which switches the placeholder syntax to Go templates (so {{.name}} instead of the older {{name}}), and the selector, which narrows the cluster generator to clusters you have labelled env=prod.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: monitoring
  namespace: argocd
spec:
  goTemplate: true              # {{.name}} instead of {{name}}
  generators:
    - clusters:                 # one Application per registered cluster
        selector:
          matchLabels:
            env: prod           # only clusters labelled env=prod
  template:
    metadata:
      name: 'monitoring-{{.name}}'
    spec:
      project: monitoring
      source:
        repoURL: https://git.acme.internal/platform/monitoring.git
        path: chart
        targetRevision: v2.1.0  # pinned tag
      destination:
        server: '{{.server}}'
        namespace: monitoring
      syncPolicy:
        automated:
          selfHeal: true
          prune: true

Apply it the same way, then list what the controller produced. Every generated app carries the label argocd.argoproj.io/application-set-name, so you can filter on it.

kubectl apply -n argocd -f monitoring-appset.yaml
kubectl get applications -n argocd -l argocd.argoproj.io/application-set-name=monitoring

applicationset.argoproj.io/monitoring created
NAME                    SYNC STATUS   HEALTH STATUS
monitoring-ap-south-1   Synced        Healthy
monitoring-eu-west-1    Synced        Healthy
monitoring-us-east-1    Synced        Healthy

Three prod clusters, three Applications, from one definition. Register a fourth prod cluster tomorrow, label it env=prod, and a fourth app appears on its own with no edit from you. That is the whole appeal, and the whole danger.

The blast radius, and how to shrink it

Everything that makes this powerful is the same thing that makes it risky. The parent or the template is a lever with the entire fleet on the other end. Change one field in the template, and every generated child changes at once, on every cluster it targets. A good deploy goes everywhere in seconds. So does a broken one.

There is a sharper edge for defenders. A Git or SCM generator turns 'can push to this repo' into 'can create an Argo CD Application.' If those generated apps land in an AppProject (Argo CD's guardrail object that whitelists which repos, clusters, and namespaces a group of apps may touch) that is wide open, then anyone who can add a folder to the watched repo can deploy to anywhere the project allows. Scope the AppProject down to the exact destinations you mean, and treat write access to a generator's repo as production access. The upside for the blue team: every fan-out leaves a commit with an author behind it, so the Git history is your audit trail.

Lock down who can create ApplicationSets with Kubernetes RBAC (role-based access control, the rules for who may touch which objects). The controller builds those Applications with its own broad permissions, so anyone allowed to create an ApplicationSet effectively borrows that reach. Then roll changes out in waves instead of all at once. ApplicationSets support progressive syncs: a RollingSync strategy that updates a labelled slice of clusters (say, canary), waits for them to report Healthy, then moves on to prod. It is off by default and sits behind a feature flag. Set applicationsetcontroller.enable.progressive.syncs to "true" in the argocd-cmd-params-cm ConfigMap (a Kubernetes object that stores configuration as key-value pairs), or the strategy in your manifest does nothing at all.

Before you trust a new ApplicationSet in production, dry-run the fan-out. List the generated Applications (argocd appset get, or kubectl get applications filtered on the set's label) and confirm the target list is exactly the clusters and namespaces you meant to hit. Then check each generated app's targetRevision is a pinned tag, not a moving branch. If either one surprises you, your generator or your AppProject is wider than you think, and that gap is precisely where an attacker walks in.

Try this

Run kubectl apply -n argocd -f platform-root.yaml on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: one edit, every cluster. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related