Helm & Kustomize sources

Render tools inside Argo.

A cook can do two separate jobs. One is turning a recipe into a finished plate of food. The other is carrying that plate out to the table. Helm and Kustomize normally do both. Helm (the package manager for Kubernetes, which fills in reusable templates called charts) renders your templates into real Kubernetes objects and then installs them. Kustomize (a tool that layers edits on top of plain YAML, the text format Kubernetes uses to describe objects, with no templating language) builds the final YAML and can apply it too. Argo CD (the GitOps controller that keeps a cluster matching what is committed in Git, where GitOps means treating a Git repository as the single source of truth for what runs) hires both tools for the prep job only. It asks them to render, takes the plain manifests they hand back, and does the comparing and applying itself. The render tools never touch your cluster.

That split is the whole design. There is no Tiller (the old privileged in-cluster server that Helm 2 used to push releases), no Helm release secret holding history, and no kubectl apply hidden inside a shell wrapper. Rendering and applying are two clean steps, owned by two different things. For a defender, that is the useful part. Because Argo always renders first and applies second, there is a moment where the finished YAML exists and can be read before a single object reaches the cluster.

Render here, apply there

The rendering runs inside one specific Argo component, the argocd-repo-server. That pod clones your Git repositories, holds your repository credentials, and runs helm template or kustomize build on the source you pointed it at. It is the prep kitchen: it has the pantry keys and the knives, but it does not carry food to the table. Two things follow. First, rendering is code execution. A Helm chart runs template logic, and a Kustomize build can call generators (small helpers that produce manifests during the build), so whoever controls the source controls what runs inside that pod. Second, you can reproduce the render yourself, line for line, because it is the same helm template and kustomize build you already have on your laptop.

# Render the chart the same way argocd-repo-server will, on your own machine
helm template podinfo charts/podinfo \
  --values charts/podinfo/values-prod.yaml \
  --set replicaCount=3 \
  --set image.tag=6.7.0

---
# Source: podinfo/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: podinfo
  labels:
    app.kubernetes.io/name: podinfo
    helm.sh/chart: podinfo-6.7.0
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 9898
      targetPort: http
  selector:
    app.kubernetes.io/name: podinfo
---
# Source: podinfo/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: podinfo
  template:
    metadata:
      labels:
        app.kubernetes.io/name: podinfo
    spec:
      containers:
        - name: podinfo
          image: ghcr.io/stefanprodan/podinfo:6.7.0
          ports:
            - name: http
              containerPort: 9898
          securityContext:
            runAsNonRoot: true
            runAsUser: 1000
            readOnlyRootFilesystem: true

Read that output the way you would read a security review, because that is what it is. The replica count came out as 3 and the image resolved to 6.7.0, which tells you your values and overrides landed. The container carries runAsNonRoot and a read-only root filesystem, which tells you the hardening from your values file survived the merge. If a pull request changed the chart and this render suddenly grew a privileged container or a hostPath mount (a volume that exposes a directory from the host node straight into the pod, a classic way to break out of a container), the diff of this plain text is where you catch it, before Argo applies anything.

Helm as a render engine

You point an Argo Application (the custom Kubernetes object that describes one thing Argo deploys) at a Helm source in one of two ways. Either path names a chart directory committed inside your Git repo, or chart names a chart to pull from a Helm repository at a pinned version. You feed in values three ways, and the order matters. valueFiles lists value files by path, relative to the chart. values is an inline block written straight into the Application. parameters are typed key and value pairs, the same as passing --set on the command line. Argo layers them in that order, so an inline value overrides a value file, and a parameter overrides both. releaseName sets what the chart sees as .Release.Name while it templates.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: podinfo
  namespace: argocd
spec:
  project: default
  destination:
    server: https://kubernetes.default.svc
    namespace: podinfo
  source:
    repoURL: https://github.com/org/gitops.git
    targetRevision: main
    path: charts/podinfo      # a chart dir committed to Git
    helm:
      releaseName: podinfo    # sets .Release.Name during templating
      valueFiles:
        - values-prod.yaml    # relative to path/
      values: |               # inline block; overrides valueFiles
        replicaCount: 3
      parameters:
        - name: image.tag     # equivalent to --set image.tag=6.7.0
          value: "6.7.0"

That precedence has teeth. Say your shared values file sets runAsNonRoot to true as a house rule. If someone adds an inline value or a parameter that flips it back to false, the file still reads as hardened while the running object is not. The file is not the source of truth. The render is. And because Argo keeps no Helm release history to consult, the only authoritative answer to what is deployed is the rendered manifest, which Argo prints for you on demand.

# Ask Argo for the exact manifests it renders for this Application
argocd app manifests podinfo | grep -E 'replicas:|image:|runAsNonRoot:'

  replicas: 3
          image: ghcr.io/stefanprodan/podinfo:6.7.0
            runAsNonRoot: true

Kustomize overlays and image pins

Kustomize works differently from Helm. There is no templating language. You keep a shared base of plain manifests and lay environment-specific edits over it, the way you drop a sheet of tracing paper over a drawing and change a few lines. Argo runs kustomize build against the path you name, usually the overlay for one environment. Kustomize also has built-in transformers that rewrite the result, and you can declare them in two places: in the overlay's kustomization.yaml, or in the Application's kustomize block, where Argo layers them on top. images rewrites image references (the same field the image-bump automation in ag-secrets edits when it pins a new tag). namePrefix and nameSuffix bolt strings onto every resource name. commonLabels stamps one label onto everything. replicas overrides a replica count. Whichever place you put them, the output is the same rendered YAML.

# --- overlays/prod/kustomization.yaml ---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
namePrefix: prod-
commonLabels:
  env: prod
images:
  - name: myapp
    newName: registry.example.com/myapp
    newTag: 1.4.2
patches:
  - path: replicas.yaml    # strategic-merge patch over the base
 
# --- or the same transformers, driven from the Application ---
spec:
  source:
    repoURL: https://github.com/org/gitops.git
    targetRevision: main
    path: overlays/prod
    kustomize:
      namePrefix: prod-
      commonLabels:
        env: prod
      images:
        - myapp=registry.example.com/myapp:1.4.2

kustomize build overlays/prod

apiVersion: v1
kind: Service
metadata:
  labels:
    env: prod
  name: prod-myapp
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: myapp
    env: prod
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    env: prod
  name: prod-myapp
spec:
  replicas: 4
  selector:
    matchLabels:
      app: myapp
      env: prod
  template:
    metadata:
      labels:
        app: myapp
        env: prod
    spec:
      containers:
        - image: registry.example.com/myapp:1.4.2
          name: myapp

Two things in that output earn a second look. The image pin landed as registry.example.com/myapp:1.4.2, so your supply-chain control is visible in the finished YAML. And commonLabels did more than add env: prod to metadata. It also wrote that label into the Deployment's selector and into the pod template. Selectors are immutable on a running Deployment, so adding or changing a common label later forces Argo to replace the object instead of patching it. Plan label changes with that in mind.

One overlay feature needs a deliberate decision before it works at all. Kustomize can inflate a Helm chart itself, through a helmCharts field in the kustomization. Argo refuses to run that by default. You switch it on by setting kustomize.buildOptions to --enable-helm in the argocd-cm ConfigMap (a ConfigMap is a Kubernetes object that holds configuration as key and value pairs, and argocd-cm is Argo's main one).

# Kustomize's helmCharts field stays off until you enable it globally
kubectl -n argocd patch configmap argocd-cm --type merge \
  -p '{"data":{"kustomize.buildOptions":"--enable-helm"}}'
kubectl -n argocd rollout restart deploy/argocd-repo-server

configmap/argocd-cm patched
deployment.apps/argocd-repo-server restarted

Values in a second repo

A common shape is a chart you do not own, from a public Helm repository, paired with values that you do own and keep under review in your own repo. You would rather not copy the whole upstream chart into Git to get there. The multi-source Application handles it. spec.sources, note the plural, lets one Application read from several repositories at once. You tag the repo that holds your values with ref: values, then point the chart source at it with the $values placeholder inside a value file path. Argo checks out both, resolves the path, and renders them as one unit. The upstream chart stays untouched at a pinned version, and your environment config stays in Git where a reviewer can see every change.

spec:
  sources:
    - repoURL: https://stefanprodan.github.io/podinfo
      chart: podinfo          # pulled from a Helm repo
      targetRevision: 6.7.0
      helm:
        valueFiles:
          - $values/envs/prod/values.yaml   # from the ref below
    - repoURL: https://github.com/org/gitops.git
      targetRevision: main
      ref: values             # exposes this repo as $values

When Helm's lifecycle quietly disappears

Here is the part that trips up people who know Helm well. Real Helm keeps a history of releases, so it knows whether a given apply is a fresh install or an upgrade of something already there. helm template does not. It renders once, with no memory. Under Argo, that freezes the pseudo-values a chart reads to tell the two apart. .Release.IsInstall is always true, .Release.IsUpgrade is always false, and .Release.Revision is stuck at 1, on every sync. Hook ordering does survive, because Argo maps Helm's hook annotations onto its own phases. helm.sh/hook pre-install and pre-upgrade become the PreSync phase (the step Argo runs before the main apply). post-install and post-upgrade become PostSync (the step after). helm.sh/hook-weight becomes sync-wave order (sync waves are how Argo applies resources in ordered batches). The ordering carries over. The install-versus-upgrade meaning does not.

So watch for run-once chart logic, because it no longer runs once. Any branch gated on first install renders on every Argo sync, since .Release.IsInstall is permanently true. A one-time migration Job, a PVC (PersistentVolumeClaim, a request for a chunk of storage) created only on install, a seed step, all of these keep reappearing and can fire again. That is how a run-once database migration ends up running during a routine config change. Argo has no Helm release history to notice it already ran. When you need real run-once behaviour, express it with Argo sync waves and hooks (covered in ag-waves), not with Helm's lifecycle values.

So keep one habit. Never trust the source file on its own. Run helm template or kustomize build locally, or ask argocd app manifests for the effective render, and read the YAML that will actually reach the cluster. That rendered text is where an override, a dropped security context, or a smuggled hostPath shows itself, and it costs you seconds to check on every change.

Try this

Run argocd app manifests podinfo | grep -E 'replicas:|image:|runAsNonRoot:' on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: --enable-helm turns your repo-server into a chart downloader. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related