Securing Helm in CI/CD & GitOps

Template-render, scan, and deploy safely.

Handing your CI pipeline (continuous integration, the automated system that builds, tests, and packages your code every time someone pushes a change) cluster-admin rights and letting it run helm install is like giving the courier who drops your packages a master key to every apartment in the building. It works. It feels convenient. And now every leaked token, every poisoned chart, every typo in a deploy script lands in production directly, with nothing standing in between.

Securing Helm changes that shape. You render the chart down to plain Kubernetes manifests, scan those manifests the way you would scan any other build output, and then let a GitOps controller (a small program that lives inside the cluster and constantly drags the running system back toward what is written in Git) be the only thing that ever holds a cluster credential. The pipeline builds and gates. The controller deploys. No build job ever needs to talk to the cluster's API server (the single front door every change to the cluster has to pass through).

Render The Chart To Plain Manifests First

helm template is like printing the finished assembly sheet from a blank form with every gap filled in, before you build a thing. It takes your chart, resolves the values, the functions, and the named templates, and prints the exact YAML (the plain-text format Kubernetes reads to know what to create) that would be applied. It runs offline, with zero cluster access. The result is a static file, and a static file is something a scanner can read top to bottom.

This is a different check from helm lint and helm test, which the linting and testing lesson covers. Those ask whether the chart is well-formed and whether the app actually comes up. Here you ask a security question about the rendered result. Does any container run privileged? Does anything mount a hostPath (a folder borrowed straight from the underlying node, a favorite escape route for a container trying to break out onto the host)? Are CPU and memory limits missing? Is an image pinned to the latest tag, which can change under you without a word? Run this on every pull request (a proposed change sitting in the queue for review) and fail the build when it finds something, so a risky manifest never gets promoted.

# Fill in every value and helper offline, no cluster contact
helm template myapp ./charts/myapp \
  --values values-prod.yaml \
  --namespace prod > rendered.yaml
 
grep -c '^kind:' rendered.yaml

9

Why render first instead of scanning the chart's template files directly? Because the templates are full of blanks. The same chart renders as a locked-down Deployment with one values file and a wide-open, privileged one with another. A single conditional can add or drop an entire securityContext block depending on one flag. Scanning the source tells you what the chart might do. Scanning the rendered output for the exact values file you actually ship tells you what it will do, and that is the thing that reaches the cluster.

Scan The Manifests Like Any Other Artifact

First question: does every object actually match the Kubernetes API schema? kubeconform checks each resource against the real schemas the API server uses. The -strict flag rejects unknown fields, so a typo like resurces: instead of resources: gets caught right here, instead of being silently dropped by the cluster and leaving your pods running with no limits at all.

kubeconform -strict -ignore-missing-schemas -summary rendered.yaml

Summary: 9 resources found in 1 file - Valid: 8, Invalid: 0, Errors: 0, Skipped: 1

That one skipped resource is a custom resource, an object of a type that some CustomResourceDefinition (a CRD, an extra object type bolted on top of core Kubernetes, like a Prometheus ServiceMonitor) added to the cluster. kubeconform ships no built-in schema for that type. The -ignore-missing-schemas flag is what makes it skip the resource instead of failing hard on it. When you want those validated too, point it at the matching schemas with -schema-location, otherwise a broken custom resource slides straight through the gate.

Second question: does the manifest follow safe defaults? kube-linter runs policy checks, things like containers running as root, writable root filesystems, and missing resource requests.

kube-linter lint rendered.yaml

KubeLinter 0.6.8
 
rendered.yaml: (object: prod/myapp apps/v1, Kind=Deployment) container "app"
    does not have a read-only root file system (check: no-read-only-root-fs,
    remediation: Set readOnlyRootFilesystem to true in the container
    securityContext.)
 
rendered.yaml: (object: prod/myapp apps/v1, Kind=Deployment) container "app"
    has cpu request 0 (check: unset-cpu-requirement, remediation: Set the CPU
    request for the container.)
 
Error: found 2 lint errors

That last line matters more than it looks. Error: found 2 lint errors means the process exits with a non-zero status, and a non-zero exit fails the pipeline stage. That is the gate doing its job on its own, with no human in the loop deciding whether to care.

Third question: trivy config scans the same file against a large built-in rule set and sorts what it finds by severity.

trivy config --severity HIGH,CRITICAL rendered.yaml

rendered.yaml (kubernetes)
 
Tests: 34 (SUCCESSES: 33, FAILURES: 1, EXCEPTIONS: 0)
Failures: 1 (HIGH: 1, CRITICAL: 0)
 
HIGH: Container 'app' of Deployment 'myapp' should set
'securityContext.runAsNonRoot' to true
════════════════════════════════════════════════════════════
Force the running image to run as a non-root user to ensure least privilege.
 
See https://avd.aquasec.com/misconfig/ksv012

One caveat on scope. trivy config reads the YAML, not the software packed inside the container. To find known vulnerabilities (CVEs, short for Common Vulnerabilities and Exposures, the public catalog of tracked security flaws) in the image itself, run trivy image myrepo/myapp:1.4.2 as a separate step. Wire both to fail on HIGH and CRITICAL with --exit-code 1, and the pull request cannot merge until someone fixes the finding or records an explicit, reviewed exception.

Hand The Deploy To A Controller, Not The Pipeline

GitOps (a way of running systems where Git holds the desired state and a controller keeps reality matching it) works like the thermostat on your wall. You write down the target temperature. The thermostat reads that number and keeps nudging the room toward it, over and over, forever. You never walk over and light the furnace by hand.

Argo CD (a widely used GitOps controller) is that thermostat. It runs inside the cluster, reads a small object called an Application that points at a Git repository and a path, renders the chart, and applies the result. The deploy credential lives with the controller, not in a CI secret that a thousand build jobs can read. Your pipeline's job ends the moment the reviewed change merges.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: team-a            # a scoped project, never 'default'
  source:
    repoURL: https://github.com/acme/charts.git
    path: charts/myapp
    targetRevision: v1.4.2   # a tag or digest, not a moving branch
    helm:
      valueFiles:
        - values-prod.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: prod
  syncPolicy:
    automated:
      prune: true            # delete objects removed from Git
      selfHeal: true         # revert manual drift back to Git

Two flags in syncPolicy carry most of the safety. prune deletes objects you removed from Git, so deleting a file actually removes the resource instead of leaving an orphan running forever. selfHeal watches for drift and reverts it. If someone runs kubectl edit to loosen a security setting at 2 a.m., Argo CD quietly puts it back to what Git says within minutes. Git becomes the single ledger of what is running, and every change to that ledger goes through a reviewed pull request. Pinning targetRevision to a tag or a commit digest instead of a branch like main means a force-push to that branch cannot sneak new code past your review.

Fence Each Project And Keep Secrets Out Of Git

An Application sitting on Argo CD's default project is a contractor badge that opens every door in the building. It can deploy any kind of object, into any namespace, on any cluster the controller knows about, up to and including a ClusterRoleBinding (a cluster-wide permission grant under RBAC, the Role-Based Access Control system that decides who is allowed to do what). A poisoned chart on the default project can quietly hand its own account cluster-admin, and you would not see it coming.

An AppProject is a badge cut for one floor. It lists the exact repositories an app may deploy from, the exact clusters and namespaces it may deploy to, and the exact resource kinds it may create. Scope it down and a compromised Application cannot reach another team's namespace or mint cluster-scoped permissions, because the controller flatly refuses to sync anything outside the fence.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-a
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/acme/charts.git   # only this repo
  destinations:
    - server: https://kubernetes.default.svc
      namespace: prod                       # only this namespace
  clusterResourceWhitelist: []              # no cluster-scoped objects
  namespaceResourceWhitelist:
    - group: '*'
      kind: '*'

The other easy mistake is dropping a real secret into values.yaml. Git keeps history forever, so a password committed once lives in every clone of the repo, and rotating it means rewriting history across everyone's machines. Treat a committed secret as burned. Keep only a reference in Git and store the real value in a secrets manager (a dedicated vault such as HashiCorp Vault or AWS Secrets Manager). Something like the External Secrets Operator pulls the value in at sync time, or SOPS (Secrets OPerationS, a tool that encrypts only the sensitive fields in a file so the file itself stays safe to commit) keeps it out of plaintext.

The defender's move is to catch the plaintext secret before it ever merges. A scanner like gitleaks running in the same CI stage reads the diff and blocks the pull request the moment it spots something shaped like a key.

gitleaks detect --source . --redact

    ○
    │╲
    │ ○
    ○ ░
    ░    gitleaks
 
Finding:     password: REDACTED
Secret:      REDACTED
RuleID:      generic-api-key
Entropy:     3.94
File:        charts/myapp/values-prod.yaml
Line:        14
 
1:34PM INF 128 commits scanned.
1:34PM INF scan completed in 214ms
1:34PM WRN leaks found: 1

leaks found: 1 exits non-zero, the stage fails, and the secret never reaches main. Rotate the exposed value anyway, because it sat in a commit long enough for someone to clone it.

Prove The Pipeline Cannot Reach The Cluster

The wiring is not done until you have checked that the credential really is gone from CI. Search the pipeline definition for a kubeconfig file or a service-account token and confirm there is none. The deploy stage should hold a git push or a merge and nothing that talks to the API server. Then confirm the controller, not a human and not a build job, owns what is running.

argocd app get myapp

Name:               argocd/myapp
Project:            team-a
Server:             https://kubernetes.default.svc
Namespace:          prod
URL:                https://argocd.example.com/applications/myapp
Source:
- Repo:             https://github.com/acme/charts.git
  Target:           v1.4.2
  Path:             charts/myapp
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune, SelfHeal)
Sync Status:        Synced to v1.4.2 (a3f9c21)
Health Status:      Healthy

Synced to v1.4.2 (a3f9c21) names the exact Git commit running in prod. If that hash is not in your history, or it does not match the tag you reviewed, you have found either drift or a deploy that skipped the gate, and you have a specific commit to go read.

Try this

Run grep -c '^kind:' rendered.yaml on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: argo CD renders with helm template, so lookup returns nil. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related