Remote bases & supply chain

Pinning and trusting external bases.

You order a sauce from an outside supplier and pour it, unchanged, into every dish that leaves your kitchen. It saves you the chopping and the simmering, and you inherit the supplier's improvements for free. You have also given a stranger a vote in what your customers eat, and nothing stops them from quietly changing the recipe between one delivery and the next. A remote base in Kustomize is that sauce.

Kustomize (the Kubernetes tool that customizes plain configuration files without templating them) normally reads manifests from local paths. It will also take a Git URL (a web address for a Git repository) as a resources: entry. At build time it fetches those files over the network and folds them into your output. What comes back becomes real objects in your cluster: Deployments, RBAC (role-based access control, Kubernetes' permission system) rules, and CRDs (custom resource definitions, which teach Kubernetes brand-new object types). Whoever controls that URL controls a slice of what you run.

The reuse is genuine. A platform team publishes one hardened base, and every app team builds on it instead of copy-pasting YAML that drifts out of sync over the months. You pick up their fixes automatically, on your schedule. That same wire is a supply-chain edge: manifests you did not write, from a repository you do not control, run with the permissions your pipeline hands them. Treat a remote base the way you treat any third-party dependency. Know its version, know what it does, and know who can change it.

How a remote base is referenced

The URL packs three things into one string. First the repository. Then a // separator that marks the path inside that repository. Then a ?ref= query that pins the version. Kustomize understands shorthand for the common hosts (github.com, gitlab.com, bitbucket.org) and an explicit git:: form for anything else, including private repositories reached over SSH (secure shell, an encrypted channel to another machine). The path after // can point at a subdirectory that carries its own kustomization.yaml, so local and remote pieces compose in a single build. The same syntax works for a components: entry as well as resources:.

resources:
  # host shorthand:  repo // path-in-repo ? ref
  - github.com/acme/platform-base//overlays/secure?ref=v1.4.0
  # git:: form works for any host, including private repos over SSH
  - git::ssh://[email protected]/acme/private-base//base?ref=v2.1.0
  # local files compose right alongside the remote ones
  - ./deployment.yaml

Run the build and Kustomize reaches out, clones what it needs, and assembles everything into one stream of YAML (the text format Kubernetes objects are written in). That stream is what actually gets applied. Read it, because the URL is only a promise about where the manifests came from, and the build output is the thing your cluster obeys. Piping the first few dozen lines through head, the way the command below does, is the cheapest security check you will ever run.

$ kustomize version
v5.4.3
$ kustomize build . | head -n 38

apiVersion: v1
kind: ServiceAccount
metadata:
  name: secure-app
  namespace: prod
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: platform-base-controller
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: platform-base-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: platform-base-controller
subjects:
- kind: ServiceAccount
  name: secure-app
  namespace: prod
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-app
  namespace: prod
spec:
  replicas: 3
  ...

Look at what arrived. A ClusterRole named platform-base-controller grants every verb on every resource in every API group, and a ClusterRoleBinding wires it to the app's ServiceAccount. That is cluster-admin in all but name, and none of it is in a file you wrote. It rode in on the remote base. The tidy resources: line in your repo looked harmless; the reality is a workload that can read every Secret and delete every namespace. This is the whole reason the topic exists.

Pin to an immutable ref

A branch or tag name is like the phrase 'this week's issue' of a magazine. The words stay the same while the pages behind them change. A commit SHA (secure hash algorithm; here, the fingerprint Git computes for every commit) is like citing one printed issue by its exact serial number. ?ref=main resolves to whatever HEAD (Git's pointer to the newest commit on a branch) happens to be at the moment you build, so your deploy shifts every time upstream pushes. A tag such as ?ref=v1.4.0 is better, but a tag can be deleted and re-pushed onto different content. A full 40-character commit SHA cannot move. It names one exact set of bytes, and that is what high-assurance setups pin to.

resources:
  # BAD: no ref resolves to the default branch HEAD on every build
  - github.com/acme/platform-base//overlays/secure
  # BAD: a branch name still moves whenever upstream pushes
  - github.com/acme/platform-base//overlays/secure?ref=main
  # OK: a tag is stable until someone re-pushes it onto new content
  - github.com/acme/platform-base//overlays/secure?ref=v1.4.0
  # BEST: a full commit SHA is immutable, the same bytes every build
  - github.com/acme/platform-base//overlays/secure?ref=9f2c1ab3e0d7c4b8a6f5e2d1c0b9a8f7e6d5c4b3

You can watch a branch move without touching your own repository. git ls-remote asks the remote what a name points at right now. Run it today, run it after the next upstream merge, and the SHA on the left changes while refs/heads/main on the right stays put. That churning left-hand column is exactly what a bare URL or ?ref=main hands your cluster.

$ git ls-remote https://github.com/acme/platform-base.git main

7d4e9c2a1b8f0e6d3c5a4b2e1f0d9c8b7a6e5d4c	refs/heads/main

Here is the attack that pinning stops. A maintainer's token leaks, or a maintainer is careless, and a privileged pod or a widened RBAC rule lands on main between two of your own unrelated commits. If you track the branch, that change flows into your next build with nothing in your Git history to show for it, because your files never changed. Pin to the SHA and the same push is inert until you deliberately bump the ref, which shows up as a reviewed one-line diff that a human approves. The upgrade becomes a decision instead of an accident, and a defender reading the pull request sees the new fingerprint before it ever reaches a node.

Build and inspect before you trust

Pinning fixes which bytes you fetch. It says nothing about what those bytes do. A remote base is code you run, so read the assembled manifests before you trust them. The fast triage is to grep the build output for the objects that grant power: ServiceAccounts, Roles and ClusterRoles, and their bindings.

$ kustomize build . | grep -E '^kind: (ClusterRole|Role|ServiceAccount)'

kind: ServiceAccount
kind: ClusterRole
kind: ClusterRoleBinding

Three grants from a base you thought only set some labels is a reason to open the full output and read the rules line by line. Beyond RBAC, scan for the other ways a manifest reaches off its leash: a hostPath volume that mounts the node's own filesystem, hostNetwork set on the pod, or privileged: true in a container's securityContext, an image pulled from a registry you do not recognize, or a validating webhook that phones an external URL. None of these are visible from the resources: line. All of them are plain in the built YAML, which is the only view that tells the truth.

Vendor it with kustomize localize

Instead of phoning the supplier for sauce on every single order, you buy a batch, store it in your own pantry, and label the jar. A future recipe change reaches you only when you choose to restock, and you can taste the new batch before it goes near a customer. Vendoring a remote base does the same thing. kustomize localize (available in Kustomize v5 and later) downloads the remote references and writes a self-contained copy into a directory in your own repository.

$ kustomize localize \
    'github.com/acme/platform-base//overlays/secure?ref=9f2c1ab3e0d7c4b8a6f5e2d1c0b9a8f7e6d5c4b3' \
    ./vendor
$ ls -F vendor

SUCCESS: localized "github.com/acme/platform-base//overlays/secure?ref=9f2c1ab3e0d7c4b8a6f5e2d1c0b9a8f7e6d5c4b3" to directory /home/deploy/app/vendor
kustomization.yaml  deployment.yaml  rbac.yaml  localized-files/

Now your build has no live network dependency. The base lives in your repo under vendor/, and any remote references it makes were rewritten to point at the downloaded copy under vendor/localized-files/. Every future upstream change arrives the only way it can, as a commit to your repository that a reviewer reads before it merges. Build the vendored copy offline and run it through the same schema check you use on your own manifests. kubeconform (a fast validator that checks Kubernetes YAML against the official object schemas) is a common choice, and it belongs in the same CI gate as the rest of your policy checks.

$ kustomize build ./vendor | kubeconform -strict -summary -

Summary: 6 resources found parsing stdin - Valid: 6, Invalid: 0, Errors: 0, Skipped: 0

You do not want to rely on everyone remembering the rule. Encode it. A short grep in continuous integration (the automated pipeline that builds and checks every change) can fail the build the moment a remote base is pinned to anything other than a full commit SHA.

# Fail CI if any remote base is not pinned to a full 40-char commit SHA.
$ grep -rEn 'github\.com|gitlab\.com|git::' overlays/ \
    | grep -vE '\?ref=[0-9a-f]{40}\b' \
    && { echo 'unpinned remote base found'; exit 1; } || true

overlays/prod/kustomization.yaml:5:  - github.com/acme/platform-base//overlays/secure?ref=main
unpinned remote base found

That single check is worth more than any one careful review, because it never gets tired and never waves through a branch ref by reflex on a busy Friday. Wire it in before your first remote base ships, and an unpinned dependency stops being something you hope a reviewer notices and becomes something the pipeline flatly refuses to build.

Try this

Run kustomize version on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a branch-tracked base is untrusted live code. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related