Multi-tenancy & RBAC

Isolate teams safely.

One Kubernetes cluster, five teams, everyone wants to ship at once. The lazy move is to hand every team a wide-open login and hope nobody fat-fingers a change into someone else's space. Multi-tenancy is the discipline that lets those teams share one cluster without reaching into each other's work, or into the cluster's control plane (the components that actually run and govern the cluster, like a building's management office).

Here is the picture to keep in your head. The cluster is an apartment building. Each team gets a unit, which is its namespace (Kubernetes' way of drawing a boundary around a group of objects so that names and access stay separate). Each team gets a key, which is a ServiceAccount (SA for short, the non-human identity Kubernetes hands to a workload so the API server knows who is asking). The building's locks are RBAC (Role-Based Access Control, the rules that decide which identity may perform which verb on which resource). Flux is the doorman. When it applies a team's manifests (the YAML files that describe what the team wants running), it does not reach for its own master key. It borrows the tenant's key and opens exactly the doors that key opens, nothing more.

Flux has no special tenant object. There is no 'project' resource to configure and no second permission system to keep in sync. Multi-tenancy here is three ordinary Kubernetes pieces (namespaces, ServiceAccounts, RBAC) plus one field, spec.serviceAccountName, that tells a controller whose key to borrow. Get that field and the locks behind it right, and the cluster's own API server becomes the fence.

Ask The API Server What A Tenant Can Do

Before you trust any of this, learn to interrogate it. kubectl (the command-line tool that talks to the cluster's API server, the single front door every request passes through) can pretend to be another identity and ask a yes-or-no question with the --as flag. This is your cheapest, most honest audit. You are not reading YAML and guessing what it grants. You are asking the same authorizer that Flux will hit at apply time.

# Impersonate tenant team-a's ServiceAccount and probe two verbs:
kubectl auth can-i create deployments -n team-a \
  --as=system:serviceaccount:team-a:team-a
 
kubectl auth can-i create clusterrolebindings \
  --as=system:serviceaccount:team-a:team-a

Read that carefully. The tenant's key can create a Deployment (a running application) inside its own namespace: yes. It cannot create a ClusterRoleBinding, a cluster-wide grant that would let the tenant hand itself new powers: no. That gap between the two answers is the entire boundary. Everything below exists to make sure Flux applies changes as this identity and no other.

Scaffold A Tenant

The platform team owns the cluster-wide Flux install and hands each tenant a furnished workspace: a namespace, a ServiceAccount to reconcile as (reconcile is Flux's word for making the cluster match what is written in Git), and a binding that grants that account what it needs inside its own walls. The flux CLI (command-line interface) writes exactly that set of objects. Use --export to print the YAML (a plain-text configuration format) to your terminal, so you can commit it to Git and review it in a pull request.

flux create tenant team-a \
  --with-namespace=team-a \
  --cluster-role=cluster-admin \
  --export

Look at the last object. It is a RoleBinding, not a ClusterRoleBinding. That one-word difference is why the earlier audit answered 'no' on cluster-scoped resources. A RoleBinding grants the powers of the ClusterRole it references (here, cluster-admin) only inside the namespace it lives in. So even passing --cluster-role=cluster-admin makes team-a an admin of team-a and nowhere else. That is safe, but it is more power than most tenants need. Prefer a tighter ClusterRole, or a plain namespaced Role, so a tenant cannot mint new RBAC or reach resources outside its remit even within its own namespace.

Impersonation Is The Whole Fence

The tenant now commits their own manifests. Their Kustomization (Flux's object that points at a source and applies the manifests it finds there) carries one field that turns RBAC into a hard wall: spec.serviceAccountName. Set it, and the kustomize-controller (the Flux component that reads Kustomizations and applies them) impersonates that ServiceAccount for every apply and every prune (prune deletes objects Flux previously created once they leave Git).

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: team-a-apps
  namespace: team-a
spec:
  interval: 10m
  serviceAccountName: team-a          # controller applies AS this key
  sourceRef:
    kind: GitRepository
    name: team-a
    namespace: team-a                 # tenant's own source, same namespace
  path: ./apps
  prune: true

The payoff is symmetry. What a tenant can deploy through Flux is exactly what their key can do at the API server, no more and no less. If their ServiceAccount cannot create a resource, the reconcile fails with an ordinary RBAC 'forbidden' error, the same one they would see running kubectl apply by hand as that account. Nothing new to learn, nothing extra to keep in sync. Keep each tenant's source (its GitRepository object, which tells Flux which repo to pull from) in the tenant's own namespace and reference it there, so a team's GitOps (the practice of keeping desired cluster state in Git and continuously reconciling the cluster to match) can only pull from repositories the team owns.

Watch It Fail Closed

Say the tenant's Git tree includes a manifest that would grant them cluster-admin across the whole cluster, a classic escalation attempt. A ClusterRoleBinding is cluster-scoped, and the tenant's key only holds namespaced rights. The reconcile stops dead. Here is what a defender sees when they check the tenant's Kustomizations.

flux get kustomizations -n team-a

That MESSAGE is the whole model working out loud. The controller tried to apply the escalation as team-a's key, the API server refused, and the failure is loud and specific: which identity, which verb, which resource, which scope. This is exactly the signal you want feeding your alerts. A tenant Kustomization that flips to READY=False with a 'forbidden' message is either a misconfiguration or someone probing the walls, and either way you want to know tonight, not next quarter.

Fail Closed On The Controllers

Setting serviceAccountName on today's Kustomizations is necessary but not sufficient, and this is where most self-inflicted breaches live. By default the controllers fall back to their own ServiceAccount whenever the field is missing, and that account is highly privileged. A sourceRef (the pointer from a Kustomization to the source it pulls from) can also point across namespaces at another team's repository. Close both holes once, cluster-wide, with three arguments on the kustomize-controller, and the first two on the helm-controller (the component that reconciles HelmRelease objects, which install and upgrade Helm charts, the packaged apps of Kubernetes).

You inspect a running controller's arguments the same way you would check any Deployment's container args. On a fresh bootstrap (the initial Flux install) they look like this, with none of the isolation flags present, which means the install is default-open.

kubectl -n flux-system get deploy kustomize-controller \
  -o jsonpath='{range .spec.template.spec.containers[0].args[*]}{@}{"\n"}{end}'

To make the fence permanent, add a patch to the Flux bootstrap Kustomization so the flags are reapplied on every reconcile and survive upgrades. This file lives in your fleet repo alongside the generated Flux components.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patches:
  # Applies to BOTH controllers:
  - target:
      kind: Deployment
      name: "(kustomize-controller|helm-controller)"
    patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-cross-namespace-refs=true   # sources must be same-namespace
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --default-service-account=default # empty field => unprivileged SA
  # kustomize-controller only:
  - target:
      kind: Deployment
      name: kustomize-controller
    patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --no-remote-bases=true            # no arbitrary remote bases

Read each flag as a lock. --default-service-account=default means any Kustomization or HelmRelease that forgets serviceAccountName reconciles as the namespace's default ServiceAccount, which carries no RBAC bindings and can therefore do nothing. A missing field now fails closed instead of open. --no-cross-namespace-refs stops a team pointing its sourceRef at another tenant's repository. --no-remote-bases blocks pulling arbitrary remote kustomize bases (external manifests fetched at build time) that nobody reviewed. After Flux reconciles the patch, confirm the change actually landed on the live Deployment.

kubectl -n flux-system get deploy kustomize-controller \
  -o jsonpath='{range .spec.template.spec.containers[0].args[*]}{@}{"\n"}{end}'

Re-run the kubectl auth can-i checks from the top of this lesson after every RBAC change you ship. Impersonate the tenant SA and confirm 'no' on the verbs that would let it escape: create clusterrolebindings, get secrets in other namespaces, patch nodes. That is the fastest way to catch a binding that grants more than you meant, before Flux ever applies it in anger.

Try this

Run flux get kustomizations -n team-a on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a default-open controller quietly grants cluster-admin. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related