Projects, multi-cluster & RBAC

Boundaries and permissions.

An Argo CD Project is like the work permit you hand a contractor. The permit says nothing about who the person is. It says which sites they may enter, which materials they may bring in, and which rooms are off limits. Argo CD (a tool that keeps a Kubernetes cluster continuously matching the configuration you store in Git, a practice called GitOps) works the same way. Kubernetes (the system that runs and schedules containers across a fleet of machines) is where the work lands. A Project, stored as an object called an AppProject, wraps a group of Applications and pins down what they may do: which Git repositories they may pull from, which clusters and namespaces they may deploy into, and which kinds of Kubernetes resource they may create.

There are two separate gates here, and confusing them is where teams get burned. One gate is the AppProject: what an Application is allowed to touch. The other is RBAC (role-based access control, meaning your permissions come from the role you hold, not from who you are personally), which decides who may press buttons in the Argo CD web interface or drive its API (application programming interface, the machine-to-machine way to control it). The Project fences the app. RBAC fences the human. You need both, and Argo CD checks them independently.

The Default Project Trusts Everyone

Fresh installs ship with a project named default, and it is wide open. The fields that decide where an app can reach are all set to a wildcard. Any Application you drop into it can pull from any repo and deploy anywhere Argo CD holds credentials. For one small team running their own Argo CD, that is convenient. For a shared platform serving many teams, it is a hole. Look at what default actually grants.

argocd proj get default

Name:                         default
Description:
Destinations:                 *,*
Repositories:                 *
Source Namespaces:            <none>
Scoped Repositories:          <none>
Allowed Cluster Resources:    */*
Scoped Clusters:              <none>
Denied Namespaced Resources:  <none>
Signature keys:               <none>
Orphaned Resources:           disabled

Read those lines like a permit that says anything, anywhere. Destinations *,* means any cluster and any namespace. Repositories * means any Git URL. Allowed Cluster Resources */* means an app in this project may create cluster-scoped objects, including a ClusterRoleBinding (an object that grants a set of cluster-wide permissions to whoever it names) that could hand that power to anyone it likes. That is the blast radius of one careless Application, and it is why serious deployments never leave real workloads in default.

Fencing a Team With an AppProject

To pen a team in, you write your own AppProject. Here is one for a payments team, kept deliberately tight. Notice that it never names a person. It only describes limits on the apps.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: payments
  namespace: argocd
spec:
  description: Payments team apps
  sourceRepos:
    - https://git.acme.internal/payments/*   # only repos under this path
  destinations:
    - server: https://10.0.1.5:6443          # only the prod cluster
      namespace: 'payments-*'                # only namespaces starting payments-
  clusterResourceWhitelist: []               # may create NO cluster-scoped resource
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota                     # may not touch quotas, even its own

sourceRepos is an allowlist of Git URLs, and wildcards are allowed. If a repo is not on the list, an Application in this project cannot sync from it, even when Argo CD holds working credentials for that repo somewhere else. destinations pairs a cluster (named by its API server address) with the namespaces it may write to. Now the quirk worth learning by heart. Cluster-scoped resources follow an allowlist, so setting clusterResourceWhitelist to an empty list allows nothing at all: no ClusterRole, no CustomResourceDefinition (a way to teach Kubernetes an entirely new kind of object), no ClusterRoleBinding. Namespace-scoped resources work the opposite way. They are all permitted unless you deny them, so you reach for namespaceResourceBlacklist to subtract the dangerous kinds one at a time. Here the team may not create a ResourceQuota (the group is the empty string because ResourceQuota lives in the core Kubernetes group), so they cannot quietly widen their own resource limits.

kubectl apply -f payments-project.yaml
argocd proj get payments

appproject.argoproj.io/payments created
 
Name:                         payments
Description:                  Payments team apps
Destinations:                 https://10.0.1.5:6443,payments-*
Repositories:                 https://git.acme.internal/payments/*
Source Namespaces:            <none>
Scoped Repositories:          <none>
Allowed Cluster Resources:    <none>
Scoped Clusters:              <none>
Denied Namespaced Resources:  /ResourceQuota
Signature keys:               <none>
Orphaned Resources:           disabled

Now point a payments Application at a repo outside the allowlist, or at the kube-system namespace, and the sync fails the project's checks before a single object reaches the cluster. That is the payoff. The check runs before anything is applied, not after the damage is done. To confirm a project after you write it, read it back with argocd proj get and check that every line is narrower than a wildcard.

One Argo CD, Many Clusters

Argo CD reaches other clusters the way a building manager carries one ring of keys. It keeps a single credential per cluster, and every Application names, in its destination, which door to open. You register a cluster once, from a shell that already has kubectl access to it.

argocd cluster add prod-context

WARNING: This will create a service account `argocd-manager` on the cluster referenced by
context `prod-context` with full cluster level privileges. Do you want to continue [y/N]? y
INFO[0001] ServiceAccount "argocd-manager" created in namespace "kube-system"
INFO[0002] ClusterRole "argocd-manager-role" created
INFO[0002] ClusterRoleBinding "argocd-manager-role-binding" created
Cluster 'https://10.0.1.5:6443' added

Read that warning slowly, because it is the most important sentence in this lesson. Adding a cluster creates a ServiceAccount (a non-human identity that programs use to talk to the cluster) named argocd-manager, then gives it its own ClusterRole, argocd-manager-role, that permits every action on every resource. That is cluster-admin power under a different name. Argo CD now holds a key that can do anything on that cluster. Do this for ten clusters and the Argo CD control plane becomes the most valuable target in your estate: whoever owns Argo CD owns every cluster it was handed.

argocd cluster list
kubectl get secret -n argocd -l argocd.argoproj.io/secret-type=cluster

SERVER                          NAME        VERSION  STATUS      MESSAGE  PROJECT
https://10.0.1.5:6443           prod        1.28     Successful
https://kubernetes.default.svc  in-cluster  1.28     Successful
 
NAME                          TYPE     DATA   AGE
cluster-10.0.1.5-2451092648   Opaque   3      2m

Each remote cluster is a Kubernetes Secret tagged with the label argocd.argoproj.io/secret-type: cluster. That label is how Argo CD finds these credentials, and it is also the first thing a defender greps for during a review. Open one and the exposure is obvious.

apiVersion: v1
kind: Secret
metadata:
  name: cluster-10.0.1.5-2451092648
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: prod
  server: https://10.0.1.5:6443
  config: |
    {
      "bearerToken": "eyJhbGciOi...redacted...",
      "tlsClientConfig": { "caData": "LS0tLS1CRUdJTi...redacted..." }
    }

RBAC: Who May Press the Button

The Project fences the app. Now fence the human. Argo CD ships its own badge system for its web interface and API, configured in a ConfigMap (a Kubernetes object that holds plain configuration) named argocd-rbac-cm. A badge system is only as good as its rule for a badge that matches nothing. If an unrecognised badge opens every door, you have no security at all. So the first thing you set is what happens to a request that matches no rule.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:''          # logged in but no matching rule -> nothing
  scopes: '[groups]'               # take the user's groups from their login token
  policy.csv: |
    # p = permission:  subject, resource, action, object, effect
    p, role:payments-dev, applications, get,  payments/*, allow
    p, role:payments-dev, applications, sync, payments/*, allow
    # g = group binding:  identity-provider group -> role
    g, acme:payments-team, role:payments-dev

Each p line reads left to right: a subject (here a role), a resource (applications, clusters, repositories, projects, and so on), an action (get, sync, create, delete, override), the object the action applies to, and finally allow or deny. For applications the object is written as project/app, so payments/* means every Application in the payments project and nothing outside it. The g line binds a group to a role. acme:payments-team is a group that comes from your identity provider, not something you define inside Argo CD.

Wire Argo CD to your company's single sign-on (SSO, the one login that already works across your other tools) through OIDC (OpenID Connect, a common web login standard) or SAML (Security Assertion Markup Language, its older enterprise cousin), and the groups a person belongs to ride along inside their login token. The scopes: '[groups]' line tells Argo CD to trust that groups claim. Now nobody signs in with a local password. People log in as themselves through your IdP (identity provider, the system that vouches for who they are), and their group decides their role. Remove someone from the group in one place and their Argo CD access disappears with it.

Grant the smallest thing that works. A developer needs sync and get on their own project, not admin over the whole install. To see what a rule actually grants before you trust it, ask Argo CD directly. can-i answers as the identity that is currently logged in, so log in as an ordinary payments developer and run it.

argocd account can-i sync applications payments/payments-api
argocd account can-i sync applications infra/cluster-autoscaler

yes
no

That is the fastest way to prove a fence is real. A payments developer can sync their own app and gets a flat no on the infra project. Put those two checks in a script that runs after every RBAC change, so a fat-fingered edit to policy.csv fails your test instead of surprising you during an incident.

Keep one habit for any shared Argo CD. For each project, log in as an ordinary member and run argocd account can-i twice: confirm they can sync their own apps, and confirm they get a flat no on someone else's project. If both hold, your fences are real. If either fails, an attacker who phishes that one developer's login inherits whatever the gap allows, right down to whichever clusters Argo CD holds keys for.

Try this

Run argocd proj get default on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the control plane is your soft underbelly. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related