Crossplane in GitOps & securing it

Argo/Flux drive infra; protect the plane.

Your cluster is already a control plane. Crossplane watches objects in the Kubernetes API (the cluster's central database and command surface) and drives real cloud infrastructure to match them, on a loop that never stops. GitOps moves the one manual step that is left, the kubectl apply you type by hand (kubectl is the command-line tool that talks to the cluster), back into a Git repository: a version-controlled folder of text files that records every change and the person who made it. A separate program watches that folder and applies whatever appears. It works like a work-order book bolted to the wall beside a locked door, with a robot inside that reads each order and carries it out using a master key it keeps locked away. Nobody carries the key now. You write in the book, and the robot makes the building match the book.

GitOps is that arrangement with real names attached. Git holds the desired state. A GitOps controller, either Argo CD or Flux (two programs that run inside Kubernetes and watch a repository for you), sees a new commit and applies it into the cluster. Crossplane then reconciles the cluster into the cloud. Two control loops run back to back, both pointing the same way: the controller keeps the cluster matching Git, and Crossplane keeps the cloud matching the cluster. One commit, reviewed and merged, can travel all the way to a running database. That reach is the point. It is also the danger. A wrong commit reaches the cloud exactly as faithfully as a right one, so most of this lesson is about guarding the control plane. Aiming it at real work is the easy half.

Point Argo CD at a repo of claims

Keep two Applications, not one. Think of a workshop with two supply lines: one delivers the heavy machinery and the wiring, the other delivers the day's work orders. The first Application installs Crossplane itself, its providers (the plugins that each talk to one cloud's API), its functions and compositions (the templates that turn a request into real resources), and its ProviderConfigs (each one names the credential a provider should use). That is the machinery. The second Application holds the requests: your claims (a developer's ask for a piece of infrastructure) and any composite resources, or XRs (a composite resource is one object that fans out into many cloud resources), that you commit directly. Machinery and requests move on different clocks and carry different risk. Upgrading a provider is a control-plane event you want reviewed slowly. Filing a claim is routine. Splitting them means a broken claim sync cannot wedge a provider upgrade, and a botched provider rollback cannot sweep away everyone's live infrastructure in one go. The Application below watches only the claims path.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: platform-claims
  namespace: argocd
spec:
  project: platform                 # a locked-down AppProject (below), never "default"
  source:
    repoURL: https://github.com/acme/infra.git
    targetRevision: main
    path: claims                    # only claims and composites live here
  destination:
    server: https://kubernetes.default.svc
    namespace: team-a
  syncPolicy:
    automated:
      prune: true                   # DELETES real cloud infra on removal - read the warning
      selfHeal: true                # reverts hand-edits in the cluster back to Git
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true            # shares field ownership; also dodges the 256 KiB client-side limit
      - RespectIgnoreDifferences=true   # so selfHeal honors ignoreDifferences below
  ignoreDifferences:
    - group: '*'
      kind: '*'
      managedFieldsManagers:
        - crossplane                # stop flagging fields the provider fills in as drift

Three lines in that file carry the weight. ServerSideApply=true changes how Argo writes the object. Instead of stamping the whole thing down as one blob, Argo shares ownership of the fields, like two people editing the same shared document where each one owns only the paragraphs they wrote. Argo owns the fields you put in Git; Crossplane and the API server own the fields they fill in. The older client-side method keeps a full copy of the object inside an annotation called kubectl.kubernetes.io/last-applied-configuration, and all of an object's annotations together can hold at most 262144 bytes, which is 256 KiB (kibibytes). Crossplane's generated objects run large. The provider CRDs (Custom Resource Definitions, the generated schemas that teach the cluster about every cloud resource) are the worst offenders, and a heavy managed resource can crowd the ceiling too, so client-side apply overruns the limit and fails with metadata.annotations: Too long. Server-side apply carries no such annotation, so it sidesteps the limit and shares fields cleanly at the same time. selfHeal: true means that if someone edits the live object by hand with kubectl edit, Argo notices the drift from Git and puts it back. ignoreDifferences with managedFieldsManagers: [crossplane] tells Argo to stop counting the fields that Crossplane's own field manager writes (the values a provider fills in after a resource exists) as drift. Pair that with RespectIgnoreDifferences=true, or selfHeal will ignore those rules during automated sync and revert the very fields you told it to leave alone, and Argo and Crossplane will wrestle over them forever.

# is the GitOps controller happy, and did the cloud infra actually come up?
kubectl get application -n argocd platform-claims
kubectl get managed

NAME              SYNC STATUS   HEALTH STATUS
platform-claims   Synced        Healthy
 
NAME                                          READY   SYNCED   EXTERNAL-NAME     AGE
bucket.s3.aws.upbound.io/acme-logs-prod       True    True     acme-logs-prod    4m
instance.rds.aws.upbound.io/orders-db-9f3a1   True    True     orders-db-9f3a1   4m

Or let Flux drive it

Flux fits Crossplane like a glove, because Flux is itself a set of Kubernetes controllers, the same shape as Crossplane. You give it a GitRepository object, which tracks a URL and a branch and pulls the repo on an interval, and a Kustomization object, which applies a chosen path from that source on its own interval. The line that earns its place is wait: true. It makes Flux block until every object it applied reports Ready. For a Crossplane claim, Ready does not mean the YAML landed. It means the composed cloud resources actually came up, because Flux reads each object's status conditions with a small library called kstatus that understands the standard Ready condition Crossplane sets. That turns a sync into a real health gate: green means the database exists, not that the file parsed.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: infra
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/acme/infra.git
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: platform-claims
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: infra
  path: ./claims
  prune: true          # same danger as Argo's prune - read the warning
  wait: true           # block until XRs report Ready (cloud infra truly up)
  timeout: 5m

flux get kustomizations

NAME              REVISION            SUSPENDED   READY   MESSAGE
platform-claims   main@sha1:2b7c9a1   False       True    Applied revision: main@sha1:2b7c9a1

The blast radius moved to the pull request

Here is the shift that rewrites your threat model. Before GitOps, an engineer who wanted to build cloud infrastructure needed a cloud credential of their own. Now they need one thing: the ability to merge a pull request (a proposed change to the repo that someone reviews before it becomes part of the main branch) into the infrastructure repo. The credential that talks to the cloud lives inside the cluster, in a Kubernetes Secret (an object that stores sensitive values like credentials) that a ProviderConfig (Crossplane's setting that names which credential to use) points at. The mechanics are in xp-providerconfig and xp-connection. Argo or Flux turns the merge into a claim, Crossplane picks it up, and Crossplane's stored credential does the real work. The person who merged never held a cloud key, yet they created or destroyed real infrastructure. Your security boundary is no longer the cloud console. It is the merge button.

An attacker reasons the same way. If they can land a commit, by phishing a maintainer, stealing a token that has write access, or self-approving their own change, they can add a claim that spins up a fat compute instance to mine crypto, quietly widen an IAM role (Identity and Access Management, the cloud's permission system), or repoint a ProviderConfig at an account you never meant to touch. Your defenses live at the pull request and in the plane's own limits: required reviews so no one merges alone, a CODEOWNERS file (which forces named owners to approve changes to specific paths) over the sensitive files like compositions and ProviderConfigs, and signed commits so a stolen token cannot forge authorship. The provider credentials never sit in Git as plaintext. You commit a SealedSecret (an encrypted blob only an in-cluster controller can decrypt) or an ExternalSecret (a pointer to a real secret in Vault or a cloud secret manager), and a controller turns it into the live Secret inside the cluster.

Fence the engine with an AppProject

An Argo CD Application, left alone, can pull from any repo and create almost any kind of object in any namespace. That is too much rope. An AppProject is a fence you draw around a set of Applications, like a job ticket that names which shelf a worker may reach and which tools they may pick up. You pin the exact repo it may read, the exact clusters and namespaces it may write to, and the exact API groups and kinds it may create. With the fence up, an infra Application can make your platform's own resources and namespaces and nothing else. If a bad commit tells it to create a ClusterRole (a cluster-wide permission grant) or some workload it has no business making, Argo refuses before Crossplane ever sees the object.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: platform
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/acme/infra.git    # this repo only
  destinations:
    - server: https://kubernetes.default.svc
      namespace: 'team-*'                  # team namespaces only
  clusterResourceWhitelist:                # cluster-scoped kinds it may create
    - group: acme.platform.io              # your composites (XRs)
      kind: '*'
    - group: ''                            # core Namespace, so CreateNamespace works
      kind: Namespace
  namespaceResourceWhitelist:              # namespaced kinds it may create
    - group: acme.platform.io              # your claims
      kind: '*'

# someone slipped a ClusterRole into the repo. did the fence hold?
argocd app get platform-claims | grep -i "not permitted"

SyncError   Resource rbac.authorization.k8s.io/ClusterRole:acme-admin is not permitted in project platform

Keep prune from bulldozing production

The feature that scares people is real, and it should. With prune: true and automated sync, the cluster is a mirror of Git. Delete a claim's file from the repo, revert a branch, or fumble a Kustomize refactor (Kustomize is the tool that assembles your final YAML from smaller overlay files) so one file drops out of the render, and the controller reads that as "this object should not exist" and removes it from the cluster. Crossplane's default deletionPolicy is Delete, so removing the claim cascades into the composed managed resources, and each one calls the cloud API to destroy the real thing. A merge meant to rename a resource can drop a production database. There is no undo on that.

Defend in layers. Start with deletionPolicy: Orphan (or a managementPolicies list that leaves Delete out) on anything that holds data. Orphan is the difference between handing back the keys and leaving the house standing, versus calling in the demolition crew: removing the Kubernetes object detaches the managed resource and leaves the live cloud resource alone instead of destroying it. Next, mark resources you cannot lose with a sync option that forbids pruning. In Argo that is the annotation argocd.argoproj.io/sync-options: Prune=false. In Flux it is kustomize.toolkit.fluxcd.io/prune: disabled. Then add PruneLast=true so any pruning that does happen runs at the very end of a sync, after everything else settles, which gives a broken sync a chance to fail loudly before it reaps anything. None of this replaces the pull request. Branch protection with required review is where you stop a destructive commit before it ever becomes a desired state, the same discipline the least-privilege work in xp-security depends on.

# the scary default: does deleting this claim destroy the real database?
kubectl get instance.rds.aws.upbound.io orders-db-9f3a1 \
  -o jsonpath='{.spec.deletionPolicy}'; echo
 
# is the claim itself fenced off from Argo's pruner?
kubectl -n team-a get postgresinstance orders-db \
  -o "jsonpath={.metadata.annotations['argocd.argoproj.io/sync-options']}"; echo

Delete
Prune=false

Make these checks a habit, not a one-time setup. Once a week, list who can merge to the infra repo and confirm it still matches your platform team. Diff the live AppProject and every ProviderConfig against what Git says, because a change there changes what your cloud credentials are allowed to do. Run kubectl get managed and reconcile the result against your claims: anything Crossplane is managing that no claim asked for is either drift you missed or something a person added outside review. The pull request is your strongest control, so keep it sharp. The day self-approval quietly comes back is the day your fence has a gate that nobody is watching.

Try this

Run kubectl get application -n argocd platform-claims on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: prune plus automated sync can delete real cloud resources. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related