A teammate deletes an S3 bucket that Crossplane manages, straight from the AWS console, to free up the name. A minute later the bucket is back. What happened?

The reconcile loop saw the live cloud no longer matched the desired object in the cluster and recreated it, because the spec is enforced on every loop. The cluster holds the desired state and a controller closes the gap continuously, so an out-of-band deletion is read as drift and corrected.

You install the Amazon Web Services (AWS) S3 (Simple Storage Service) provider into a fresh Crossplane cluster. Beforehand, kubectl knew only built-in types like Pods. What does installing that one provider actually do to the cluster?

It registers a stack of Custom Resource Definitions, one per S3 resource type, and starts a controller pod that reconciles them. A provider does two jobs at once: it adds the CRDs so the API server understands a Bucket, and it runs a controller that drives those objects toward the real cloud.

Your AWS provider authenticates with a long-lived access key stored in a Kubernetes Secret named aws-creds in the crossplane-system namespace. A teammate's ServiceAccount is granted get on Secrets in that namespace. In a security review, how should you rate this, and what is the stronger fix?

It is a serious exposure: that ServiceAccount can read the raw cloud credentials, so binding the provider to a workload identity (Identity and Access Management Roles for Service Accounts) removes the standing key entirely. A static key in a Secret makes Secret-read access equal to cloud access, and workload identity is the stronger default because it removes the long-lived key.

Courses CrossplaneWhat Crossplane is & the control-plane model

What Crossplane is & the control-plane model

Kubernetes as a universal control plane.

Intermediate12 min · lesson 1 of 12

There are two ways to get a house built and kept standing. You can hire a contractor: they arrive with the blueprint, build to it, hand you the keys, and drive off. If a pipe bursts in February, that is your problem until you call someone back out. Or you can employ a live-in superintendent who keeps the blueprint on the wall, walks the building every morning, and quietly repairs whatever has drifted from the plan. Terraform and CloudFormation are the contractor. They read your files, make the cloud match, and exit. Crossplane is the superintendent. You install it into a Kubernetes cluster (the system most teams already run to schedule their containers), and from then on it holds your desired infrastructure and works without stopping to keep the real cloud lined up with it.

What a control plane actually is

That superintendent is a control plane. The term sounds grand, but it points at something specific: the part of a system that holds a desired state and works continuously to make reality match it. Kubernetes is already a control plane, for containers. You declare that you want three copies of a web app running, and a component called a controller (a small program that watches objects and drives the world toward them) compares the running copies against that number, again and again, replacing one the moment it dies. That checking loop has a name, reconciliation: read what you asked for, observe what actually exists, close the gap, and repeat for as long as the object lives. Crossplane takes that exact engine and aims it at your cloud account instead of at pods (the running copies of your containers). The word 'universal' in universal control plane means the loop is no longer boxed into containers. Give it the right plugin and it can hold a database, a network, or a DNS record (Domain Name System, the internet's address book). Anything reachable through an API (application programming interface, the set of commands a service exposes for other programs to call) is fair game.

One-shot run vs standing control plane

CLI infra-as-code run (Terraform, CloudFormation)

Runs, then exits

you trigger apply; it stops when done

State file on the side

a separate record that can drift, lock, or corrupt

Drift found on the next run

tampering sits untouched until someone re-applies

Crossplane control plane

Never exits

a controller keeps reconciling in the background

State is the cluster

desired stored in the cluster, observed refetched each loop

Drift fixed continuously

an out-of-band change is reverted within a poll

Same desired-state idea, delivered as a live loop instead of a command you have to remember to run.

Your API server learns to speak cloud

Here is the machinery underneath. Kubernetes lets you teach its API brand-new object types through a Custom Resource Definition (CRD, a way to register a kind of object the cluster has never heard of before). Crossplane ships plugins called providers, and installing one does two jobs at once: it registers a stack of CRDs, one per cloud resource type, and it starts a controller that knows how to reconcile them. Add the Amazon Web Services (AWS) storage provider and your cluster suddenly understands a Bucket. Add the database provider and it learns Instance for a managed database. From that point on, an S3 bucket (Simple Storage Service, Amazon's object storage) is a Kubernetes object you create with kubectl (the command-line tool for talking to a cluster), the same command you already point at pods. Right after an install, the control plane itself is only a couple of small pods, and each provider adds its own controller pod beside them.

terminal

kubectl get pods -n crossplane-system

output

NAME                                          READY   STATUS    RESTARTS   AGE
crossplane-796d5c9b8f-4t2xn                   1/1     Running   0          12m
crossplane-rbac-manager-5c7f4d9b6c-p8k9d      1/1     Running   0          12m
provider-aws-s3-3f1c9a2b7d4e-6d8f5c9b7-x2mtl  1/1     Running   0          6m

Those two core pods are the whole control plane at rest: the main crossplane controller (which also manages packages) and the RBAC manager (role-based access control, the rules for who may do what) that grants the fine-grained permissions each provider's CRDs need. The third pod is the provider, and it is what teaches the cluster a cloud's vocabulary. Ask the API server what it now knows in the S3 group, and every bucket-related type is there as a first-class resource you can get, label, and gate with policy.

terminal

kubectl api-resources --api-group=s3.aws.upbound.io

output

NAME                        SHORTNAMES   APIVERSION                  NAMESPACED   KIND
buckets                                  s3.aws.upbound.io/v1beta2   false        Bucket
bucketacls                               s3.aws.upbound.io/v1beta1   false        BucketACL
bucketpolicies                           s3.aws.upbound.io/v1beta1   false        BucketPolicy
bucketpublicaccessblocks                 s3.aws.upbound.io/v1beta1   false        BucketPublicAccessBlock
bucketversionings                        s3.aws.upbound.io/v1beta1   false        BucketVersioning

Read the NAMESPACED column: it says false. These cloud resource types are cluster-scoped, not tucked inside a namespace (a named folder that walls off one team's objects from another's) the way a Pod or a Secret is. That single fact drives the security model later in this lesson, because a namespaced Role can never grant them. Only a cluster-wide grant can, and cluster-wide grants are exactly the ones people hand out too freely.

Watch it hold the line

Descriptions are cheap, so watch the loop earn its keep. You apply a bucket the same way you apply anything else, from a manifest (the YAML file that describes a Kubernetes object; YAML is the plain-text format Kubernetes reads).

bucket.yaml

apiVersion: s3.aws.upbound.io/v1beta2
kind: Bucket
metadata:
  name: secopslog-demo-3f9c2a1        # this is also the real S3 bucket name
spec:
  forProvider:
    region: us-east-1
  providerConfigRef:
    name: default                      # which cloud credentials to use (xp-providerconfig)

terminal

kubectl apply -f bucket.yaml
 
# once the cloud call completes, both status columns read True:
kubectl get bucket secopslog-demo-3f9c2a1

output

bucket.s3.aws.upbound.io/secopslog-demo-3f9c2a1 created
 
NAME                     SYNCED   READY   EXTERNAL-NAME            AGE
secopslog-demo-3f9c2a1   True     True    secopslog-demo-3f9c2a1   73s

So far this looks like any provisioning tool. The difference shows the instant something changes the cloud behind Crossplane's back, which is what a mistake, a rogue script, or an attacker actually does. Delete the real bucket straight from the Amazon command line and wait one poll interval (the provider re-checks every resource about once a minute by default).

terminal

# delete the real bucket behind Crossplane's back (a fat-finger, or an attacker)
aws s3 rb s3://secopslog-demo-3f9c2a1
 
# give the controller one poll interval, then look again
kubectl get bucket secopslog-demo-3f9c2a1
kubectl describe bucket secopslog-demo-3f9c2a1 | grep -A4 Events

output

remove_bucket: secopslog-demo-3f9c2a1
 
NAME                     SYNCED   READY   EXTERNAL-NAME            AGE
secopslog-demo-3f9c2a1   True     True    secopslog-demo-3f9c2a1   11m
 
Events:
  Type    Reason                   Age   From                              Message
  ----    ------                   ----  ----                              -------
  Normal  CreatedExternalResource  9s    managed/bucket.s3.aws.upbound.io  Successfully requested creation of external resource

The bucket is back, and notice the Kubernetes object's age never reset to zero: only the cloud resource was rebuilt, because the desired object in the cluster never went away. The desired state lives in the cluster and is enforced on every reconcile, not saved up for the next apply. For a defender that cuts both ways. Casual tampering and accidental deletion get reverted for free, which is a real resilience win. But the cloud console has stopped being the source of truth, so an incident responder who deletes a compromised resource there will watch it reappear the moment the loop next runs. To actually stop it you change the spec in the cluster, narrow which actions the loop is allowed to take (the managementPolicies field, covered in xp-reconcile), or cut the provider's credentials. The console alone loses the argument every time.

One API for apps and infrastructure

Why move infrastructure into the same cluster that already runs your apps? Two payoffs keep pulling teams in. The first is that everything you built around Kubernetes now reaches your cloud as well. The same kubectl, the same RBAC, the same GitOps sync (a controller keeping the cluster matched to a Git repository) and admission checks (policy gates that vet an object before the cluster accepts it) that gate a Deployment (a running app workload) now gate a database, a network, and a bucket. One toolchain, one audit trail, one place to write and enforce policy. The second payoff is that a platform team can publish its own higher-level types. Rather than handing a developer a raw database resource with forty settings, you define a PostgresInstance that means what your organization decides it means (encryption on, backups scheduled, sane size limits) and let people request one by name. Those custom, self-serve APIs are what the later composition lessons (xp-xrd, xp-composition) build toward.

The cluster becomes a cloud root key

All that convenience has a sharp edge. The provider's controller runs around the clock holding credentials that can create and destroy real infrastructure, so the cluster Crossplane lives in has become one of the most sensitive systems you operate. Put plainly, your Kubernetes RBAC is now your cloud IAM (Identity and Access Management, the cloud's own permission system) for everything Crossplane manages. Anyone who can create a managed resource can spend money and widen your attack surface without ever signing in to a cloud console. That deserves a direct look rather than a comfortable assumption, and two checks tell you most of what you need.

terminal

# Managed resources are cluster-scoped, so creating cloud infra needs cluster RBAC.
kubectl auth can-i create buckets.s3.aws.upbound.io
kubectl auth can-i create buckets.s3.aws.upbound.io \
  --as=system:serviceaccount:team-a:deployer
 
# With a static key (not workload identity), the cloud creds live in a Secret:
kubectl get secret -n crossplane-system aws-creds

output

yes
no
 
NAME        TYPE     DATA   AGE
aws-creds   Opaque   1      27m

The first check confirms who can mint cloud resources. Your admin identity says yes; the ordinary team-a deployer says no, which is the safe default you want. Because these types are cluster-scoped, the only grant that reaches them is cluster-wide: a ClusterRole (a permission set that applies across the whole cluster, not inside one namespace) handed out through a ClusterRoleBinding. One binding that is too broad, say a wildcard ClusterRole tied to every account in a namespace, quietly gives that whole group the power to build and tear down production infrastructure. The second check finds where the cloud credential actually sits. A static key stored in a Secret means anyone who can read Secrets in that namespace can read your cloud keys outright. Binding the provider to a workload identity instead (IAM Roles for Service Accounts on Amazon, shown in xp-providerconfig) removes that standing key altogether and is the stronger default for a shared control plane.

kubectl verbs are now cloud verbs

Once a provider is installed, any person or token that can create, edit, or delete a managed resource can do the same to the real cloud resource behind it, with no cloud console and no separate IAM login. Two exposures catch teams out. Managed resources are cluster-scoped, so a wildcard ClusterRole (verbs '*' on resources '*'), once bound to any user or token, silently grants the ability to provision and destroy production infrastructure, and most clusters were never reviewed with that in mind. And the provider pod carries live cloud credentials, so permission to read the credential Secret, or to exec into that pod, is effectively cloud administrator access. Treat the crossplane-system namespace like a root account: scope ClusterRoles narrowly, prefer workload identity over static Secrets (xp-providerconfig), and give that namespace the same logging and alerting you would give a domain controller.

Try this

Run kubectl get pods -n crossplane-system on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: kubectl verbs are now cloud verbs. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Installing Crossplane & a provider

What Infrastructure as Code isInfrastructure as Code & automation What Terraform is & how it worksTerraform State files are secretsTerraform security and compliance Why Terragrunt existsTerragrunt

What Crossplane is & the control-plane model

What a control plane actually is

Your API server learns to speak cloud

Watch it hold the line

One API for apps and infrastructure

The cluster becomes a cloud root key

Try this

Takeaway

Related