>_SecOpsLog_
CoursesCrossplaneProviders, ProviderConfig & credentials

Providers, ProviderConfig & credentials

How Crossplane authenticates to clouds.

Intermediate12 min · lesson 7 of 12

Installing a provider gives your control plane (the Kubernetes cluster where Crossplane runs) the software that knows how to talk to a cloud. It does not give it permission to touch anything there. Those are two different jobs, the same way a banking app on a fresh phone is a separate thing from being signed into your account. Crossplane keeps that split clean. The Provider is the app: the controller code plus the object types it teaches your cluster. The ProviderConfig is the account you sign into: the credentials and settings the controller presents when it actually calls the cloud.

Every managed resource (the Crossplane object that stands in for one real cloud thing, a storage bucket, a database, a network) points at a ProviderConfig by name. That one pointer is how a single control plane reconciles against many clouds, many accounts, and many identities at once. Get this layer right and the rest of Crossplane is plumbing. Get it wrong and your resources either stall forever or, worse, quietly act against the wrong account.

The driver and the login

The Provider package you installed earlier pulls a controller (a small program that runs in a loop, comparing what you asked for against what exists and closing the gap) into the cluster and registers CRDs (Custom Resource Definitions, the mechanism that teaches Kubernetes brand-new object types) with names like Bucket or Instance. On its own that controller authenticates to nothing. The ProviderConfig is a separate, cluster-scoped object, owned by the provider family, that answers exactly one question: when the controller reconciles a resource, which credentials does it present to the cloud's API (Application Programming Interface, the network endpoints the cloud exposes for creating and reading things)? A managed resource chooses one through spec.providerConfigRef.name, which defaults to the literal string default. That default is a convention, not magic. If nothing named default exists, every resource that leans on it sits and waits.

provider-and-config.yaml
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-s3
spec:
  package: xpkg.upbound.io/upbound/provider-aws-s3:v1.1.0
---
apiVersion: aws.upbound.io/v1beta1   # this kind is shipped by the AWS provider family
kind: ProviderConfig
metadata:
  name: default                      # matched by spec.providerConfigRef.name
spec:
  credentials:
    source: Secret                   # read a static key out of a Kubernetes Secret
    secretRef:
      namespace: crossplane-system
      name: aws-creds
      key: creds

Apply that, then check that both the provider you named and the family provider it quietly pulls in as a dependency are healthy. The family provider is the one that actually ships the ProviderConfig type, so if it is not ready, your ProviderConfig never registers and nothing downstream works.

terminal
kubectl get providers
output
NAME                          INSTALLED   HEALTHY   PACKAGE                                               AGE
provider-aws-s3               True        True      xpkg.upbound.io/upbound/provider-aws-s3:v1.1.0        3m
upbound-provider-family-aws   True        True      xpkg.upbound.io/upbound/provider-family-aws:v1.1.0    3m

Handing over a static key

With source: Secret the ProviderConfig reads one key out of one Kubernetes Secret and nothing more. For the AWS family that key holds a standard credentials file in ini format (the plain name = value style the AWS command-line tool writes), so you can lift it straight out of the ~/.aws/credentials on your laptop. Here is the file, and the command that loads it into the cluster.

aws-credentials.txt
[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
terminal
kubectl create secret generic aws-creds \
  -n crossplane-system \
  --from-file=creds=./aws-credentials.txt
output
secret/aws-creds created

The three fields under secretRef (namespace, name, and key) have to match that real Secret exactly. Get the key wrong and the controller has nothing to read. That one typo is the single most common reason a resource sits at READY: False with an authentication error buried in its events. When it all lines up, a Bucket you applied against this config reaches SYNCED and READY on its own.

terminal
kubectl get buckets.s3.aws.upbound.io
output
NAME       SYNCED   READY   EXTERNAL-NAME   AGE
tutorial   True     True    tutorial        47s

Now break it on purpose. Flip one character in the access key, or point secretRef.key at a name that does not exist, and the same Bucket cannot authenticate. Describe it, and the cloud's own rejection is quoted straight back to you in the Synced condition.

terminal
kubectl describe bucket tutorial
output
Status:
  Conditions:
    Reason:   ReconcileError
    Status:   False
    Type:     Synced
    Message:  observe failed: cannot run refresh: refresh failed:
              InvalidClientTokenId: The security token included in the request
              is invalid. status code: 403
Events:
  Type     Reason                         Age               From                              Message
  ----     ------                         ----              ----                              -------
  Warning  CannotObserveExternalResource  14s (x6 over 1m)  managed/bucket.s3.aws.upbound.io  observe failed: InvalidClientTokenId: ... status code: 403
How a resource picks up credentials
1Managed resource
spec.providerConfigRef.name: prod
2ProviderConfig "prod"
names one credential source
3Credential source
Secret / InjectedIdentity / WebIdentity
4Provider controller
the only thing that calls the cloud
5Cloud API
AWS / GCP / Azure account
The ProviderConfig is an indirection layer: a resource names a config, the config names a credential source, and only the controller ever touches the cloud.

What that Secret actually is

A Kubernetes Secret is not a safe. Its value is base64-encoded (a reversible text encoding that anyone can undo, not a lock), so a single command turns your cloud keys back into plain text. Run it the way an attacker with read access to that namespace would.

terminal
kubectl -n crossplane-system get secret aws-creds \
  -o jsonpath='{.data.creds}' | base64 -d
output
[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
A Secret is encoded, not encrypted
base64 is not encryption. That value sits in etcd (the key-value database where Kubernetes stores all of its state), so anyone who can run kubectl get secret in the namespace, or who gets hold of an etcd backup, reads the raw cloud key. Three defenses matter: tighten RBAC (Role-Based Access Control, Kubernetes' own permission system) so almost nobody can read Secrets in crossplane-system, turn on encryption at rest for etcd, and never commit the Secret to Git. A base64 value in a manifest is plaintext to anyone who finds the repo.

Then sit with the blast radius. A static key is long-lived: if it leaks it stays valid until a human rotates it in the cloud. Its reach is whatever IAM (Identity and Access Management, the cloud's permission system) policy you attached, so an over-broad key on a shared control plane hands an intruder the whole account. And there is a quieter risk built into the design. The providerConfigRef is a plain name string, and nothing stops one resource from naming a ProviderConfig meant for a different account. Crossplane holds the credential and acts on the resource author's behalf, a textbook confused deputy (a trusted helper tricked into spending its authority for someone else). So whoever can create a Bucket can, in effect, use whatever credential its ProviderConfig points at. Your real cloud-access boundary is the cluster RBAC that decides who can create which managed resource kinds, not the key sitting in the Secret.

During an incident you will want the reverse lookup: which resources are actually tied to a given credential. The provider records every link as a ProviderConfigUsage object, so you can read them straight out of the API.

terminal
kubectl get providerconfigusage.aws.upbound.io
output
NAME                                   CONFIG-NAME   RESOURCE-KIND   RESOURCE-NAME   AGE
f3a9c1b2-7d8e-4a1f-9b2c-6e5d4c3b2a10   default       Bucket          tutorial        5m

Drop the static key: cloud-native identity

The credential you least want on a shared control plane is a long-lived key, so the best move is to store none. Every major provider family supports source: InjectedIdentity, which tells the controller to borrow the ambient identity of its own pod (the smallest thing Kubernetes runs, one or more containers together) instead of reading a Secret. On EKS (Elastic Kubernetes Service, AWS's managed Kubernetes) that plumbing is IRSA (IAM Roles for Service Accounts); on GKE (Google Kubernetes Engine) it is Workload Identity; on AKS (Azure Kubernetes Service) it is Managed Identity. The cloud hands the pod a short-lived, auto-rotated token, and no static key ever lands in a Secret.

There is a catch. Crossplane creates the controller's pod and its ServiceAccount (the identity a pod runs as inside the cluster) for you, and it owns them. Annotate that ServiceAccount by hand and the change does not stick, Crossplane reconciles it away or recreates the account on the next provider upgrade. Instead you declare the cloud-identity annotation in a DeploymentRuntimeConfig (the current replacement for the deprecated ControllerConfig), Crossplane stamps it onto the ServiceAccount it manages, and you wire the Provider to that config with runtimeConfigRef. The annotation points at an IAM role by its ARN (Amazon Resource Name, the unique identifier string AWS gives every resource). After that the ProviderConfig only has to say InjectedIdentity.

irsa-runtime.yaml
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: irsa-runtime
spec:
  serviceAccountTemplate:
    metadata:
      annotations:
        # Crossplane stamps this onto the SA it manages for the controller pod
        eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/crossplane-provider-aws
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-s3
spec:
  package: xpkg.upbound.io/upbound/provider-aws-s3:v1.1.0
  runtimeConfigRef:
    name: irsa-runtime            # the controller pod's SA now carries the IAM role
---
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: InjectedIdentity      # no Secret; use the pod's own cloud identity

Do not trust that it worked because the manifest applied cleanly. Check two things. First, that the annotation actually landed on the ServiceAccount Crossplane manages. Second, that the cloud's admission webhook (a bit of cluster code that can rewrite a pod as it is being created) injected the identity into the running pod, which on EKS means an AWS_ROLE_ARN and an AWS_WEB_IDENTITY_TOKEN_FILE show up in the pod's environment. If those variables are missing, IRSA is not wired up, and the provider falls back to no credentials and fails quietly.

terminal
# 1. the role annotation is on the managed ServiceAccount
kubectl -n crossplane-system get sa \
  -l pkg.crossplane.io/provider=provider-aws-s3 \
  -o jsonpath='{.items[0].metadata.annotations}'
 
# 2. the webhook injected the identity into the running pod
kubectl -n crossplane-system get pods \
  -l pkg.crossplane.io/provider=provider-aws-s3 -o name \
  | xargs -I{} kubectl -n crossplane-system exec {} -- env | grep AWS_
output
map[eks.amazonaws.com/role-arn:arn:aws:iam::123456789012:role/crossplane-provider-aws]
AWS_DEFAULT_REGION=us-east-1
AWS_REGION=us-east-1
AWS_ROLE_ARN=arn:aws:iam::123456789012:role/crossplane-provider-aws
AWS_STS_REGIONAL_ENDPOINTS=regional
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
The default ProviderConfig trap
Because providerConfigRef defaults to the literal name default, a resource with no explicit config attaches to whatever ProviderConfig happens to be named default, or stalls forever if none exists. In a multi-account cluster this bites hard. You create a scoped ProviderConfig named prod, forget to set providerConfigRef on one resource, and it quietly reconciles against default instead, possibly the wrong account. Two habits prevent it: set providerConfigRef.name explicitly on every resource and inside every Composition (Crossplane's way of packaging several resources behind one custom API), and never leave an unowned default lying around in a cluster that talks to more than one account. The reference is name-only and ProviderConfigs are cluster-scoped, so names are global and have to be unambiguous.
Quick check
01A provider runs on EKS and you want zero long-lived cloud keys stored anywhere in the cluster. Which setup actually achieves that?
Incorrect — that is exactly the long-lived key you are trying to avoid, sitting base64-encoded in etcd.
Correct — the pod uses IRSA, the cloud injects a short-lived token, and no static key is ever stored.
Incorrect — Crossplane owns and reconciles that ServiceAccount, so a hand-applied annotation is reverted and the role never attaches reliably.
Incorrect — with no ProviderConfig the resources stall; the controller still needs an identity to call the cloud.
02The lesson calls Crossplane's credential model a 'confused deputy': providerConfigRef is only a name, and Crossplane runs every cloud call with the credential that config points at. So what actually bounds who can act against a cloud account?
Correct — the lesson names cluster RBAC over resource creation as the real cloud-access boundary.
Incorrect — IAM bounds what the credential can do once used, not who inside the cluster can invoke it.
Incorrect — the reference is a plain string with no restriction; any resource can name any ProviderConfig.
Incorrect — base64 is reversible encoding, not a control; anyone who can read the Secret reads the key.
03A Bucket sits at READY=False. kubectl describe bucket shows a Synced condition of ReconcileError with InvalidClientTokenId: The security token included in the request is invalid. status code: 403. What is the most likely cause?
Incorrect — a region problem would not surface as an invalid-token 403; this is an authentication failure.
Incorrect — an uninstalled provider leaves a resource unreconciled; here the cloud answered with a 403, so a controller is calling it.
Correct — the lesson shows exactly this: a flipped character or a bad secretRef.key yields InvalidClientTokenId 403 in the Synced condition.
Incorrect — a name collision returns a conflict, not an invalid-security-token 403.

One command turns all of this into an audit you can run today. List every ProviderConfig next to the credential source it uses. Any row that still says Secret is a long-lived key you are choosing to keep.

terminal
kubectl get providerconfig.aws.upbound.io \
  -o custom-columns=NAME:.metadata.name,SOURCE:.spec.credentials.source
output
NAME      SOURCE
default   InjectedIdentity
prod      Secret

The default row uses the pod's own identity and stores nothing you have to rotate. The prod row still carries a static key, so that is the one to put on a rotation schedule, watch in your cloud audit log, and move onto WebIdentity or a cross-account role the moment that account can support it.

Try this

Run kubectl get providers on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a Secret is encoded, not encrypted. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

XRDs & self-service claimsConnection secrets

Related

What Infrastructure as Code isInfrastructure as Code & automationWhat Terraform is & how it worksTerraformState files are secretsTerraform security and complianceWhy Terragrunt existsTerragrunt