Connection secrets

Deliver endpoints and passwords to apps.

A hotel doesn't hand you the plumbing diagrams when you check in. You get one key card, already tied to a room that's cleaned and ready, waiting at the desk for you to pick up. A Crossplane connection secret works the same way. When a database or other resource finishes provisioning, Crossplane takes the details an app needs to reach it (the address, the port, the username, the password) and writes them into a Kubernetes Secret (a built-in object that holds small pieces of sensitive data as base64 text, a reversible encoding rather than encryption) that you named. The app reads that Secret and connects. It never sees how the database was built, how it was wired, or where the password came from.

That last part is the security story. The password in a connection secret is almost always machine-generated. No engineer typed it, no engineer knows it, and the connection secret is often the only place it exists. Get the delivery right and the credential reaches exactly one app and nowhere else. Get it wrong and you have a live production database password sitting in a namespace (Kubernetes' folder-like partition that keeps one team's objects separate from another's) that half the cluster can read. So this lesson is as much about who can see the Secret as about how the Secret gets filled.

Where the details come from

Every managed resource (Crossplane's name for one external thing it controls, like a single Amazon RDS instance, where RDS is Amazon's managed relational database service) can publish its own connection details the moment it goes ready. RDS hands back its endpoint, its port, the master username, and the password it generated. Crossplane writes those into a raw Secret in the crossplane-system namespace. Here's the part that trips people up: the key names inside that Secret are whatever the provider decided to call them, and they differ from one provider to the next. Don't guess them. Look.

# the composed RDS instance writes a raw connection Secret; find where
kubectl get instance.rds.aws.upbound.io orders-db-8n4m2 \
  -o jsonpath='{.spec.writeConnectionSecretToRef.namespace}/{.spec.writeConnectionSecretToRef.name}{"\n"}'

crossplane-system/orders-db-8n4m2-conn

# list the raw provider keys inside that Secret
kubectl -n crossplane-system get secret orders-db-8n4m2-conn -o json \
  | jq -r '.data | keys[]'

address
attribute.password
endpoint
host
password
port
username

Read that list closely, because two lessons hide in it. The friendly names (address, endpoint, host, port, username, password) come from a mapping the AWS provider's authors wrote by hand for RDS, so you get human-readable keys instead of raw internals. The odd one out, attribute.password, is the raw copy. Upjet (the engine that turns a Terraform provider into a Crossplane one) takes every field the underlying Terraform schema marks secret and republishes it untouched under an attribute. prefix, as a fallback. Two traps grow out of this. First, endpoint is not the hostname: for RDS it is the address and port stuck together (orders-db.abc123.us-east-1.rds.amazonaws.com:5432), while address and host are the bare hostname on its own. Point an app at endpoint when it wants a host and the connection fails. Second, these names are one provider's choice, not a standard. A resource whose authors never wrote that friendly mapping hands you only the raw attribute.-prefixed keys, and a different cloud's provider spells the same idea a third way. Hardcode a key your template guessed at, the provider publishes a different one, the value comes back empty, and your app boots with a blank host. That mismatch is the most common reason a connection secret ships half-filled.

Renaming keys in the Composition

A Composition (the template that says how one high-level request becomes real cloud resources) is where you translate. Each composed resource carries a connectionDetails block that does two jobs at once: it picks which of the provider's keys to pull out, and it renames them to the stable words your platform promises. Callers always see host and username, whether AWS, Google Cloud, or Azure built the thing underneath.

# composition.yaml: each composed resource selects the keys it contributes
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xpostgres.aws
spec:
  writeConnectionSecretsToNamespace: crossplane-system   # where the raw MR secrets land
  compositeTypeRef:
    apiVersion: database.example.org/v1alpha1
    kind: XPostgreSQLInstance
  resources:
    - name: rdsinstance
      base:
        apiVersion: rds.aws.upbound.io/v1beta1
        kind: Instance
        spec:
          forProvider:
            engine: postgres
            instanceClass: db.t3.micro
      connectionDetails:
        - name: host                              # stable, platform-facing name
          type: FromConnectionSecretKey
          fromConnectionSecretKey: address        # the bare hostname, not the address:port endpoint
        - name: username
          type: FromConnectionSecretKey
          fromConnectionSecretKey: username
        - name: password
          type: FromConnectionSecretKey
          fromConnectionSecretKey: password
        - name: port
          type: FromValue                         # a literal, not read from the cloud
          value: "5432"

The composite resource (the single high-level object, an XR for short, that stands in for the whole stack of cloud pieces) gathers the union of every composed resource's connectionDetails. Add a cache and a bucket to this Composition and their keys pile into the same connection secret. The port here uses FromValue because Postgres always listens on 5432; the provider does publish a port key too, so you could read it with FromConnectionSecretKey instead, but pinning the literal makes the contract explicit. FromConnectionSecretKey pulls from a resource's own connection secret; FromFieldPath copies a value straight off the resource's fields, say status.atProvider.endpoint. Newer Crossplane runs this through a patch-and-transform function instead of inline resources, and the connectionDetails block reads the same either way.

The allow-list: XRD and the claim

Now the boundary, and this is where the security lives. The raw per-resource secrets over in crossplane-system hold everything each provider published: the master username, the generated password, and sometimes keys you never want a developer near, like an internal replication credential. Picture the XRD (Composite Resource Definition, the file that defines your custom API and what a developer is allowed to ask for) as the guest list on the door. Its connectionSecretKeys field is an allow-list, not a pass-through. Crossplane copies only the keys named on that list into the composite's combined secret and onward into the caller's namespace. Anything left off stays behind in the raw crossplane-system secret, where only cluster admins reach it, and never lands beside the app.

# xrd.yaml: only these keys are allowed to reach the combined secret and the claim
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xpostgresqlinstances.database.example.org
spec:
  group: database.example.org
  names:
    kind: XPostgreSQLInstance
    plural: xpostgresqlinstances
  claimNames:
    kind: PostgreSQLInstance
    plural: postgresqlinstances
  connectionSecretKeys:      # the allow-list; anything not here never leaves crossplane-system
    - host
    - port
    - username
    - password
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object       # spec.parameters schema omitted for brevity

A claim (the namespaced request a developer actually writes, a PostgreSQLInstance here) asks for one of these by setting writeConnectionSecretToRef with only a name. No namespace, because the claim already lives in a namespace and Crossplane drops the Secret right next to it. The app wires that Secret in with envFrom (a container setting in a Deployment that turns every key in a Secret into an environment variable) and learns nothing about RDS, subnet groups, or how the password was born.

# claim.yaml: namespaced request; writeConnectionSecretToRef takes a name only
apiVersion: database.example.org/v1alpha1
kind: PostgreSQLInstance
metadata:
  name: orders-db
  namespace: orders
spec:
  parameters:
    storageGB: 20
  writeConnectionSecretToRef:
    name: orders-db-conn
---
# deployment.yaml: the app consumes the Secret as environment variables
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
  namespace: orders
spec:
  template:
    spec:
      containers:
        - name: api
          image: registry.example.org/orders-api:1.7.3
          envFrom:
            - secretRef:
                name: orders-db-conn    # host, port, username, password become env vars

Reading what you shipped

Trust nothing; check it. Once the claim reports ready, look at the Secret it produced and confirm it holds the keys you expect, spelled the way you promised.

kubectl -n orders get secret orders-db-conn -o json | jq -r '.data | keys[]'

host
password
port
username

Four keys, platform names, good. Now the uncomfortable part. A Kubernetes Secret is base64-encoded, not encrypted. base64 is an encoding (a reversible way to write binary data as plain text), not a lock. Anyone who can read the Secret can read the password, and it takes one command.

kubectl -n orders get secret orders-db-conn -o jsonpath='{.data.password}' | base64 -d; echo

xZ9qB7-Lm2wPk4Rt6Vn0Ss1

There it is in the clear. That is the first thing an attacker who lands a foothold in the namespace runs. So the defender's question is who holds a key to this room. Kubernetes RBAC (Role-Based Access Control, the permission system that decides who can do what to which objects) answers it for any identity, and kubectl auth can-i lets you ask on someone else's behalf.

kubectl auth can-i get secrets -n orders \
  --as=system:serviceaccount:orders:analytics-runner

yes

That yes is a finding. An unrelated batch job in the orders namespace can read the production database password. The fix is a tight Role that grants get on only the named Secrets a workload actually needs (RBAC supports resourceNames for exactly this), bound to only that workload's service account. Then turn on encryption at rest, so a stolen disk or a backup doesn't hand over every credential in plaintext.

Getting the secret out of the cluster

etcd (the key-value store where Kubernetes keeps every object, Secrets included) writes Secret data to disk as base64 by default. Turn on encryption at rest and the API server encrypts that data before it ever reaches etcd, like sealing the valuables in a safe instead of leaving them in a labeled drawer. You point the API server at an EncryptionConfiguration file.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:                       # encrypts new writes with this key
          keys:
            - name: key1
              secret: <base64-encoded 32-byte key>
      - identity: {}                  # fallback so existing plaintext still reads

Start the API server with --encryption-provider-config pointing at that file. One catch: switching it on only encrypts new writes. Secrets already in etcd stay plaintext until you rewrite them, which you force with kubectl get secrets -A -o json | kubectl replace -f -. For production, a KMS provider (Key Management Service, where a cloud service or a hardware security module holds the master key so it never sits on the node) beats a static key sitting in a file on the control-plane host.

Encryption at rest still leaves the credential living inside the cluster. To push it into a dedicated secret manager, the External Secrets Operator (ESO, a controller that syncs secrets between Kubernetes and outside stores) has a PushSecret resource that copies a Crossplane-written Secret out to Vault (HashiCorp's secret-management system) or a cloud store. ESO's usual direction is the opposite, pulling external secrets down into the cluster; PushSecret runs it in reverse.

# pushsecret.yaml: copy the Crossplane connection Secret into Vault
apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: orders-db-to-vault
  namespace: orders
spec:
  selector:
    secret:
      name: orders-db-conn          # the Crossplane-written Secret
  secretStoreRefs:
    - name: vault-backend
      kind: SecretStore
  data:
    - match:
        secretKey: password
        remoteRef:
          remoteKey: orders/db
          property: password

Crossplane also ships its own alpha feature, External Secret Stores, where a resource sets publishConnectionDetailsTo pointing at a StoreConfig and Crossplane writes connection details straight into an external store with no cluster Secret in between. It works only where that feature flag is turned on, and the maintainers now steer new setups toward ESO, so treat it as legacy unless you already run it.

Nothing exists until Ready

The Secret isn't there the instant you apply the claim. Crossplane fills it only after the underlying resource actually reaches Ready in the cloud, and for a database that is often several minutes. Gate your rollout on that fact instead of assuming the credential is already present.

kubectl -n orders wait --for=condition=Ready \
  postgresqlinstance/orders-db --timeout=10m

postgresqlinstance.database.example.org/orders-db condition met

There's a subtlety in how the app reads the Secret, too. envFrom copies the values into environment variables once, at container start. If the credential rotates later, or Crossplane recreates the Secret, the running Pod keeps the old values until it restarts. A mounted Secret volume does refresh on disk after a short delay, but a process that read the file only at boot still holds the stale value in memory. So restart the app when the credential changes, and gate the first rollout on readiness with kubectl wait or an initContainer that blocks until the Secret has data.

Try this

Run kubectl -n orders get secret orders-db-conn -o json | jq -r '.data | keys[]' on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: an empty Secret is worse than a missing one. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related