>_SecOpsLog_
CoursesCrossplaneReconciliation & state without a state file

Reconciliation & state without a state file

The control loop is the source of truth.

Intermediate12 min · lesson 4 of 12

If you came up on Terraform, you learned to guard the state file like the deed to your house. You lock it. You back it up. You never let two people run apply at the same time. That file is the only record Terraform keeps of what it actually built, so if it goes missing or gets corrupted, Terraform loses the thread. It will cheerfully build a second copy of everything you own, or tear down the wrong resource, because its map no longer matches the ground. Crossplane keeps no such file. There is nothing to lock, nothing to corrupt, nothing to leak. That sounds reckless right up until you see what stands in its place.

A thermostat keeps no diary. You set it to 21 degrees and walk off. It does not remember last night's temperature or how many times the heater kicked on. It reads the room, compares that number to your setpoint, and nudges. Again and again, forever. Crossplane runs the same way. Your setpoint is the spec of a Kubernetes object you applied. The live reading is whatever the cloud's control API (application programming interface, the machine-to-machine control surface a cloud exposes) reports at this second. A controller (a small program running a loop) sits between them and closes the gap. Those two values, your setpoint and the fresh reading, are the only things that are ever true. No third file sits in the middle to drift out of sync with reality, because there is no third file.

Where the state actually lives

Break 'state' into two halves and the mystery goes away. The desired state is the spec of a managed resource, Crossplane's word for one cloud thing: a single S3 bucket (S3 is Amazon's Simple Storage Service, its object storage), one database, one IAM role (IAM, Identity and Access Management, is the cloud's permission system). That spec is an ordinary object living in etcd, the key-value store behind every Kubernetes cluster. The observed state is pulled fresh from the cloud API on every pass of the loop and written into that same object's status. Nothing on disk is trusted as the truth. If you want to know what Crossplane thinks exists, you ask the cluster, and the cluster asks the cloud.

Because both halves are plain Kubernetes objects, kubectl (the Kubernetes command-line tool) is your entire inspection kit. No state pull, no remote backend, no import ritual. Start with a live inventory of every managed resource across every installed provider (a provider is the plugin that teaches Crossplane how to talk to one cloud, like AWS (Amazon Web Services) or Google Cloud).

terminal
kubectl get managed
output
NAME                                       READY   SYNCED   EXTERNAL-NAME     AGE
bucket.s3.aws.upbound.io/prod-app-assets   True    True     prod-app-assets   18m
 
NAME                                  READY   SYNCED   EXTERNAL-NAME   AGE
user.iam.aws.upbound.io/ci-deployer   True    True     ci-deployer     18m
 
NAME                                    READY   SYNCED   EXTERNAL-NAME   AGE
instance.rds.aws.upbound.io/orders-db   False   True     orders-db       3m

Two columns tell most of the story, and we will come back to them: READY and SYNCED. Look at the RDS (Relational Database Service, Amazon's managed database) instance sitting at SYNCED=True, READY=False. The create call was accepted; the database is still booting. Look at EXTERNAL-NAME too. That is the resource's real identity in the cloud, kept in plain sight as an annotation (a labeled note stuck on the Kubernetes object), and it is the join key a defender uses to tie 'this object in my cluster' to 'that resource in the AWS account' during an investigation.

Pull one object in full and you can read both halves at once. Use the fully qualified name, bucket.s3.aws.upbound.io, so a Bucket kind from a different provider cannot collide with this one.

terminal
kubectl get bucket.s3.aws.upbound.io prod-app-assets -o yaml
output
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
  annotations:
    crossplane.io/external-create-pending: "2026-07-21T09:02:10Z"
    crossplane.io/external-create-succeeded: "2026-07-21T09:02:12Z"
    crossplane.io/external-name: prod-app-assets
  name: prod-app-assets
spec:
  deletionPolicy: Delete
  forProvider:            # desired state: what you asked for
    region: us-east-1
    tags:
      env: prod
  managementPolicies:
  - '*'
  providerConfigRef:
    name: default
status:
  atProvider:            # observed state: refreshed live each loop
    arn: arn:aws:s3:::prod-app-assets
    id: prod-app-assets
    region: us-east-1
  conditions:
  - lastTransitionTime: "2026-07-21T09:02:13Z"
    reason: Available
    status: "True"
    type: Ready
  - lastTransitionTime: "2026-07-21T09:02:12Z"
    reason: ReconcileSuccess
    status: "True"
    type: Synced

Read the annotations, because that is where the closest thing to a state file hides. crossplane.io/external-name is the resource's identity in the cloud (for a bucket, its name; for an EC2 (Elastic Compute Cloud, Amazon's virtual machines) instance, the i-0abc... ID the cloud generates). The external-create-pending and external-create-succeeded pair solves one genuinely dangerous problem: if the controller calls Create and then crashes before it can record the ID the cloud handed back, how does it avoid making a second copy on restart? It writes 'pending' before the call and 'succeeded' after. A resource that is pending but not succeeded, and cannot yet be observed, makes the controller stop and wait rather than create again. That safety record lives on the object, out in the open, not in a file you have to guard with your life.

That trade has teeth. A Terraform state file does four jobs at once: it maps your config to real resource IDs, caches each resource's last-known attributes, locks so two runs cannot collide, and remembers dependency order. Crossplane splits those jobs up and hands each to something that cannot quietly rot. The ID map becomes the external-name annotation on each object. The attribute cache becomes status, refreshed from the live API instead of trusted as gospel. Locking becomes Kubernetes' own optimistic concurrency: a resourceVersion the API server bumps on every write, so two writers cannot both win. Dependency order becomes cross-references plus patient retry, the loop requeuing until the thing it depends on finally exists. No single file holds all four, so no single file can take all four down.

Reading the loop's verdict

Every managed resource wears two status lights, like a dishwasher with a 'running' light and a 'clean' light. Synced is the running light: the controller reached the provider API and your request was accepted, so the reconcile (one full pass of the loop) itself worked. Ready is the clean light: the external resource actually exists and is usable. They move independently. A fresh database sits at Synced=True, Ready=False for several minutes while it provisions, and that is healthy, not a fault.

kubectl describe shows the conditions plus recent events, and the events are where operations and security meet.

terminal
kubectl describe bucket.s3.aws.upbound.io prod-app-assets
output
Name:         prod-app-assets
Namespace:    
Labels:       <none>
Annotations:  crossplane.io/external-create-pending: 2026-07-21T09:02:10Z
              crossplane.io/external-create-succeeded: 2026-07-21T09:02:12Z
              crossplane.io/external-name: prod-app-assets
API Version:  s3.aws.upbound.io/v1beta1
Kind:         Bucket
Metadata:
  Creation Timestamp:  2026-07-21T09:02:09Z
  Finalizers:
    finalizer.managedresource.crossplane.io
  Generation:        2
  Resource Version:  58213
Spec:
  Deletion Policy:  Delete
  For Provider:
    Region:  us-east-1
    Tags:
      Env:  prod
  Management Policies:
    *
Status:
  At Provider:
    Arn:     arn:aws:s3:::prod-app-assets
    Id:      prod-app-assets
    Region:  us-east-1
  Conditions:
    Last Transition Time:  2026-07-21T09:02:13Z
    Reason:                Available
    Status:                True
    Type:                  Ready
    Last Transition Time:  2026-07-21T09:02:12Z
    Reason:                ReconcileSuccess
    Status:                True
    Type:                  Synced
Events:
  Type    Reason                   Age   From                              Message
  ----    ------                   ----  ----                              -------
  Normal  CreatedExternalResource  18m   managed/bucket.s3.aws.upbound.io  Successfully requested creation of external resource
  Normal  UpdatedExternalResource  2m    managed/bucket.s3.aws.upbound.io  Successfully requested update of external resource

That second event tells a story. UpdatedExternalResource, two minutes ago, on a bucket eighteen minutes old that nobody redeployed. That is the loop correcting drift: something changed the bucket out of band, and the next pass put it back to match the spec. When the provider returns an error, its text lands verbatim on the Synced condition's message field, so the cloud's own complaint shows up right on the object. A stream of UpdatedExternalResource events on a resource no one is deploying is a real signal worth chasing: either a second tool is fighting Crossplane, or a person keeps poking the console.

For scripts and health checks, read the two conditions straight off the object.

terminal
kubectl get bucket.s3.aws.upbound.io prod-app-assets -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
output
Ready=True Available
Synced=True ReconcileSuccess
The reconcile loop is the source of truth
1Observe
read the live resource from the cloud API
2Compare
desired spec against observed status
3Act
create, update, or delete to close the gap
4Record
write conditions and events, refresh status
5Wait, then repeat
a spec change or the poll timer wakes it again
No apply step and no state file: the loop wakes on every spec change and on the provider's poll timer, then runs this cycle forever.
Synced=True is not done
The most common misread of the loop is treating Synced=True as finished. Synced only means the controller reached the provider and the API accepted the call; the resource may still be provisioning, failing a health check, or blocked on a dependency, which is exactly what Ready tracks. Gate automation and kubectl wait on condition=Ready, never on Synced. The flip side: a resource stuck at Synced=False with reason ReconcileError was rejected somewhere in the loop. Read the message field on that condition before you change anything, because it carries the provider's verbatim reason. It is often a bad ProviderConfig (wrong credentials or missing permissions), but at least as often the spec itself: an invalid value, a change to an immutable field, or a reference to a resource that does not exist yet.

Drift correction is a security control, and a foot-gun

Because the loop never stops, out-of-band changes are temporary by construction. An attacker who flips something you declared (opens a bucket, widens a security group, loosens an IAM policy) is fighting a program that re-reads the real resource on a timer and reverts anything that does not match your spec. Your declared state is the recovery baseline, and the blast radius of a console change is capped by how often the loop runs. For the Upbound AWS provider family, the usual choice for AWS on Crossplane, that timer is the --poll flag, ten minutes by default, on top of an immediate reconcile every time the spec changes.

Now the sharp edge, and it catches people mid-incident. You reach into the console to contain something fast: revoke a rule, lock a bucket, kill a session. Crossplane reads your emergency fix as drift and undoes it on the next pass, restoring the exact state the attacker was enjoying. Before you change Crossplane-managed infrastructure by hand, pause the loop for that resource.

terminal
kubectl annotate bucket.s3.aws.upbound.io prod-app-assets crossplane.io/paused=true
output
bucket.s3.aws.upbound.io/prod-app-assets annotated

Paused, the controller stops reconciling that one object and reports ReconcilePaused on its Synced condition. Your manual change now holds. Once the incident is closed and you have fixed the real spec in Git (your version-controlled source of desired state), remove the annotation and the loop picks back up. Treat pausing as the switch you flip before any manual surgery, never after you have already been overwritten.

Telling the loop what it may touch

By default the controller carries a full keyring: it may observe, create, update, delete, and back-fill blanks from the cloud. Sometimes that is more authority than you want it holding. managementPolicies is how you take keys off the ring. It is a list of the actions the loop is allowed to perform, and the default is ['*'], every action.

The actions are Observe (read the live resource), Create, Update, Delete, and LateInitialize (copy cloud-set defaults back into your spec so they stop looking like drift). Drop Delete from the list and the loop becomes structurally unable to destroy the real resource. Even kubectl delete on the object leaves the cloud resource standing, orphaned on purpose (the same outcome as deletionPolicy: Orphan, but enforced by removing the loop's ability to delete at all). For a production database you never want a stray delete to reach, that is a guardrail with real teeth.

bucket-observe-only.yaml
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
metadata:
  name: legacy-audit-logs
  annotations:
    # Point the object at a bucket that already exists in AWS, so the loop
    # adopts it instead of observing nothing and trying to create one.
    crossplane.io/external-name: legacy-audit-logs
spec:
  # Read-only adoption: observe the live resource, write nothing back.
  # Widen to ["Observe","Create","Update"] to manage it but never delete it.
  managementPolicies:
    - Observe
  forProvider:
    region: us-east-1
  providerConfigRef:
    name: default

Narrow the list all the way to ['Observe'] and you get a safe, read-only adoption. The controller fills in status from the live resource and writes nothing back to the cloud, ever. Pair it with the crossplane.io/external-name annotation pointing at a resource that already exists, and Crossplane begins tracking infrastructure it never built, with zero risk of changing it. That is the sane first move when you bring an existing cloud account under management: observe first, confirm status matches what you expect, then hand the loop more keys once you trust it.

After any change to what the loop may do, check the object, not your intentions. Block automation on Ready so a script only proceeds once the resource truly exists.

terminal
kubectl wait --for=condition=Ready bucket.s3.aws.upbound.io/prod-app-assets --timeout=5m
output
bucket.s3.aws.upbound.io/prod-app-assets condition met
Quick check
01Mid-incident, you edit a security group rule directly in the AWS console to lock out an attacker. That rule is declared in a Crossplane managed resource with default settings. Ten minutes later the hole is open again. What happened, and what should you have done first?
Incorrect — etcd holds the desired spec, not a cache of cloud state; observed state is read live from the provider, so there is nothing stale to restart.
Correct — On its poll interval the loop reverts anything that does not match the declared spec, so you pause that resource before making out-of-band changes.
Incorrect — You never used kubectl; the loop runs server-side regardless of your context, and apply would only re-assert the spec that caused the revert.
Incorrect — Ready reports usability and triggers no rollback, and deletionPolicy governs what happens when the managed resource is deleted, not drift correction.
02A managed resource's YAML shows two annotations wrapped around its creation moment: crossplane.io/external-create-pending and crossplane.io/external-create-succeeded. What problem does this pair solve?
Correct — the pair is a crash-safety record, so a resource that is pending but not yet succeeded and not observable makes the controller pause rather than duplicate.
Incorrect — concurrency between writers is handled by the object's resourceVersion, not by these create annotations.
Incorrect — they exist for create-safety, not for billing.
Incorrect — the poll interval is a provider setting, unrelated to these annotations.
03A continuous-integration pipeline applies an Amazon Relational Database Service (RDS) Instance, then runs a smoke test the instant the object reports SYNCED=True. The test fails intermittently because the database is not reachable yet. What is the correct fix?
Incorrect — a blind sleep is fragile and still races provisioning that can take longer or shorter.
Correct — Synced and Ready move independently, so kubectl wait --for=condition=Ready is the right signal that the database exists and works.
Incorrect — SYNCED is not a readiness signal; it only reports that the last reconcile reached the API.
Incorrect — Observe-only stops the controller from creating the database at all, which does not solve a readiness race.

One last thing about pausing, because it catches people in the middle of an incident. A paused object stops being reconciled, and that means its status stops refreshing too. The controller is no longer reading the cloud, so status.atProvider freezes at whatever it saw right before you paused. It will not show the manual change you made in the console, and it will not show what the attacker does next either. While a resource is paused, treat the cloud's own console or CLI as ground truth, not the object in your cluster. The moment you unpause, the loop reads the world fresh and the status starts telling the truth again.

Try this

Run kubectl get managed on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: synced=True is not done. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Managed resourcesComposite resources & compositions

Related

What Infrastructure as Code isInfrastructure as Code & automationWhat Terraform is & how it worksTerraformState files are secretsTerraform security and complianceWhy Terragrunt existsTerragrunt