Labels, selectors & annotations

The glue behind almost every Kubernetes object.

Almost nothing in Kubernetes finds a pod by name. Objects do name each other in plenty of places: a pod mounts a Secret or a ConfigMap by name, and an Ingress names the Service behind it. But the moment something needs a whole group of pods, names drop out. A Service doesn't keep a list of the pods behind it (a pod is the smallest thing Kubernetes runs, usually a single container with its own IP address). A ReplicaSet, the controller that keeps a fixed number of identical pods alive, doesn't track the ones it made by name either. And when a pod needs a particular kind of machine, it asks for one by node label instead of naming a host. Every one of those relationships runs on labels: little key-value stickers you put on objects, plus queries that grab everything wearing a matching sticker. Get the labels wrong and a Service can go dark while every pod behind it stays perfectly healthy. That's why this topic sits underneath almost everything you'll touch as an admin.

Sticky notes, and the search that finds them

A big warehouse runs on colored stickers, not serial numbers. Every box wears a red dot for fragile, a blue dot for this week's orders. Nobody hunts down box number 84213. They shout 'bring me everything with a blue dot,' and the pickers do. A label works the same way. It's a key-value pair on an object's metadata, like app=web or env=prod, and on its own it means nothing. The power is in the selector: a query that says 'give me every object where env=prod.' You rarely name pods yourself. You stick labels on them, then let selectors do the finding.

Selectors come in two flavors. Equality-based is the plain kind: app=web, or tier!=cache. Set-based reads almost like English: env in (prod,staging), or just 'has a release-track label at all.' The -l flag on kubectl speaks both, and you'll reach for it constantly to slice up what's running.

# Add a label to a running pod (needs --overwrite to change an existing one)
kubectl label pod web-6d9f tier=frontend
 
# Equality-based query: both pods carry app=web
kubectl get pods -l app=web
 
# Set-based query: in / notin / a bare key means 'exists'
kubectl get pods -l 'tier in (frontend)'

pod/web-6d9f labeled
 
NAME       READY   STATUS    RESTARTS   AGE
web-6d9f   1/1     Running   0          12m
web-7c2a   1/1     Running   0          12m
 
NAME       READY   STATUS    RESTARTS   AGE
web-6d9f   1/1     Running   0          12m

How a Service actually finds its pods

So who runs these queries? This is where labels stop being a naming convenience and become load-bearing. Take a Service, the stable front door that sits in front of a set of pods. You give it a selector and nothing else. Behind the scenes a controller called the EndpointSlice controller watches every pod in the namespace, keeps re-running that selector, and writes the matching pod IP addresses into EndpointSlice objects. kube-proxy (the small networking agent on each node) reads those slices and programs the routing rules. The Service never learns a single pod name. It knows a query, and a controller keeps the answer up to date as pods come and go.

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:          # equality-only: a flat map of key=value, AND-ed
    app: web
  ports:
    - port: 80
      targetPort: 8080

kubectl get endpointslices -l kubernetes.io/service-name=web
 
NAME        ADDRESSTYPE   PORTS   ENDPOINTS               AGE
web-x7k2p   IPv4          8080    10.244.1.7,10.244.2.4   3m

One quirk worth burning into memory: a Service selector is equality-only. It's that flat map of key=value, with no in or notin. ReplicaSets and Deployments are fancier. (A Deployment is the object you usually create; it manages ReplicaSets for you.) Their selector has matchLabels (equality) and matchExpressions (set-based), and it decides which pods the controller owns and heals. That selector is frozen at creation. Try to change a Deployment's selector later and the API server rejects it as immutable, because quietly rewiring ownership on a live workload is exactly how you'd strand running pods.

Steering pods onto the right nodes

Labels don't only group pods. They also decide where pods run, which is the whole reason this topic lives in the scheduling section. Nodes wear labels too. Some come from the cloud provider (a region, an instance type, whether the disk is SSD), and some you add by hand. A pod can carry a nodeSelector, a short wish list of node labels it needs. When the scheduler (the control-plane component that assigns pods to machines) sees an unscheduled pod, it filters the nodes down to the ones whose labels satisfy that wish list, then picks one. The kubelet (the agent running on that chosen node) then pulls the image and starts the container.

# Give one node a label the scheduler can target
kubectl label node worker-2 disktype=ssd
 
# Apply a pod that names that label in its nodeSelector
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: cache
spec:
  nodeSelector:
    disktype: ssd
  containers:
    - name: redis
      image: redis:7
EOF
 
kubectl get pod cache -o wide

node/worker-2 labeled
pod/cache created
 
NAME    READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
cache   1/1     Running   0          8s    10.244.2.9   worker-2   <none>           <none>

If no node carries disktype=ssd, the pod doesn't error. It sits in Pending, and kubectl describe pod shows a FailedScheduling event reading '0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector.' That line is the scheduler telling you your label math didn't add up. Fix the node label or loosen the selector and the pod schedules on the next pass.

Labels vs annotations

Annotations look identical to labels, same key-value shape, and people mix them up all the time. The difference is intent. Labels are for identity and grouping, the stuff you query. Annotations are for everything else you want to attach but never search on: a note about who changed what, a config checksum, a controller's private bookkeeping. Nothing selects on an annotation, ever. That's the whole rule. If you'll ever run kubectl get -l on it, or point a Service, policy, or the scheduler at it, it's a label. If it's just information riding along, it's an annotation.

Annotations also get big in a way labels can't. A label value is capped at 63 characters; annotations hold structured, sometimes huge blobs. kubectl stashes the entire last-applied manifest under kubectl.kubernetes.io/last-applied-configuration so it can compute diffs on the next apply. An ingress controller reads its whole config from annotations. And the kubernetes.io/change-cause annotation is what fills the CHANGE-CAUSE column when you inspect a rollout. Set it right after the change that triggered the rollout, because the note only creates a history row if a revision already exists: annotating on its own changes nothing about the pod template, so it starts no rollout. The Deployment controller copies the value down onto the ReplicaSet behind the current revision, and that's the row rollout history prints it against.

# The change itself is what creates a new revision
kubectl set image deployment/web nginx=nginx:1.27
 
# Now label that revision with the reason
kubectl annotate deployment web \
  kubernetes.io/change-cause="bump image to nginx:1.27" --overwrite
 
kubectl rollout history deployment web

deployment.apps/web image updated
deployment.apps/web annotated
 
deployment.apps/web
REVISION  CHANGE-CAUSE
1         <none>
2         bump image to nginx:1.27

Stable app and version labels beat clever one-offs. Agree a vocabulary and enforce it in CI.

Selectors are equality or set-based. Learn both so NetworkPolicy and Deployments do not surprise you.

Changing labels on live pods can orphan them from their controller. Prefer rolling new pods with the new labels.

A Service selector is equality-only, so anything a Service must match on has to stay a plain key=value pair.

Try this

Label a pod, select it with kubectl, and patch an annotation. Then break a Service selector on purpose and watch endpoints go empty.

# Label a running pod, then find it with each selector style
$ kubectl label pod web-6d9f tier=frontend --overwrite
pod/web-6d9f labeled
 
$ kubectl get pods -l app=web
NAME       READY   STATUS    RESTARTS   AGE
web-6d9f   1/1     Running   0          12m
web-7c2a   1/1     Running   0          12m
 
$ kubectl get pods -l 'tier in (frontend)'
NAME       READY   STATUS    RESTARTS   AGE
web-6d9f   1/1     Running   0          12m
 
# Attach an annotation. Nothing will ever select on it
$ kubectl annotate pod web-6d9f owner=platform-team --overwrite
pod/web-6d9f annotated
 
$ kubectl get pod web-6d9f -o jsonpath='{.metadata.annotations.owner}'
platform-team
 
# Who is the Service routing to right now?
$ kubectl get endpointslices -l kubernetes.io/service-name=web
NAME        ADDRESSTYPE   PORTS   ENDPOINTS               AGE
web-x7k2p   IPv4          8080    10.244.1.7,10.244.2.4   3m
 
# Break the selector on purpose, then look again
$ kubectl patch svc web -p '{"spec":{"selector":{"app":"web-typo"}}}'
service/web patched
 
$ kubectl get endpointslices -l kubernetes.io/service-name=web
NAME        ADDRESSTYPE   PORTS   ENDPOINTS   AGE
web-x7k2p   IPv4          8080    <unset>     4m
 
# Both pods are still Running. Put the selector back and they return
$ kubectl patch svc web -p '{"spec":{"selector":{"app":"web"}}}'
service/web patched

Takeaway

Labels are for selection; annotations are for notes tools read. Almost every controller relationship is a label query.

Related