>_SecOpsLog_
CoursesKubernetes administrationEvents & the event stream

Events & the event stream

What the cluster is telling you, in order.

Intermediate8 min · lesson 54 of 65
In plain terms
Events are the cluster narrating what just happened — “couldn’t seat this pod,” “image failed to load.” It’s spoken aloud but not written down for long, so catch it while it’s still fresh.

A Pod sits in Pending for ten minutes and nothing you look at explains it. The Deployment is healthy. The image name is spelled right. Then you run kubectl describe on the Pod, scroll to the very bottom, and one line settles it: 0/6 nodes are available: 6 Insufficient memory. The cluster knew all along. It had been telling you through Events the whole time, and you weren't reading them. (A Pod is the smallest thing Kubernetes runs, one or more containers scheduled together as a unit.)

A dispatcher's radio channel runs all shift. Units call in what they're doing and what went wrong. Can't reach the address. Engine won't turn over. Handing this one to unit four. It's spoken out loud, useful in the moment, and nobody keeps a recording. Kubernetes runs on the same idea. As each component does its work, it narrates. The scheduler, the part that picks which machine a Pod runs on, says why it couldn't place one. The kubelet, the agent sitting on every worker node, reports that an image failed to pull or a health probe came back sick. Controllers call out scale-ups and rollouts. Every one of those announcements is an Event.

An Event is a real object, not a log line

This is the part people skip past. An Event isn't text buried in a file somewhere. It's a first-class API object, kind: Event, stored in etcd (the cluster's key-value database) right alongside your Pods and Services, and it lives inside a namespace. Components don't print events. They POST them to the API server through a small helper called an EventRecorder. That's the whole reason kubectl get events works: you're listing stored objects, exactly the way you'd list Pods. You can list them, watch them, select them by field, even pipe them through jq, because underneath they really are just objects. Every Event carries the same handful of fields, and once you know them the output stops looking like noise.

terminal
$ kubectl get event -n prod checkout-7d9c8-x2fjq.17f0a9b2c3d4e5f6 -o yaml
apiVersion: v1
kind: Event
metadata:
  name: checkout-7d9c8-x2fjq.17f0a9b2c3d4e5f6
  namespace: prod
type: Warning
reason: FailedScheduling
message: '0/6 nodes are available: 6 Insufficient memory. preemption: 0/6 nodes are available.'
involvedObject:
  kind: Pod
  name: checkout-7d9c8-x2fjq
  namespace: prod
source:
  component: default-scheduler
count: 9
firstTimestamp: '2026-07-16T09:14:02Z'
lastTimestamp: '2026-07-16T09:31:40Z'

Look at count: 9. The scheduler did not create nine Event objects. It created one. Every time the same failure repeated, the EventRecorder found that existing Event (same object, same reason, same message) and bumped the count while moving lastTimestamp forward. That's why describe prints it as (x9 over 18m). The folding is deliberate, and it happens before anything reaches storage. A crash-looping Pod could otherwise fire an event every second and bury etcd, so the same recorder machinery also rate-limits how many events it will send for one object. Worth keeping in mind during an incident: under a real storm the recorder drops some events instead of sending them all, so an empty result doesn't always mean nothing happened.

If you go digging you'll find events actually live under two API groups. There's the original core/v1 Event, and a newer events.k8s.io/v1 that went stable in v1.19, built to aggregate high-frequency events more cheaply (it groups repeats into a series instead of one rolling count). kubectl bridges both, so kubectl get events and kubectl events show you everything regardless of which group produced it. You rarely pick a group by hand. It matters mostly when you write a controller that emits its own events.

Reading the stream three ways

describe is the fast path when you already know the object. It pulls that object's recent events and prints them at the bottom, which is where nine out of ten 'why is this stuck' answers live. When you don't know the object yet, or several things are failing at once, you want the whole namespace's stream, sorted so the sequence makes sense. And recent kubectl versions ship a dedicated command, kubectl events, built for exactly this instead of bending kubectl get to the job.

terminal
$ kubectl get events -n prod --sort-by=.lastTimestamp
LAST SEEN   TYPE      REASON             OBJECT                      MESSAGE
6m          Normal    Scheduled          pod/checkout-7d9c8-k4p2j    Successfully assigned prod/checkout-7d9c8-k4p2j to node-2
5m          Normal    Pulled             pod/checkout-7d9c8-k4p2j    Container image "reg/checkout:1.8" already present on machine
90s         Warning   FailedScheduling   pod/checkout-7d9c8-x2fjq    0/6 nodes are available: 6 Insufficient memory.
40s         Warning   BackOff            pod/payments-api-77-9mnpq   Back-off restarting failed container
 
$ kubectl get events -n prod --field-selector type=Warning,reason=FailedScheduling
LAST SEEN   TYPE      REASON             OBJECT                      MESSAGE
90s         Warning   FailedScheduling   pod/checkout-7d9c8-x2fjq    0/6 nodes are available: 6 Insufficient memory.

The reason field is the real signal, and a small vocabulary covers most of what you'll meet. FailedScheduling means no node fit: not enough resources, or a taint or an affinity rule ruled every node out. Failed usually means the kubelet couldn't pull the image (a bad tag, a private registry, a missing pull secret), and you'll watch the Pod's status flip from ErrImagePull to ImagePullBackOff as it keeps retrying. Unhealthy means a liveness or readiness probe came back bad. BackOff means the container keeps crashing and the kubelet is now waiting longer between restarts. FailedMount means a volume wouldn't mount. Each word points straight at one layer, so it usually tells you where to look next.

terminal
$ kubectl events --for pod/checkout-7d9c8-x2fjq -n prod --types=Warning
LAST SEEN           TYPE      REASON             OBJECT                     MESSAGE
2m (x9 over 18m)    Warning   FailedScheduling   Pod/checkout-7d9c8-x2fjq   0/6 nodes are available: 6 Insufficient memory.
 
# after you raise the memory limit and the Pod finally schedules, watch it clear:
$ kubectl events --for pod/checkout-7d9c8-x2fjq -n prod --watch
LAST SEEN   TYPE     REASON      OBJECT                     MESSAGE
0s          Normal   Scheduled   Pod/checkout-7d9c8-x2fjq   Successfully assigned prod/checkout-7d9c8-x2fjq to node-4
0s          Normal   Pulled      Pod/checkout-7d9c8-x2fjq   Container image "reg/checkout:1.8" already present on machine
0s          Normal   Started     Pod/checkout-7d9c8-x2fjq   Started container checkout

Events expire, so capture them

One thing the output won't warn you about: events don't last. The API server expires them on a timer set by --event-ttl, one hour by default, so the events explaining this morning's outage may be gone by lunch. This isn't the normal garbage collector doing it. The API server prunes events itself. Two consequences follow. Read them while the problem is fresh, not tomorrow. And if you need history for a post-incident review, ship events off the cluster (a tool like eventrouter or the Kubernetes event-exporter that streams them to your logging stack) so the 'why' survives past the hour.

How a FailedScheduling event reaches you
1Scheduler triesno node fits the Pod2EventRecorderdedups, rate-limits, builds…3API servervalidates and admits it4etcdstored, event-ttl 1h5kubectl describeyou read it back
Same path for image pulls, probe failures, and evictions. The recorder never touches etcd directly; it POSTs to the API server like everything else in the cluster.
Node and volume events hide in the 'default' namespace
An Event lives in its involved object's namespace. But Nodes, PersistentVolumes, and Namespaces are cluster-scoped, so they have no namespace of their own, and their events get created in default. A node went NotReady and kubectl get events -n kube-system shows nothing? You're looking in the wrong place. Use kubectl get events -n default --field-selector involvedObject.kind=Node, or just kubectl describe node <name>, which pulls them for you.

Normal versus Warning is a hint, not a severity system. A flood of Normal Scheduled can still hide one Warning FailedScheduling.

Events are not an audit log. Prefer API audit for who did what.

Spammy controllers can drown useful events. Filter by involvedObject when the stream is noisy.

Try this

Sort events by lastTimestamp in a namespace with a failing pod. Read the newest warnings and map them to scheduling, pulling, or probing.

terminal
$ kubectl get event -n prod checkout-7d9c8-x2fjq.17f0a9b2c3d4e5f6 -o yaml
apiVersion: v1
kind: Event
metadata:
  name: checkout-7d9c8-x2fjq.17f0a9b2c3d4e5f6
  namespace: prod
type: Warning
reason: FailedScheduling
message: '0/6 nodes are available: 6 Insufficient memory. preemption: 0/6 nodes are available.'
involvedObject:
  kind: Pod
  name: checkout-7d9c8-x2fjq
  namespace: prod
source:
  component: default-scheduler
count: 9
firstTimestamp: '2026-07-16T09:14:02Z'
lastTimestamp: '2026-07-16T09:31:40Z'
 
$ kubectl get events -n prod --sort-by=.lastTimestamp
LAST SEEN   TYPE      REASON             OBJECT                      MESSAGE
6m          Normal    Scheduled          pod/checkout-7d9c8-k4p2j    Successfully assigned prod/checkout-7d9c8-k4p2j to node-2
5m          Normal    Pulled             pod/checkout-7d9c8-k4p2j    Container image "reg/checkout:1.8" already present on machine
90s         Warning   FailedScheduling   pod/checkout-7d9c8-x2fjq    0/6 nodes are available: 6 Insufficient memory.
40s         Warning   BackOff            pod/payments-api-77-9mnpq   Back-off restarting failed container
 
$ kubectl get events -n prod --field-selector type=Warning,reason=FailedScheduling
LAST SEEN   TYPE      REASON             OBJECT                      MESSAGE
90s         Warning   FailedScheduling   pod/checkout-7d9c8-x2fjq    0/6 nodes are available: 6 Insufficient memory.
 
$ kubectl events --for pod/checkout-7d9c8-x2fjq -n prod --types=Warning
LAST SEEN           TYPE      REASON             OBJECT                     MESSAGE
2m (x9 over 18m)    Warning   FailedScheduling   Pod/checkout-7d9c8-x2fjq   0/6 nodes are available: 6 Insufficient memory.
 
# after you raise the memory limit and the Pod finally schedules, watch it clear:
$ kubectl events --for pod/checkout-7d9c8-x2fjq -n prod --watch
LAST SEEN   TYPE     REASON      OBJECT                     MESSAGE
0s          Normal   Scheduled   Pod/checkout-7d9c8-x2fjq   Successfully assigned prod/checkout-7d9c8-x2fjq to node-4
0s          Normal   Pulled      Pod/checkout-7d9c8-x2fjq   Container image "reg/checkout:1.8" already present on machine
0s          Normal   Started     Pod/checkout-7d9c8-x2fjq   Started container checkout

Takeaway

Events are the cluster narration. They expire. Read them while debugging, not a day later.

Quick check
01kubectl describe pod shows: Warning BackOff (x847 over 3h) Back-off restarting container. Why is there a single Event with a count of 847 instead of 847 separate Event objects?
Incorrect — No. It reports each restart; the recorder is what folds identical ones together.
Correct — Same object, reason, and message get deduplicated into one Event with a rolling count and lastTimestamp.
Incorrect — No. The recorder deduplicates before anything is written, so etcd holds one Event carrying count 847. kubectl just prints that stored number.
Incorrect — No. The count field is a real, intended part of the Event object.
02You go to pull the Events that explain an outage from three hours ago, and kubectl get events for the namespace shows nothing about it. Assuming defaults, why?
Correct — Events are short-lived and the API server itself removes them after an hour, which is why post-incident review needs an exporter.
Incorrect — Events do expire on a timer, so their absence here is age, not silence.
Incorrect — expired Events are deleted, not relocated to another namespace.
Incorrect — it isn't the normal GC; the API server prunes Events on its own TTL regardless of the pod.
03A node just went NotReady. kubectl get events -n kube-system shows nothing about it. Where are that node's Events, and why?
Incorrect — Nodes do generate Events; they just don't land where you'd expect.
Incorrect — Nodes aren't namespaced, and there is no per-node namespace for their Events.
Incorrect — kube-node-lease holds Lease objects, not the node's Events.
Correct — an Event lives in its involved object's namespace, and cluster-scoped objects like Nodes and PersistentVolumes fall back to default.
Application logsDebugging running pods

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep