The API server

The only front door, and the lifecycle of one request.

Everything in a Kubernetes cluster goes through one process. Every kubectl command you type, every node checking in, every controller quietly doing its job, all of it lands on the kube-apiserver first (Kubernetes is the system that runs your containers across a fleet of machines, and the API server is its front desk). Think of a secured building with exactly one staffed reception desk. You show ID, you're told which floors you can visit, someone checks your paperwork, and only then does anything get filed in the vault. The vault here is etcd, the cluster's key-value database that holds every object. The API server is the only component allowed to open it. Nothing else touches etcd directly. That single fact explains most of how the cluster stays secure and most of where your requests get rejected.

Here's the mental model that makes the rest click. Every object in the cluster is a thing at a web address. A Pod (the smallest unit you deploy, one or more containers that run together) lives at a URL like /api/v1/namespaces/prod/pods/web. A Deployment lives under /apis/apps/v1/. When you run kubectl get pods, kubectl isn't doing anything magic. It's an HTTPS client sending a GET to that address and printing the JSON that comes back as a table. A create is a POST, a delete is a DELETE. This is also how the API server grows new tricks: a CustomResourceDefinition (CRD) teaches it a brand-new address for a brand-new kind of object, and suddenly kubectl get works on things Kubernetes never shipped with.

What one request actually goes through

A request doesn't just arrive and get saved. It walks a line of gates, and any gate can turn it away. First, authentication: who are you? You prove it with a client certificate, a bearer token, or OIDC (OpenID Connect, the same sign-in standard many websites use). The API server doesn't store passwords, it just verifies the credential you present, and if it can't, you're stopped right there with a 401. Second, authorization: you're inside the building, but may you do this exact thing? The answer comes from RBAC (Role-Based Access Control), rules that map identities to the verbs (get, create, delete) they're allowed on each resource. No matching rule means a 403. Third, admission, the step people forget. There's a clerk at this stage who takes your filled-out form, quietly corrects a couple of fields, checks it against the rulebook, and only then files it. Admission runs twice. Mutating admission can change your object first (this is how a sidecar container gets injected, or a default value filled in). The object is then checked against its schema. Finally validating admission casts a yes-or-no vote. Only when every gate agrees does the API server write the object to etcd.

This pipeline is where cluster policy actually lives, which is why admins spend so much time in it. RBAC at the authorization gate decides who can do what. The admission gate enforces the rules about the objects themselves: this namespace can't run privileged containers, every Pod must set resource limits, images must come from our own registry. Some of that is built-in controllers, the rest is admission webhooks (Kyverno and Gatekeeper are the common ones). If you've ever wondered where a cluster says no to a bad workload, it's almost always one of these two gates.

So when a command fails, read the error instead of guessing. A 401 is authentication: a bad or expired certificate or token. A 403 is where people go wrong, because two different gates return it. An RBAC denial names the user, the verb and the resource. A webhook denial, including anything Kyverno or Gatekeeper rejects, comes back as a 403 too, but the message says admission webhook "..." denied the request. So read the words, not just the number: if it names a webhook or a policy, hunting for a missing RoleBinding will waste your afternoon. A malformed object is a third case, usually a 400 or a 422 that points at the offending field. And you can ask the authorization gate a question directly, without attempting the write at all, which is the fastest way to prove or rule out an RBAC problem.

$ kubectl apply -f deploy.yaml           # a 403 from the authorization gate
Error from server (Forbidden): error when creating "deploy.yaml": deployments.apps is forbidden: User "dev" cannot create resource "deployments" in API group "apps" in the namespace "prod"
 
$ kubectl apply -f pod.yaml              # also a 403, but from the admission gate
Error from server (Forbidden): error when creating "pod.yaml": admission webhook "validate.example.com" denied the request: privileged containers are not allowed in prod
 
$ kubectl auth can-i create deployments --namespace prod
no
 
$ kubectl auth can-i --list --namespace prod
Resources                                       Non-Resource URLs   Resource Names   Verbs
selfsubjectaccessreviews.authorization.k8s.io   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                  []               [create]
pods                                            []                  []               [get list watch]
pods/log                                        []                  []               [get]

Why nothing polls it in a loop

Nothing in Kubernetes sits in a loop asking the API server "anything new?". That would melt it. Instead, components subscribe. The scheduler, every controller, and the kubelet (the agent running on each node) open a long-lived watch connection and get a message streamed to them the instant something they care about changes. That's the engine of the whole system: you write down a desired state, the watchers notice, and they go make reality match it. It also means the API server is the single point everything leans on. If it goes down, already-running Pods keep serving traffic, but nothing new can be created, scheduled, or scaled, and no controller can reconcile. That's why production clusters run several API server replicas behind a load balancer. You can tap the same watch stream yourself.

$ kubectl get pods --watch --output-watch-events
EVENT      NAME            READY   STATUS              RESTARTS   AGE
ADDED      web-6f8-2xk9q   0/1     Pending             0          0s
MODIFIED   web-6f8-2xk9q   0/1     ContainerCreating   0          1s
MODIFIED   web-6f8-2xk9q   1/1     Running             0          5s

Checking it's alive, and where it lives

On a cluster built with kubeadm, the API server is itself a Pod, a static one the kubelet starts straight from a file on disk at /etc/kubernetes/manifests/kube-apiserver.yaml. Most of its configuration is command-line flags in that file: which admission plugins are on, where etcd is, how authentication works. The bigger pieces do not fit on a command line, so a flag points at a separate file instead. The audit policy, encryption at rest, per-plugin admission settings and, on recent clusters, the structured authentication config that maps OIDC claims all live in files of their own. Two endpoints tell you its state: /livez (is the process healthy) and /readyz (is it ready to serve). On a managed service like EKS, GKE, or AKS (the hosted Kubernetes offerings from Amazon, Google, and Microsoft) you won't see this file at all, because the provider runs the control plane for you and hands you only the endpoint.

$ kubectl get --raw='/livez?verbose'
[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/apiservice-registration-controller ok
livez check passed
 
$ sudo grep -- --enable-admission-plugins /etc/kubernetes/manifests/kube-apiserver.yaml
    - --enable-admission-plugins=NodeRestriction

If the API server is unreachable, kubectl lies to you with connection errors and every controller stalls. Apps may still run. That split is why API health is a first-class alert.

Aggregation layers and extension APIs still front through the same door, so a custom resource gets no side entrance: your call is authenticated and authorized by the same API server before anything else sees it.

Rate limits and API Priority and Fairness exist because the API server is a shared, scarce resource. A CI job looping on kubectl get pods across every namespace can eat the budget and leave a real operator waiting on a simple read.

Try this

First ask the API server who it thinks you are (kubectl 1.28 and later), then run auth can-i for a create you care about, then list everything that identity is allowed to do in that namespace. Read the list against the answer you got: if the resource you were denied is missing from it, you have found your gap, and if it is listed and the write still fails, the denial came from admission and you are looking at the wrong gate. You are standing at the only front door and asking what it will let through.

$ kubectl auth whoami
ATTRIBUTE   VALUE
Username    [email protected]
Groups      [developers system:authenticated]
 
$ kubectl auth can-i create deployments --namespace prod
no
 
$ kubectl auth can-i --list --namespace prod
Resources                                       Non-Resource URLs   Resource Names   Verbs
selfsubjectaccessreviews.authorization.k8s.io   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                  []               [create]
pods                                            []                  []               [get list watch]
pods/log                                        []                  []               [get]

Takeaway

Nothing useful talks to etcd, the scheduler, or a kubelet except through the API server. Authn, authz, and admission all happen here before a write lands.

Related