Kubernetes interview questions

A Pod is basically the smallest thing you schedule in Kubernetes — one or more containers that share a network namespace and volumes, so they always land on the same node and can talk over localhost.

# one-off (debugging)
kubectl run web --image=nginx --restart=Never

# real workloads use a controller, not a bare Pod
apiVersion: v1
kind: Pod
metadata: { name: web }
spec:
  containers:
    - name: web
      image: nginx:1.27

A Service gives you a stable virtual IP and DNS name in front of Pods that keep coming and going. It only load-balances to Ready backends. ClusterIP stays internal; NodePort and LoadBalancer are how you expose things outside.

kubectl expose deploy web --port=80 --target-port=8080
# DNS: web.<namespace>.svc.cluster.local

I'd reach for a Deployment for interchangeable stateless replicas. StatefulSet when each replica needs a stable name, ordered rollout, and its own PVC — think databases. DaemonSet when I want exactly one Pod per eligible node, like agents or CNI.

kind: Deployment   # stateless web/api — N identical pods
kind: StatefulSet  # db-0, db-1 … stable DNS + own volume
kind: DaemonSet    # exactly one pod per node (agent, CNI)

Both inject config as env vars or files. I'd put sensitive stuff in a Secret — but I'd also say out loud that base64 in etcd isn't encryption. Real protection is tight RBAC plus encryption at rest.

kubectl create configmap app --from-literal=LOG_LEVEL=info
kubectl create secret generic db --from-literal=PASSWORD=s3cr3t
# base64 is NOT encryption — enable etcd encryption at rest

Labels are just key/value tags on objects; selectors query them. Services, Deployments, NetworkPolicies, PDBs — they all target Pods by label, not by name. That's what makes the model declarative.

kubectl get pods -l app=web,tier=frontend
kubectl label pod web release=canary

It's a virtual cluster scope for names, quotas, NetworkPolicies, and RBAC. I'd use namespaces to separate teams or environments — but I wouldn't treat a namespace alone as a hard security boundary. Pair it with RBAC, quotas, and NetworkPolicy.

kubectl create namespace team-a
kubectl -n team-a get pods

A Job runs Pods until they succeed — migrations, batch work. A CronJob just creates Jobs on a schedule. I'd always bound failing Jobs with backoffLimit and activeDeadlineSeconds so they don't retry forever.

kubectl create job db-migrate --image=migrate:1
kubectl create cronjob nightly --image=report:1 --schedule="0 2 * * *"

ClusterIP is internal-only — that's the default. NodePort opens the same high port on every node. LoadBalancer puts a cloud LB in front of that. ExternalName is just a DNS CNAME to something outside the cluster; there's no proxying.

kubectl expose deploy web --type=ClusterIP --port=80
kubectl expose deploy web --type=LoadBalancer --port=80

I'd start with get for the summary, then describe so I can read events and conditions, then logs — and logs --previous if it just crashed. That trio covers most of what I need before I go deeper.

kubectl get pod web -o wide
kubectl describe pod web        # events at the bottom
kubectl logs web -c app --previous

A ReplicaSet just keeps N identical Pods running. A Deployment owns ReplicaSets so you get versioned rolling updates and rollback. In practice I'd almost always create a Deployment, never a bare ReplicaSet.

Each rollout creates a new ReplicaSet; the previous one is kept (often scaled to 0) so undo is instant.

kubectl get deploy,rs,pods -l app=web
kubectl rollout history deploy/web

The endpoints controller tracks Ready Pods matching the selector into EndpointSlices. Then kube-proxy — or an eBPF CNI — programs each node to DNAT the Service ClusterIP to a backend Pod IP. If a Pod isn't Ready, it never shows up in that slice.

Unready Pods never appear in the EndpointSlice, so readiness is the real traffic gate.

kubectl get endpointslices -l kubernetes.io/service-name=web
kubectl describe svc web | grep -i endpoints

A PVC is the request for storage; a PV is the actual volume. StorageClass is how you get dynamic provisioning through a CSI driver. And the reclaim policy — Delete or Retain — decides what happens when the PVC goes away.

WaitForFirstConsumer delays binding until a Pod is scheduled so the volume lands in the same zone.

kubectl get pvc,pv
kubectl get storageclass

The Deployment spins up a new ReplicaSet and shifts Pods gradually, respecting maxSurge and maxUnavailable. Readiness is what actually gates traffic onto the new Pods. If something goes sideways, kubectl rollout undo takes me back to the previous ReplicaSet.

maxSurge / maxUnavailable control how aggressive the swap is.

kubectl set image deploy/web app=web:v2
kubectl rollout status deploy/web
kubectl rollout undo deploy/web

Readiness pulls a Pod out of Service endpoints until it can serve — without killing it. Liveness restarts a wedged container. Startup is the shield for slow boots so liveness doesn't kill you during init. I'd never point liveness at a shared dependency like the DB.

Never point liveness at a dependency (DB, cache) — a dependency outage will thrash-restart every Pod.

readinessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  failureThreshold: 3
startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30

Requests are what the scheduler reserves for placement and QoS. Limits are the cgroup caps: CPU over the limit gets throttled, memory over the limit gets OOM-killed — that's your exit 137.

resources:
  requests: { cpu: "100m", memory: "128Mi" }
  limits:   { cpu: "500m", memory: "256Mi" }

Taints push Pods away unless they carry a matching toleration. Affinity and anti-affinity pull Pods together or apart. topologySpreadConstraints are what I'd use to balance replicas across failure domains.

kubectl taint nodes gpu-1 gpu=true:NoSchedule

tolerations:
  - key: gpu
    operator: Equal
    value: "true"
    effect: NoSchedule

It samples a metric — CPU, memory, or custom via the metrics API — and computes desiredReplicas from current times currentMetric over target, then clamps to min/max. There's a stabilization window so it doesn't flap every minute.

kubectl autoscale deploy/web --cpu-percent=70 --min=3 --max=20
kubectl get hpa web --watch

I'd check rollout status and events first. Usually the new Pods never become Ready — ImagePullBackOff, bad probes, or a PDB / maxUnavailable blocking progress. I'd fix the new ReplicaSet or roll back; I wouldn't just raise surge and hope.

A rolling update only moves forward when new Pods become Ready. If readiness never passes, the Deployment stalls on purpose and old Pods keep serving. I'd describe the new Pods, look for ImagePullBackOff and probe failures, and check whether a PDB is blocking voluntary eviction of the old set. kubectl rollout undo is the safe move while I fix the image or probe.

kubectl rollout status deploy/web
kubectl describe deploy web | tail -40
kubectl get pods -l app=web -o wide
kubectl rollout undo deploy/web

Interviewer often follows with: Would you raise maxUnavailable to force progress, and what could go wrong?

It caps how many replicas can be voluntarily evicted at once — drains, node upgrades, that kind of thing — so you don't take an app below minAvailable. It doesn't protect against a node crash or an OOM kill. Interviewers love catching that myth.

kubectl drain and cluster autoscaler scale-down are voluntary, so they have to respect the PDB. A kernel panic, hardware failure, or preemption is involuntary and ignores it. I'd set minAvailable or maxUnavailable so upgrades can't break quorum, and I'd size replica count so the PDB is actually achievable — minAvailable: 2 with only two replicas blocks every drain.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: web }
spec:
  minAvailable: 2
  selector: { matchLabels: { app: web } }

Interviewer often follows with: What happens if minAvailable equals your replica count during a drain?

HPA changes replica count from metrics. VPA right-sizes requests and limits. Cluster Autoscaler adds or removes nodes when Pods can't schedule or nodes are empty. They solve different bottlenecks — and they can fight if you wire them carelessly.

HPA needs headroom on existing nodes; if every node is full, Pending Pods stay Pending until CA (or a human) adds capacity. VPA in update mode rewrites requests and may recreate Pods — pointing it at the same CPU metric as HPA without care causes oscillation. What I'd run in production: HPA on custom or CPU for replica count, VPA in recommendation mode or on memory only, CA watching unschedulable Pods. And I'd keep HPA minReplicas high enough for the PDB.

kubectl get hpa,vpa -A
kubectl describe hpa web
kubectl get nodes -o wide
kubectl -n kube-system logs -l app=cluster-autoscaler --tail=50

Interviewer often follows with: Why can HPA look healthy while users still see Pending Pods?

Filtering drops nodes that can't take the Pod. Scoring ranks what's left. Highest score wins, the scheduler writes a binding, and the kubelet on that node pulls and runs the containers. Scheduling itself never starts containers — it only picks the node.

Filter plugins remove nodes that lack resources, don't match nodeSelector or affinity, have untolerated NoSchedule taints, or can't satisfy volume topology. Score plugins then rank the rest — ImageLocality, NodeResourcesFit, inter-pod affinity, topology spread — and the binder persists the nodeName. If every node fails filter, the Pod stays Pending with FailedScheduling events. Preemption is a separate path after a failed attempt, where a higher-priority Pod can evict lower-priority ones to make room.

kubectl describe pod web | grep -A20 Events
kubectl get events --field-selector involvedObject.name=web

Interviewer often follows with: Name two filter failures that leave a Pod Pending forever.

I'd run two Deployments behind one Service. When green looks good, I flip the Service selector — or Gateway weight — to green, and I keep blue scaled up so rollback is instant.

Unlike a rolling update, blue-green keeps the old version fully intact until you deliberately shift traffic. The cutover is a selector or route change, not another kubectl set image under fire. Schema migrations that aren't backward-compatible will break a naive blue-green — I'd plan expand/contract migrations separately. And I'd document rollback as a one-liner that retargets the Service to blue.

kubectl patch svc web -p '{"spec":{"selector":{"app":"web","version":"green"}}}'
# rollback
kubectl patch svc web -p '{"spec":{"selector":{"app":"web","version":"blue"}}}'

Interviewer often follows with: How do you handle a non-backward-compatible database migration?

Liveness hitting a shared dependency causes restart storms. Aggressive liveness on a slow GC pause kills healthy Pods. Missing readiness ships traffic to half-booted processes. Startup probe is usually the fix for slow boots.

Readiness should mean "I can serve this request now." Liveness should mean "I'm wedged and need a process restart" — a narrow condition. If /healthz opens a DB connection, a database blip restarts every Pod and makes the outage worse. I'd keep liveness shallow — process up — and put the deeper checks on readiness. I'd tune failureThreshold and periodSeconds from real latency, and pair with a startupProbe so cold caches don't trip liveness during boot.

startupProbe:
  httpGet: { path: /livez, port: 8080 }
  failureThreshold: 30
  periodSeconds: 5
livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5

Interviewer often follows with: Show me a readiness check that would incorrectly remove every Pod during a dependency outage.

Ingress is the older L7 HTTP router, and you often end up in controller-specific annotations. Gateway API is the successor — role-oriented with GatewayClass, Gateway, HTTPRoute — more portable, with native weighted routing and non-HTTP routes.

# HTTPRoute — 90/10 canary
spec:
  rules:
    - backendRefs:
        - { name: web-v1, port: 80, weight: 90 }
        - { name: web-v2, port: 80, weight: 10 }

I'd almost always look for a selector/label mismatch or Pods failing readiness. Only Ready Pods whose labels match the Service selector show up in the EndpointSlice.

kubectl get endpointslices -l kubernetes.io/service-name=web
kubectl get pods --show-labels
kubectl describe svc web | grep Selector

It's a namespaced rule set that restricts Pod traffic by label selectors. Pods are wide open until a policy selects them — then only the flows you allow pass, and only if your CNI actually enforces NetworkPolicy.

Policies are additive. Select a Pod with an Ingress policy and list no allow rules, and that's deny-all ingress for that Pod. What I'd do in production is default-deny ingress — and often egress — per namespace, then explicit allows for the app and DNS. Without a policy-capable CNI like Calico or Cilium, the objects are theater. And egress deny without a DNS allow will break name resolution and look like a random outage.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny-ingress }
spec:
  podSelector: {}
  policyTypes: [Ingress]
# then add allow policies for app + DNS

Interviewer often follows with: Why did DNS break after you applied default-deny egress?

CoreDNS serves records like svc.ns.svc.cluster.local. Pods resolve short names through search domains in resolv.conf. Headless Services return Pod IPs, which is what StatefulSets lean on for peer discovery.

Pod resolv.conf usually sets ndots:5 and several search domains, so short names try multiple suffixes before the bare name — that means surprising latency and extra NXDOMAINs under load. NetworkPolicies that deny egress to kube-system DNS break resolution for that whole namespace. Headless Services skip the VIP and return Pod IPs — StatefulSets need that for peers like db-0.db. In large clusters I'd tune CoreDNS replicas and watch for conntrack saturation on the node DNS path.

kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup web.default.svc.cluster.local

Interviewer often follows with: What does ndots:5 change about how short names resolve?

I'd terminate TLS in a sidecar or CNI mesh — Istio, Linkerd, Cilium — so apps speak plaintext on localhost while the proxy mutually authenticates and encrypts on the wire.

The data plane injects a proxy next to each Pod, or uses eBPF. Certificates come from a mesh CA with short rotation. Policies decide PERMISSIVE vs STRICT mTLS. You pay some hop latency and ops cost; you get cryptographic peer identity beyond NetworkPolicy IP rules. NetworkPolicy still matters — mTLS authenticates peers but doesn't replace allow-lists for who may talk. Failure modes I've hit: sidecars not injected, clock skew breaking certs, and STRICT mode during a partial rollout black-holing pods that aren't ready yet.

kubectl get peerauthentication -A
kubectl get destinationrule -A
# Linkerd: linkerd viz edges deploy/web

Interviewer often follows with: Does mTLS replace NetworkPolicy? Why or why not?

Roles and ClusterRoles list allowed verbs on resources. RoleBindings and ClusterRoleBindings grant them to users, groups, or ServiceAccounts. Access is deny-by-default and purely additive — there aren't deny rules.

kubectl create role dev --verb=get,list --resource=pods
kubectl create rolebinding dev --role=dev --serviceaccount=team-a:builder
kubectl auth can-i list pods --as=system:serviceaccount:team-a:builder

I'd give it only the verbs and resources that pipeline actually needs in that namespace — typically get/list/watch/patch/update on specific Deployments — not cluster-admin, and not secrets get unless there's a real reason.

I'd start from the pipeline's actual API calls and write a Role that covers only those. Prefer namespaced RoleBindings over ClusterRoles. Avoid wildcards on resources and verbs. Keep image-pull secrets separate from deploy credentials. Then I'd verify with kubectl auth can-i --list as the ServiceAccount. Binding cluster-admin to a CI SA "just for Helm" is a classic interview fail — and a classic real-world mistake.

kubectl create role deploy --verb=get,list,watch,patch,update \
  --resource=deployments,deployments/status -n team-a
kubectl create rolebinding ci-deploy --role=deploy \
  --serviceaccount=team-a:ci -n team-a
kubectl auth can-i --list --as=system:serviceaccount:team-a:ci -n team-a

Interviewer often follows with: Should this SA also get secrets get? When is that justified?

It's built-in admission applying Pod Security Standards — privileged, baseline, restricted — per namespace via labels. It blocks or warns on privileged containers, host namespaces, and dangerous capabilities.

Three levels times three modes — enforce, audit, warn — set with namespace labels. I'd roll out warn and audit first so I see violations without breaking deploys, then flip enforce. restricted wants non-root, drop ALL capabilities, no hostPath/hostNetwork, and seccomp RuntimeDefault among other rules. PSA isn't a full policy engine — I'd use Gatekeeper or Kyverno for custom org rules — but it's the right default baseline for every tenant namespace.

kubectl label ns team-a \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

Interviewer often follows with: What breaks first when you enforce restricted on a legacy DaemonSet?

No — base64 is just encoding. I'd enable EncryptionConfiguration on the API server, ideally with a KMS provider, tighten RBAC on secrets, and keep high-value material in an external manager synced in short-lived form.

Without encryption at rest, anyone who gets an etcd snapshot reads every Secret as base64-decoded plaintext. A KMS provider keeps the data-encryption key outside the cluster so the snapshot alone is useless. identity: {} as a trailing provider lets you read old plaintext during migration. Encryption at rest doesn't stop a privileged API client with secrets get — RBAC and external stores still matter.

resources:
  - resources: ["secrets"]
    providers:
      - kms:
          name: cloudkms
          endpoint: unix:///var/run/kms.sock
      - identity: {}

Interviewer often follows with: Does encryption at rest stop a stolen kubeconfig with secrets get?

runAsNonRoot, a non-zero runAsUser, readOnlyRootFilesystem, allowPrivilegeEscalation: false, drop ALL capabilities and add back only what you need, and seccompProfile RuntimeDefault. That's the checklist I'd recite in an interview.

securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities: { drop: ["ALL"] }
  seccompProfile: { type: RuntimeDefault }

Validating and mutating webhooks intercept API requests after authn/authz. Mutating injects defaults or sidecars; validating rejects non-compliant objects. Gatekeeper evaluates ConstraintTemplates against the request.

The path is authenticate → authorize → mutating admission → schema validation → validating admission → persist. A misconfigured webhook with failurePolicy=Fail and a bad selector can block namespace creation cluster-wide — I've seen that. Always set timeouts, narrow namespaceSelectors, and watch webhook latency. Gatekeeper is great for org policy like required labels or banned registries, but it's not a substitute for PSS on pod hardening. I'd start constraints in dry-run or warn mode.

kubectl get constrainttemplates
kubectl get constraints -A
kubectl get validatingwebhookconfigurations

Interviewer often follows with: What is failurePolicy=Fail vs Ignore, and which do you choose for a security gate?

I'd use Kubernetes auth: the Pod presents its projected ServiceAccount JWT, Vault verifies it with the TokenReview API, and you get a short-lived Vault token bound to a role and policy.

Map Vault roles to ServiceAccount name plus namespace. Prefer projected tokens with audience and short expiration over the legacy secret-based SA token. I'd combine that with External Secrets Operator or the Vault Agent injector so the Kubernetes Secret — if you materialize one — is a cache, not the source of truth. Rotate Vault policies and Kubernetes Roles independently. Failure modes I've hit: wrong audience, TokenReview RBAC missing for Vault's SA, and clock skew.

# Pod uses projected SA token (audience=vault)
# Vault role bound to serviceaccountnames=app, serviceaccountnamespaces=team-a
kubectl exec deploy/app -- ls /var/run/secrets/tokens

Interviewer often follows with: Why is the legacy auto-mounted SA secret token a problem?

It keeps the source of truth in Vault or a cloud secret manager and materializes Kubernetes Secrets your Pods can mount, on a refresh interval — instead of copying secrets into Git or leaving long-lived values only in etcd.

SecretStore or ClusterSecretStore holds provider auth; ExternalSecret maps remote keys to a target Secret. You still need etcd encryption and RBAC because the materialized Secret is readable via the API. I'd treat ESO as sync, not a replacement for encryption or least privilege. And I'd watch for stale sync when the provider is down — pods may keep old values until refresh succeeds.

kubectl get secretstores,externalsecrets -A
kubectl describe externalsecret app-db -n team-a

Interviewer often follows with: If someone can kubectl get secret, did ESO make that safe?

I'd sign at build with cosign, and enforce verify in admission — Kyverno, Ratify, policy-controller — so unsigned or wrong-identity digests never schedule.

Tag mutability makes tag-only deploys unsafe — pin digests and verify the signature over that digest, and optionally a provenance attestation. Admission should fail closed for production namespaces. Keyless OIDC signing ties the identity to the CI workload, which is stronger than a shared static key in the pipeline. I'd pair that with a private registry allow-list so verification isn't bypassed by an alternate unsigned tag.

cosign sign --yes registry.example/app@sha256:...
cosign verify registry.example/app@sha256:...

Interviewer often follows with: Why is verifying a mutable tag weaker than verifying a digest?

Privileged pods, hostPath mounts — especially docker.sock — host PID or network namespaces, and dangerous capabilities like SYS_ADMIN. I'd close them with PSS restricted, dropped caps, no host mounts, and runtime detection.

Containers share the host kernel — isolation is namespaces, cgroups, and seccomp, not a VM boundary. A writable hostPath to /var/run/docker.sock or / is effectively host root. --privileged disables most isolations. Defense in depth for me: PSA enforce=restricted, Gatekeeper bans on hostPath and privileged, read-only rootfs, seccomp RuntimeDefault, and Falco or Tetragon rules for unexpected shell or mount activity. Multi-tenant clusters with untrusted workloads need stronger isolation — gVisor, Kata, or separate clusters.

kubectl get pods -A -o json | jq -r '
  .items[]
  | select(.spec.containers[]?.securityContext?.privileged==true
        or .spec.hostNetwork==true)
  | "\(.metadata.namespace)/\(.metadata.name)"'

Interviewer often follows with: Is a namespace enough isolation for an untrusted tenant?

Namespace per tenant with ResourceQuota and LimitRange, dedicated ServiceAccounts and namespaced RBAC, PSA enforce=restricted, and default-deny NetworkPolicies — and I'd be clear that soft multi-tenancy isn't a hostile-tenant boundary.

Trusted teams can share a cluster with namespace isolation plus quota, RBAC, PSA, and NetworkPolicy. Hostile or regulated tenants still share one kernel and one apiserver, so escape and noisy-neighbor risk remain — I'd use separate node pools, virtual clusters, or separate clusters for hard isolation. Disable automountServiceAccountToken where unused. Never give tenants cluster-admin or nodes/proxy. And I'd audit with Kubernetes audit logs and runtime detection.

kubectl create ns tenant-a
kubectl label ns tenant-a pod-security.kubernetes.io/enforce=restricted
kubectl create resourcequota t-a --hard=cpu=20,memory=40Gi,pods=50 -n tenant-a
# apply default-deny NetworkPolicy + allow DNS

Interviewer often follows with: When do you refuse soft multi-tenancy and split clusters?

Admission is preventative on the API object. Runtime detection watches syscalls and process behavior after the container starts — catching exploits admission can't see.

A compliant Pod can still get compromised via an app RCE. Falco or eBPF agents alert on unexpected shells, sensitive file reads, or breakout patterns. It doesn't replace PSS or NetworkPolicy; it closes the gap between "allowed to run" and "behaving as expected." I'd tune rules to cut noise, route alerts to on-call, and pair with network egress controls so a compromised pod can't freely exfiltrate.

kubectl -n falco get pods
kubectl -n falco logs -l app.kubernetes.io/name=falco --tail=50

Interviewer often follows with: Give an example attack that passes PSA restricted but Falco should catch.

kube-apiserver is the front door and the only component that talks to etcd. etcd stores cluster state. kube-scheduler places Pods. kube-controller-manager runs the reconcile loops. Nodes run kubelet and kube-proxy — or an eBPF dataplane.

kubectl get --raw='/readyz?verbose'
kubectl get componentstatuses  # legacy; prefer metrics / provider status

I'd describe for events and lastState, then logs --previous for the crashed process. From there I'm looking at exit code, probes, OOM — 137 — config or secret mounts, and the entrypoint. kubectl debug if I need a shell.

CrashLoop means the container exits and the kubelet backs off (10s → 20s → 40s …).

kubectl describe pod web
kubectl logs web --previous
kubectl debug web -it --image=busybox --target=app

Wrong image name or tag, registry auth missing or expired, rate limits, or the node can't reach the registry. I'd fix the reference, imagePullSecrets, or network path — restarts alone won't help.

kubectl describe pod web | grep -A15 Events
kubectl get event --field-selector reason=Failed
# check imagePullSecrets on the ServiceAccount / Pod

I'd start with describe Events for FailedScheduling. Then I'm checking resource requests vs free capacity, taints and tolerations, affinity, PVC binding, and ResourceQuota. Pending is almost never "the app is broken" — it's placement.

I'd read the scheduler message carefully: Insufficient cpu/memory, node(s) had taint, persistentvolumeclaim not bound, didn't match Pod's node affinity, and so on. Quota can reject Pod creates before scheduling even starts. Unbound PVCs with WaitForFirstConsumer wait for a feasible node; Immediate binding in the wrong zone can leave you Pending forever. If CA is enabled, I'd confirm it sees the unschedulable Pod and that the node group can satisfy the constraints — GPU, zone, taints.

kubectl describe pod web | grep -A30 Events
kubectl get pods -o wide
kubectl describe pvc data
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory

Interviewer often follows with: How do you tell ResourceQuota rejection from FailedScheduling?

kubectl sends the object to the apiserver, which authenticates, authorizes via RBAC, runs admission, validates, and persists to etcd. Controllers reconcile desired state; the scheduler binds Pods; kubelets start containers.

Client-side apply stores a last-applied annotation and three-way merges; server-side apply uses field managers and is what I'd prefer for multi-actor control. I'd diff before apply in CI. After persist, nothing "runs" inside the apiserver — controllers watch and act. A successful apply that creates a Deployment can still leave Pods Pending or CrashLooping, so I'd always follow with rollout status and Pod events.

kubectl diff -f app.yaml
kubectl apply -f app.yaml --server-side
kubectl rollout status deploy/web

Interviewer often follows with: What is the difference between client-side and server-side apply?

etcd is the source of truth for cluster state — lose it and you lose the control plane's memory. I'd protect it with TLS and peer auth, encryption at rest for Secrets, network isolation, and tested snapshots.

Run an odd number of members — 3 or 5 — for Raft quorum; even counts hurt fault tolerance. etcd is latency-sensitive — local SSD, low peer RTT. Restrict network access so only apiservers speak to it. Snapshot regularly and practice restore on a non-prod cluster; an untested backup isn't a backup. Managed Kubernetes hides etcd, but you still own encryption-at-rest settings and the provider's control-plane backup posture.

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=ca.crt --cert=peer.crt --key=peer.key snapshot save snap.db
etcdctl snapshot status snap.db -w table

Interviewer often follows with: What happens to quorum if you lose two members of a five-node etcd cluster?

The node stops heartbeating; after grace periods it's NotReady and controller-owned Pods get evicted and rescheduled — bare Pods don't. I'd check kubelet, runtime, disk or memory pressure, and connectivity to the apiserver.

node-monitor-grace-period marks NotReady; pods go NotReady quickly, but mass eviction waits for the eviction timeout so brief blips don't thrash the cluster. DaemonSet pods stay tied to the node. Common causes I've seen: kubelet crash, container runtime dead, disk pressure, CNI failure, or a lost route to the API. I'd cordon before maintenance so nothing new lands while I drain with PDB awareness.

kubectl describe node n1 | grep -A8 Conditions
kubectl get pods -A -o wide --field-selector spec.nodeName=n1
kubectl cordon n1
kubectl drain n1 --ignore-daemonsets --delete-emptydir-data

Interviewer often follows with: Why might a drain hang even though the node is NotReady?

I'd combine namespace quotas, PSA, NetworkPolicy, dedicated node pools with taints for noisy or sensitive workloads, and Pod anti-affinity or topology spread so one zone or node failure can't take the whole service down.

Blast radius is an architecture interview favorite. Soft tenancy limits who can break whom via API and network; topology spread and anti-affinity limit correlated failure; separate system, ingress, and tenant node pools keep a rogue DaemonSet from starving core addons. PDBs only protect voluntary disruption. I'd measure with failure injection — cordon a zone, break DNS NetworkPolicy, revoke a deploy SA — and confirm blast radius matches the design.

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector: { matchLabels: { app: web } }
kubectl taint nodes -l pool=ingress dedicated=ingress:NoSchedule

Interviewer often follows with: What still correlates if every replica lands in one AZ despite a Deployment?

I'd bet the new Secret materialized wrong — missing key, wrong name, or bad mount path. I'd describe lastState and exit, pull logs --previous for "file not found" or auth errors, then diff the Secret keys against what the container expects. Fix the Secret or volume mapping and let the kubelet remount — I wouldn't restart blindly.

Last time I saw this, ExternalSecrets had written a partial Secret after a Vault blip. Other common paths: key rename in the provider while volume items still reference the old key; envFrom or secretKeyRef pointing at a key that no longer exists so the container exits immediately; optional: false on a missing key failing the Pod before start. Base64 in the Secret is encoding only — a "rotated" value that looks fine in kubectl get secret -o yaml can still be wrong plaintext. I'd also check projected vs classic mounts, immutable Secrets that block updates, and whether the app caches credentials at start so a remount alone isn't enough without a rollout.

kubectl describe pod api-7f9c -c api | grep -A20 'Mounts\|Events\|Last State'
kubectl logs api-7f9c -c api --previous
kubectl get secret api-creds -o jsonpath='{.data}' | jq 'keys'
# Pod expects DB_PASSWORD; volume items must match
kubectl get pod api-7f9c -o jsonpath='{.spec.volumes}' | jq .

Interviewer often follows with: Would restarting the Pod fix a missing secretKeyRef, and why or why not?

I'd stabilize readiness first — that's usually the smoking gun when endpoints flicker. Then I'd check whether HPA is thrashing short-lived Pods that never stay Ready, and whether EndpointSlice churn is overwhelming kube-proxy or conntrack. Freeze HPA max if I need breathing room.

What I've seen: readiness tied to a dependency so a DB blip flaps every Pod and you get mass 503s even though processes are alive; readiness timeout too tight vs cold start so new HPA pods never stay Ready, old pods overload, HPA scales more; maxUnavailable plus surge during rollout with aggressive probes leaving a temporary empty Ready set; EndpointSlice updates racing with LB health checks. Mitigations I'd reach for: shallow liveness, readiness that reflects the local accept loop, startupProbe, HPA stabilizationWindowSeconds and scale-down policies, and minReadySeconds. I wouldn't "fix" 503s by removing readiness — that just ships traffic to broken pods.

kubectl get pods,endpointslices -l app=api -o wide
kubectl describe hpa api
kubectl get --raw /api/v1/namespaces/prod/endpointslices | jq '.items[]|select(.metadata.labels["kubernetes.io/service-name"]=="api")'
kubectl get events --field-selector reason=Unhealthy --sort-by=.lastTimestamp | tail -20

Interviewer often follows with: How would you tell readiness flapping from genuine upstream overload?

I'd describe the PVC and Pod together. With WaitForFirstConsumer the volume waits for a feasible node; Immediate binding can pin a PV to the wrong zone so the Pod never schedules. I'd check StorageClass volumeBindingMode, topology, and zone affinity.

Classic trap I've hit: StorageClass uses Immediate, CSI provisions in zone-a, Pod anti-affinity or nodeSelector wants zone-b — forever Pending with a volume node affinity conflict. WaitForFirstConsumer delays binding until the scheduler picks a node, then provisions in-zone — that's what I'd prefer for zonal disks. I'd also check ResourceQuota on storage, wrong accessMode, a deleted PV still referenced, and topology constraints that leave no legal zone. Fix is usually recreate the PVC with the right SC, or carefully relax affinity after confirming data-loss risk.

kubectl describe pvc data-db-0
kubectl get sc -o custom-columns=NAME:.metadata.name,BINDING:.volumeBindingMode,PROVISIONER:.provisioner
kubectl describe pod db-0 | grep -A25 Events
kubectl get pv -o wide

Interviewer often follows with: When is Immediate binding still acceptable?

From a debug pod I'd nslookup kubernetes.default.svc.cluster.local and kube-dns. Check CoreDNS pods and logs, then NetworkPolicies that deny egress to kube-system:53. ndots:5 causing search-list amplification is a separate story from a total DNS outage.

Three failure classes I've chased: CoreDNS CrashLoop or Pending — cluster-wide timeouts; namespace NetworkPolicy default-deny egress without allowing UDP/TCP 53 to kube-dns — looks like "app DNS broken" while the API still works from my laptop; ndots:5 plus a long search path so short names generate many queries and under conntrack or CoreDNS saturation you see intermittent failures. Also CNI path to the CoreDNS ClusterIP, and NodeLocal DNSCache misconfig. Fix order for me: restore CoreDNS Ready, open DNS egress, then tune dnsConfig ndots for chatty clients.

kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50
kubectl run -it --rm dbg --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default.svc.cluster.local
kubectl get netpol -A
# resolv.conf: ndots and search
kubectl exec api-0 -- cat /etc/resolv.conf

Interviewer often follows with: What does ndots:5 change about how short names resolve?

Ingress controllers often load TLS Secrets into memory and only reload on watch events — or they cache by secret name. I'd confirm the Secret data actually changed, that the Ingress references the right secretName, and force a controller reload if the watch missed the update.

Causes I've hit: cert-manager renewed a different Secret than the Ingress tls[].secretName; Secret updated but controller cache went stale — bounce the ingress controller pods; multiple Ingresses or Gateway listeners sharing a hostname with conflicting certs; a CDN or external LB terminating TLS with its own expired cert in front of the cluster; HTTP→HTTPS redirect to a different host. Kubernetes doesn't hot-reload TLS into every proxy by itself — the ingress implementation does. I'd verify with openssl s_client against the LB IP and against a port-forward to an ingress pod to split edge vs controller.

kubectl get ingress api -o yaml | grep -A6 tls
kubectl get secret api-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates -subject
kubectl -n ingress-nginx rollout restart deploy/ingress-nginx-controller
openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null | openssl x509 -noout -dates

Interviewer often follows with: How do you prove the problem is the CDN versus the Ingress controller?

I'd treat apiserver/etcd latency as a cluster-wide incident. Check apiserver and etcd metrics — WAL fsync, request duration — find chatty clients and large LIST/WATCH callers, rate-limit or scale down noisy operators, and stop kubectl get -A loops. Stabilize etcd disk before tuning apps.

Watch storms happen when many controllers re-list and re-watch huge objects after timeouts, amplifying load when etcd is already slow — a positive feedback loop. Causes: saturated etcd disk, too many CRDs or objects, inefficient operators without pagination, admission webhooks adding latency. Response for me: protect etcd, shed load by disabling non-critical operators and stopping CI apply storms, fix hot webhooks, then tune. On managed control planes I'd open a provider ticket with metrics but still reduce client QPS from our side. I wouldn't "fix" by deleting etcd data.

kubectl get --raw='/readyz?verbose'
kubectl get --raw='/metrics' | grep -E 'apiserver_request_duration|etcd_request'
# self-managed: etcd fsync / disk
# find chatty controllers
kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl top pods -A | sort -k3 -n | tail

Interviewer often follows with: Why can deleting a stuck operator deployment improve apiserver latency immediately?

I'd impersonate or use auth can-i to map effective permissions, find the Role or ClusterRole rules granting secrets get/list/watch, remove or replace with least privilege, rotate anything they could have read, and review audit logs for prior access.

Audit path: kubectl auth can-i --list --as=user, check RoleBindings and ClusterRoleBindings, watch for aggregation labels and wildcards. secrets get isn't mitigated by base64 — that isn't encryption. Fix: replace with get on ConfigMaps only, or use External Secrets patterns so humans never need kubectl get secret. Rotate credentials that were readable. Longer term: admission policy blocking Role rules on secrets, periodic access reviews, and namespace separation of prod. Encryption at rest doesn't help against authorized API reads.

kubectl auth can-i get secrets -n prod [email protected]
kubectl auth can-i --list -n prod [email protected] | grep -i secret
kubectl get rolebinding,clusterrolebinding -A -o wide | grep -i dev
kubectl edit role dev-edit -n prod   # remove secrets from rules
# rotate after exposure
kubectl delete secret db-creds -n prod  # then recreate from source of truth

Interviewer often follows with: Does enabling etcd encryption at rest fix this RBAC mistake?

I'd shift weight back to stable immediately via Gateway, Ingress, or mesh config — I wouldn't wait for an image rebuild. Keep the canary Deployment around for forensics, or scale it to zero after traffic is off. Confirm metrics go green, then post-mortem.

Priority is traffic, not deleting pods. If we're on Gateway API HTTPRoute weights, I'd patch canary to 0 and stable to 100. Flagger or Argo rollouts: abort or undo. Mesh: restore destination rule subsets. Pitfalls: DNS or CDN caching of canary routes, sticky sessions still pinning users, and a canary that shared a Deployment with stable so rollout undo is the right tool instead of weights. PDB doesn't stop you from changing route weights. After cutback, capture logs from canary pods before scale-to-zero. Separate "stop the bleeding" from "root cause."

kubectl patch httproute api -n prod --type=json -p='[
  {"op":"replace","path":"/spec/rules/0/backendRefs/0/weight","value":100},
  {"op":"replace","path":"/spec/rules/0/backendRefs/1/weight","value":0}
]'
kubectl get pods -l app=api,track=canary -o wide
kubectl logs -l app=api,track=canary --tail=100

Interviewer often follows with: When is rollout undo the wrong rollback tool for a canary?

I'd relieve disk immediately — clear image and container garbage, fix log fill-up, cordon the worst nodes. PriorityClass and requests don't stop eviction when disk thresholds trip. Free space, fix the writers, then tune eviction thresholds and PriorityClass for system-critical addons only.

DiskPressure eviction is involuntary from the PDB's point of view — PDBs don't protect against kubelet eviction for disk. Rank for me: free node filesystem under /var/lib/containerd and logs, find pods writing to emptyDir without limits, check ephemeral storage requests and limits. QoS Guaranteed helps memory and CPU contention more than disk — ephemeral storage limits are the right lever. Critical DaemonSets should use high PriorityClass and careful node sizing. Root causes I've seen: unbounded logs, image pull storms, leftover failed pods. Longer term: log rotation, GC settings, separate disk for the runtime, alerts on nodefs/imagefs before the eviction threshold.

kubectl describe node n3 | grep -A12 Conditions
kubectl get pods -A -o wide --field-selector spec.nodeName=n3
kubectl get events -A --field-selector reason=Evicted --sort-by=.lastTimestamp | tail -20
# on the node (via provider console / ssh / debug): df -h /var/lib/containerd
kubectl cordon n3

Interviewer often follows with: Why will a PDB not save your API pods from DiskPressure eviction?

I'd inspect Job conditions and Pod exit codes. Completions only count successful pods. Failures may retry until backoffLimit, hang without exiting, or leave pods Running because the process never terminates. Fix the workload or set activeDeadlineSeconds and backoffLimit.

Patterns I've seen: container sleeps forever after work so the Job never sees Succeeded; crash loop consuming backoffLimit so the Job Failed; parallelism greater than 1 with Indexed jobs and some indexes stuck; a sidecar keeping the Pod Running after the main container exits. I'd check .status.succeeded vs .spec.completions, and ttlSecondsAfterFinished for cleanup. I wouldn't delete the Job blindly if I need logs — fetch logs first. For CronJobs I'd also check concurrencyPolicy and startingDeadlineSeconds for overlapping runs.

kubectl describe job migrate
kubectl get pods -l job-name=migrate -o wide
kubectl get pods -l job-name=migrate -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.containerStatuses[0].state}{"\n"}{end}'
kubectl logs job/migrate --all-containers
kubectl get job migrate -o yaml | grep -A20 'status:\|backoffLimit\|completions'

Interviewer often follows with: How do sidecars prevent a Job from reaching Completions?

Traffic often enters via the sidecar proxy. If the proxy is down, endpoints may still list the Pod IP while the data-plane redirect black-holes traffic — or readiness only checked the app container. I'd fix injection or proxy config, and make readiness reflect the proxy.

Classic mesh failure: iptables or eBPF redirects pod traffic to an Envoy or linkerd-proxy that isn't listening — app /healthz on the app port still passes if probes bypass the proxy or target the app container. I'd make sure the proxy has its own readiness probe. Also holdApplicationUntilProxyStarts or native sidecars so the app doesn't accept before the proxy is up. Non-mesh agents usually don't steal the Service path unless they share the network incorrectly — I'd focus on whether the CNI or mesh hijacks inbound. STRICT mTLS with missing sidecar peers drops traffic hard.

kubectl get pod api-1 -o jsonpath='{range .status.containerStatuses[*]}{.name}{" ready="}{.ready}{"\n"}{end}'
kubectl logs api-1 -c istio-proxy --previous
kubectl describe pod api-1 | grep -A30 'Readiness\|Events'
kubectl get endpointslices -l kubernetes.io/service-name=api -o yaml

Interviewer often follows with: Should Service readiness consider only the app container in a meshed Pod?

I'd find pods blocking voluntary eviction with describe pdb and eviction API errors. Raise replicas or temporarily relax maxUnavailable/minAvailable so the PDB is satisfiable, drain one node at a time, then restore the PDB. Force delete bypasses PDB but risks outage — that's break-glass.

Drain uses the Eviction API and must respect PDBs. Deadlock patterns: minAvailable equals replica count, multiple PDBs selecting the same pods, or HPA minReplicas too low so you can't surge healthy pods before drain. PDB doesn't block node crashes or kubelet eviction. Safe unblock for me: scale the Deployment up, wait Ready, drain; or patch PDB maxUnavailable during a change window with an explicit revert. kubectl drain --force --disable-eviction is last resort. I'd also check terminating pods waiting on finalizers and StatefulSet OrderedReady stuck on an unhealthy ordinal.

kubectl get pdb -A
kubectl describe pdb api -n prod
kubectl drain n5 --ignore-daemonsets --delete-emptydir-data --dry-run=server
# if AllowedDisruptions is 0, scale out first
kubectl scale deploy/api --replicas=4
kubectl drain n5 --ignore-daemonsets --delete-emptydir-data

Interviewer often follows with: What is the difference between a PDB blocking drain and a finalizer stuck Terminating?

I'd immediately lower HPA maxReplicas and/or pause CA scale-up — set a node-group max. Find the metric that drove scale — error loop, bad custom metric, CPU spin. Fix the app or metric, then put hard caps, scale-down stabilization, and budget alerts in place.

Common drivers I've seen: readiness or liveness misconfig causing restart CPU spikes that HPA reads as load; a custom metric that counts errors or queue depth without a ceiling; retry storms amplifying work as replicas grow; CA with unbounded max nodes. HPA and CA reinforce each other — more pods → Pending → more nodes → more scheduled pods. Mitigations: maxReplicas, CA utilization thresholds and max size, separate node pools with caps, scale-down-delay, and alerts on replica count and node count. Cost control is capacity design, not only rightsizing requests.

kubectl patch hpa api --type=merge -p '{"spec":{"maxReplicas":10}}'
kubectl describe hpa api
kubectl get nodes -o wide | wc -l
# CA logs (kube-system) for scale-up decisions
kubectl -n kube-system logs -l app=cluster-autoscaler --tail=100 | grep -i scale

Interviewer often follows with: Why can fixing CPU requests alone fail to stop this loop?

I'd run active-active or warm-standby with independent control planes, replicate config via GitOps, keep data-layer failover separate, and shift traffic at DNS or a global LB based on health. Practice failback. Avoid split-brain on stateful dependencies.

Stateless pods are the easy part; state is the interview. Patterns: active-passive with health-checked Global Accelerator or DNS failover; active-active with locality and conflict-free data, or regional DBs with async replica and an accepted RPO. GitOps promotes the same revision to both clusters. Secrets and Ingress TLS must exist in both. Health signals should check regional dependencies, not only Pod Ready. Active-active doubles cost and needs careful session affinity; active-passive is cheaper but higher RTO. PDBs and HPA are per-cluster — they don't span clusters. I'd test by withdrawing a region's pool from the global LB, not by hoping etcd somehow syncs across regions.

# each cluster exposes /readyz behind its Ingress
kubectl --context=prod-east get deploy,api,ing -n api
kubectl --context=prod-west get deploy,api,ing -n api
# traffic shift is DNS/GLB weight — not kubectl in one cluster
# verify both revisions match
kubectl --context=prod-east get deploy api -o jsonpath='{.spec.template.spec.containers[0].image}'
kubectl --context=prod-west get deploy api -o jsonpath='{.spec.template.spec.containers[0].image}'

Interviewer often follows with: What breaks if you only replicate Kubernetes YAML but not the database failover plan?

Guaranteed QoS doesn't prevent OOM when usage exceeds the limit — the cgroup still kills on limit breach. I'd raise limit and usually request together, fix the leak or spike, or carefully use memory-aware HPA/VPA. limit equals request only means QoS class, not infinite headroom.

Myths I'd kill in the interview: Guaranteed doesn't mean immune to OOM; request is scheduling reservation, limit is the cgroup hard cap. RSS plus page cache plus JVM non-heap can exceed what people expect from "heap at 80% of limit." If limit equals request and the spike is real working set, you have to raise both or reduce footprint. I'd check events for Last State OOMKilled, container_memory_* metrics, and remember that noisy neighbors don't cause a limit breach — that's local cgroup. Temporary mitigation: raise limit, add memory load testing, profile the spike.

kubectl describe pod api-3 | grep -A15 'Last State\|Limits\|Requests\|OOM'
kubectl get pod api-3 -o jsonpath='{.spec.containers[0].resources}' ; echo
kubectl top pod api-3 --containers
# events
kubectl get events --field-selector involvedObject.name=api-3

Interviewer often follows with: If limit equals request, are you protected from node-level memory pressure eviction?

I'd compare node identity and pull credentials — imagePullSecrets on the ServiceAccount or Pod versus node instance IAM, IRSA, or workload identity for the registry. Nodes in a different subnet or without the registry route will fail even when the Secret is correct.

Split-brain pull auth I've chased: Pod-level imagePullSecrets work everywhere the Secret is mounted — if only some nodes fail, look at node networking and endpoint reachability; EKS/GKE-style node IAM or kubelet credential providers — only nodes with the right role succeed, and new node groups often miss the policy; different runtimes or mirrors per pool; rate limits keyed by egress IP with one poisoned NAT gateway. Fix: align node roles, attach imagePullSecrets to the SA for consistency, verify registry firewall per subnet. describe Events show "unauthorized" vs "timeout" — I'd treat those differently.

kubectl describe pod web-a | grep -A20 Events
kubectl get pod web-a web-b -o wide
kubectl get sa default -o yaml | grep -A5 imagePullSecrets
kubectl get nodes n-good n-bad -o yaml | grep -E 'node.kubernetes.io|iam|instance'
# from each node path: can the registry be reached?
kubectl debug node/n-bad -it --image=busybox -- wget -S -O- https://registry.example.com/v2/

Interviewer often follows with: When would fixing the ServiceAccount imagePullSecrets not help the failing nodes?