Kubernetes network policies: default-deny, keep DNS up

Lock down pod-to-pod traffic with a default-deny NetworkPolicy, then re-allow DNS and the flows your apps actually need.

In a cluster without NetworkPolicy every pod can open a connection to every other pod, in every namespace, on every port. A compromised front-end can therefore reach the billing database directly, and the only thing stopping it is that nobody told it the address. NetworkPolicy objects are the fix, with two properties that shape the rollout: they are enforced by the CNI plugin, not by Kubernetes itself, and they are purely additive, so the first policy that selects a pod flips it from allow-all to deny-all for the direction the policy names, and every later policy only adds allowances.

Step 0: confirm something enforces it

A NetworkPolicy object is accepted by the API server whether or not the network plugin implements it. Calico, Cilium and the managed CNIs on the major clouds do; plain Flannel does not, and a cluster on Flannel will store your policies and enforce nothing. The test is empirical: apply a deny-all to a scratch namespace and check that a connection that worked before now times out. If it still connects, no amount of additional YAML will change that. One detail the run made obvious: test by ClusterIP, not by name. A deny of both directions also cuts the pod off from CoreDNS, so the first thing a probe by name reports is a failed lookup, which says nothing about enforcement; the timeout against the IP is the evidence.

Step 1: deny both directions

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: shop
spec:
  podSelector: {}              # every pod in the namespace
  policyTypes: [Ingress, Egress]
  # no ingress: or egress: rules, so nothing is allowed in either direction

Listing Egress in policyTypes is what most examples leave out and what makes this a real boundary: without it, a compromised pod can still reach every other namespace and the internet. With it, the namespace also loses DNS, because CoreDNS lives in kube-system and name resolution is an egress connection; that is the bad address above. Every application in the namespace now fails on the first lookup, so the next policy goes in the same change, not the next morning.

Step 2: give DNS back

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: shop
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
          podSelector:
            matchLabels: { k8s-app: kube-dns }
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }     # large responses and DNS-over-TCP fall back to TCP

The to entry combines a namespaceSelector and a podSelector in one list item, which means both must match: CoreDNS pods, in kube-system. Written as two separate list items the same selectors would mean either, which allows every pod in kube-system plus every pod labelled k8s-app: kube-dns anywhere. The kubernetes.io/metadata.name label is set on every namespace by the API server, so it is the reliable way to select a namespace by name.

Step 3: allow the flows the architecture needs, on both ends

Policies match pod labels, not Service names, so the selector must match the labels on the Pod template of the Deployment, and a flow has two ends: the destination's ingress policy must admit the source, and, once egress is denied, the source's egress policy must admit the destination. Forgetting the second half produces the classic symptom of web timing out on db:5432 although db clearly allows it; the run reproduced it exactly, with db-accepts-web applied alone the probe still timed out, and it opened only when web-reaches-db followed.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-accepts-web
  namespace: shop
spec:
  podSelector:
    matchLabels: { app: db }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: { app: web }
      ports:
        - { protocol: TCP, port: 5432 }
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-reaches-db
  namespace: shop
spec:
  podSelector:
    matchLabels: { app: web }
  policyTypes: [Egress]
  egress:
    - to:
        - podSelector:
            matchLabels: { app: db }
      ports:
        - { protocol: TCP, port: 5432 }

Step 4: prove the denied path

A policy that has only been tested from the pod it allows has not been tested. Run the same probe from a pod without the app: web label and expect a timeout; run it from another namespace and expect the same. The two results together are what the policy claims, and they are what to re-run after every CNI upgrade.

Mistakes that look like a broken cluster

When a policy change breaks the namespace: recovery in order of preference

NetworkPolicy limits where a compromised pod can connect. It says nothing about what that pod can ask the API server for (RBAC) or whether it runs as root with host mounts (Pod Security Admission); the three together are the boundary around one bad workload.

Related posts

Quick reference