Network policies & default-deny

Segment the cluster without breaking DNS.

A single compromised pod should not be able to reach your payments database. On a stock cluster it can, and so can every other pod you run. Kubernetes hands each pod its own routable Internet Protocol (IP) address and lets any pod open a connection to any other pod, on any port, in any namespace, with no Network Address Translation (NAT) sitting in between. That flat network is lovely for developers who never want to think about wiring. It is also a gift to whoever lands a shell in one of your containers. From that one foothold they can scan every service in the cluster, reach databases nobody meant to expose, and stream the data straight out to the internet with nothing in the way. This is how one exposed web pod becomes a cluster-wide incident: the initial bug gets someone into a low-value container, and the flat network does the rest of the work for them. A NetworkPolicy is how you put the walls back, and default-deny is how you make those walls the rule instead of the exception.

Think of a NetworkPolicy as a bouncer's guest list pinned to a set of pods. Here is the bit that trips everyone up the first time they use it. A pod has no bouncer at all until some policy selects it. The moment one does, that pod flips to deny-all for whichever direction the policy names, ingress or egress, and only the sources you wrote on the list get through. There is no 'block just this one bad thing' knob. Policies are additive: if two of them select the same pod, a connection gets in when either one permits it, and the only deny you can express is the empty default you start from. So you select everything, allow nothing, then add back the connections that genuinely need to exist. The object is namespaced, and the real enforcement happens in your CNI (Container Network Interface plugin), not in the API server. That last detail will bite you the first time you test a policy on a cluster whose CNI ignores it.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments
spec:
  podSelector: {}                  # empty selector = every pod in the namespace
  policyTypes: [Ingress, Egress]   # naming a type with no rules = deny that direction

$ kubectl -n payments apply -f default-deny.yaml
networkpolicy.networking.k8s.io/default-deny-all created
 
# confirm it selects every pod and permits nothing in either direction
$ kubectl -n payments describe netpol default-deny-all
  PodSelector:  <none> (Allowing the specific traffic to all pods in this namespace)
  Allowing ingress traffic:
    <none> (Selected pods are isolated for ingress connectivity)
  Allowing egress traffic:
    <none> (Selected pods are isolated for egress connectivity)
  Policy Types: Ingress, Egress
 
# every hostname lookup now hangs, because all egress just went away
$ kubectl -n payments exec deploy/web -- nslookup payments-api
;; connection timed out; no servers could be reached
command terminated with exit code 1

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns-egress, namespace: payments }
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels: { kubernetes.io/metadata.name: kube-system }
    ports:
    - { protocol: UDP, port: 53 }
    - { protocol: TCP, port: 53 }

$ kubectl -n payments apply -f allow-dns.yaml
networkpolicy.networking.k8s.io/allow-dns-egress created
 
# same lookup, now it resolves
$ kubectl -n payments exec deploy/web -- nslookup payments-api
Server:    10.96.0.10
Address:   10.96.0.10:53
Name:      payments-api.payments.svc.cluster.local
Address:   10.107.4.19

Per-consumer allowlists

With the deny in place, you reopen the exact paths that should exist and nothing more. Let inbound traffic reach the payments API only from the web pods that call it, matched by their label, never by IP address, which changes every time a pod reschedules onto a new node. Here is the part people forget. Ingress and egress are separate ledgers. An ingress rule on payments-api decides who may knock on its door, but the web pods are still sitting under the default-deny egress from earlier, so they also need an egress rule that says they may dial out to payments-api. Skip that second rule and the call dies with the ingress side looking perfect. So you write the pair together. What you get for the effort is a policy set you can read like a wiring diagram: who talks to whom, on which port, nothing left implied.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: payments-allow-web, namespace: payments }
spec:
  podSelector:
    matchLabels: { app: payments-api }    # this policy guards the API's inbound
  policyTypes: [Ingress]
  ingress:
  - from:
    - podSelector:
        matchLabels: { app: web }         # web pods in THIS namespace
    - namespaceSelector:
        matchLabels: { team: storefront } # AND a whole trusted namespace...
      podSelector:
        matchLabels: { app: web }         # ...but only its web pods
    ports:
    - { protocol: TCP, port: 8080 }
---
# egress is a separate ledger: web must also be allowed OUT to payments-api
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: web-allow-egress-payments, namespace: payments }
spec:
  podSelector:
    matchLabels: { app: web }
  policyTypes: [Egress]
  egress:
  - to:
    - podSelector:
        matchLabels: { app: payments-api }
    ports:
    - { protocol: TCP, port: 8080 }

One trap here causes real outages and real audit findings. Inside a from block, a namespaceSelector and a podSelector written as two separate list items, each with its own leading dash, get OR'd together: either one matching is enough to let traffic in. Nest both under a single list item, one dash, and they turn into an AND, meaning the pod must carry that label and live in a namespace carrying its label. Say you wanted 'web pods in the storefront namespace.' That is the AND form. Write it as two dashes by accident and you have allowed something far wider: every pod in storefront, plus any pod labelled web back in your own namespace. A bare podSelector is always scoped to the policy's own namespace, so it does not reach across the cluster, but it still waves through a whole room of pods you never meant to open.

Prove it, both directions

A NetworkPolicy you never tested is a NetworkPolicy that does not work. Check it from a pod that should be allowed, which connects cleanly, and from one that should not, which hangs and times out instead of getting a polite refusal. That silence is the tell. A deny in NetworkPolicy means the packet is dropped with no reply at all, so the client just sits there until its own timeout fires. Test both directions while you are at it, since a pod that is allowed to receive is not automatically allowed to send, so probe the egress paths the same way. And if traffic still flows from everywhere after you apply a deny, the likeliest cause is that your CNI does not enforce NetworkPolicy in the first place. Calico, Cilium, and Antrea do. Some managed defaults and plain 'bridge' plugins will happily accept your YAML, store it, and then ignore every line of it, which is worse than no policy because it looks like protection.

$ kubectl -n payments apply -f allow-from-web.yaml
networkpolicy.networking.k8s.io/payments-allow-web created
networkpolicy.networking.k8s.io/web-allow-egress-payments created
 
# allowed pod: web is on the ingress list AND has egress out to payments-api
$ kubectl -n payments exec deploy/web -- curl -sS -m 3 payments-api:8080/healthz
ok
 
# disallowed pod: not labelled web, so it hangs to the timeout (the silent drop)
$ kubectl run probe --rm -it --image=nicolaka/netshoot -n payments -- \
    curl -m 3 payments-api:8080
curl: (28) Failed to connect to payments-api port 8080 after 3001 ms: Timeout was reached

CNI support is not optional trivia. If your cluster uses a plugin that ignores NetworkPolicy, every YAML you write is documentation, not enforcement. Confirm with a deliberate deny test before you trust the policy set in an audit.

Selectors OR across list items and AND inside one item. That single grammar detail is the difference between "frontend can reach payments" and "anyone in the namespace can." Write the allow as a consumer label talking to a provider label, not as a namespace-wide blanket.

Egress to the cloud metadata IP is a classic miss. Pods that never need it should never reach 169.254.169.254. Carve it out with an ipBlock except rule once you open broader egress, and re-test with a probe pod after every change.

Policy testing belongs in the same pull request as the Deployment. Spin a probe Job that expects allow and another that expects deny. If both pass, the YAML matches the threat model. If only the allow passes, you never proved the deny and you are one label typo away from an open namespace. Write the verification command next to the control in the same pull request, and keep the sample output so the next on-call person can tell pass from fail without guessing.

Try this

Apply a default-deny and a narrow allow, then prove both directions with a temporary curl pod. You want DNS to keep working and everything else to time out until you open the exact port.

$ kubectl -n payments apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny-all }
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns }
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels: { kubernetes: kube-system }
    ports:
    - { protocol: UDP, port: 53 }
    - { protocol: TCP, port: 53 }
EOF
networkpolicy.networking.k8s.io/default-deny-all created
networkpolicy.networking.k8s.io/allow-dns created
 
$ kubectl -n payments run probe --rm -it --image=busybox:1.36 --restart=Never -- \
  wget -qO- --timeout=2 http://payments-api:8080/health || echo FAILED
wget: download timed out
FAILED

Takeaway

A NetworkPolicy that selects a pod flips that direction to deny-all. Always allow DNS with the deny, then open only the consumer-to-provider edges you can name.

Related