Cilium network policy & Hubble

Identity-based L3–L7, default-deny, encryption.

A burglar who gets into one apartment should not find that the same key opens every other door in the building. Runtime security is partly about what a process does, and partly about what that process can reach while it runs. Cilium's identity-based network policy shrinks the reach. Hubble records which doors got rattled.

Two names, in plain words. Cilium wires up pod networking using eBPF (extended Berkeley Packet Filter, a way to run small sandboxed programs inside the Linux kernel), and adds a security layer on top of it. Hubble is its flow recorder: every connection a pod makes or attempts, with a verdict attached. You reach for both when east-west traffic (pods talking to other pods inside the cluster) matters as much as which syscalls a process makes.

Policy that follows labels, not IPs

Halfway through a service mesh migration you can end up with two things answering the same question: the sidecar proxy and Cilium. Two sets of allow and deny logic produce failures nobody can explain. Write down which layer owns L7 (layer 7, the application layer where HTTP methods and paths live) for the length of the transition, and put that note where the on-call engineer will actually find it.

DNS-aware policy takes the pressure off allowing whole IP ranges. For a third-party API, reach for to_fqdns or whatever equivalent your Cilium version supports, so the rule names the vendor instead of guessing their address space. CIDR allowlists (CIDR is Classless Inter-Domain Routing, the 10.0.0.0/8 way of writing an IP range) rot quietly, because vendors move their egress IPs and nobody sends you a memo. Practice on staging until the commands bore you. Boring is reliable.

A CiliumNetworkPolicy picks the pods it applies to by label, then lists what those pods are allowed to do: ports, protocols, HTTP paths, DNS names, even Kafka topics. Switch on default-deny across the cluster and a compromised pod reaches nothing at all until some rule says otherwise. That is what zero-trust east-west looks like once it stops being a slogan.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-ingress
spec:
  endpointSelector:
    matchLabels: { app: api }
  ingress:
  - fromEndpoints:
    - matchLabels: { app: frontend }
    toPorts:
    - ports: [{ port: "8080", protocol: TCP }]
      rules:
        http:
        - method: POST
          path: "/checkout"

Start from deny

Apply a default-deny at the cluster or namespace level, then open one flow at a time based on what Hubble actually recorded, not on what someone believes the service calls. Every allow rule gets an owning team and a ticket number. Re-read the whole set every quarter. A rule that outlives the service it was written for turns into a road an attacker can drive down.

What Hubble shows you

The Hubble user interface and its command line show live flows with L7 detail, which is how you write a tight policy without adding a single line of code to the app, and how you reconstruct an incident afterwards. Ship the flows to Prometheus and you can alert when a pod starts talking to a destination it has never talked to before.

hubble observe --namespace prod --pod payments-api-7c9 --last 5
 
# example output:
1.2.3.4:443 <- 10.0.4.22:51234 POST /checkout HTTP/1.1 200
10.0.8.1:53  -> 10.0.4.22:38491 DNS qname=api.internal

Encryption and identity do different jobs

WireGuard or IPsec transparent encryption scrambles pod-to-pod traffic on the wire without asking every application team to bolt TLS onto their code. The quieter win is that identity survives a reschedule. Policy attaches to labels, not to IP addresses that get handed back and reissued on every rollout.

cilium status | grep -E "KubeProxyReplacement|ClusterMesh"
 
# example output:
KubeProxyReplacement: Strict   [OK]
ClusterMesh:          0/0 remote clusters

Living beside Kubernetes NetworkPolicy

Cilium will enforce plain Kubernetes NetworkPolicy objects next to its own CiliumNetworkPolicy, so both can be live in the same cluster. Know which of the two owns L7 rules in your estate before the night you have to debug one. A standard NetworkPolicy cannot express "POST to /checkout only". Cilium can, and that is where the extra depth earns its keep.

hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true

Who owns each allow rule

Give every allow rule a named owning team, a ticket, and a date when a human reviews it again, either in an annotation or as a link into your CMDB (configuration management database, the inventory of who owns what). Hubble flow logs are what makes the audit conversation short: you opened 443 because six months of flows show POST /checkout arriving only from the frontend tier. When a service is switched off, delete the rules that fed it. Leftover allows outlive the reason anyone wrote them.

cilium connectivity test
 
# example output:
✅ All 14 tests passed (4.2s)

Run the connectivity test after every policy change. If the pipeline that applies a CNP (CiliumNetworkPolicy) cannot prove that core probes still reach kube-dns and the dependencies the app needs, that pipeline should fail loudly rather than ship a quiet outage.

Field notes from real clusters

A flat pod network behaves like an office where every badge opens every door, because the doors have no idea who is holding the badge. Cilium is an eBPF-powered CNI (Container Network Interface, the plugin that wires up pod networking) with a security layer on top, and it gives each workload an identity derived from its labels. Pods then talk because of who they are, rather than because they happen to share a subnet. Runtime detection that watches processes and ignores reachability leaves the lateral-movement corridor wide open.

Default-deny in your sensitive namespaces is the moment this work turns real. Allow only the egress the app genuinely needs: its database, the mesh, DNS, the specific external APIs. Allow ingress only from callers that have to call you. If your Cilium version and your change process support a staged, audit-friendly rollout, use it. Do not pitch a tent there. "We meant to deny" is not a control.

Hubble is the pane you keep open: flows with identities attached, a verdict on each one, DNS names spelled out. When a policy blocks payments from reaching redis, Hubble says exactly that, instead of leaving a developer staring at a timeout. Teach developers to open Hubble before they file a ticket asking for 0.0.0.0/0.

Encryption and identity plug different holes. WireGuard or IPsec mode, whichever your setup runs, cuts down sniffing and spoofing on the node network. It hides the bytes. It has no opinion about who may connect, so it never replaces policy. Run both when your threat model asks for both.

Plain Kubernetes NetworkPolicy objects still have a place. Cilium understands them, and adds CiliumNetworkPolicy plus its cluster-wide variant with richer L7 and DNS rules on top. Pick one authoring style per team so two objects never encode contradicting intent, and keep the policies in Git next to the app chart so they ship and roll back together.

Policies need owners the same way services do. Delete a service and leave its policy behind, and nine months later somebody hits a denial nobody can explain. Have CI (continuous integration, the automated checks that run on every change) verify that each allow rule still matches a workload that exists. A stale allow is as dangerous as a missing deny.

Pair network policy with the process controls from the rest of this course. A compromised pod that cannot reach the cloud metadata API or the secrets service next door does far less damage even when someone lands a shell inside it. Reachability and behavior are two separate layers, and you want both.

Default-allow feels harmless right up until a compromised frontend pod walks the whole ClusterIP range to see what answers. Hubble will show you those successful connections in full detail, which is the point, though the fix is a policy rather than another dashboard. Start with one sensitive namespace. Deny egress except DNS and its own data stores, watch Hubble for what breaks, repair it, then take the next namespace. One finished namespace teaches your organisation more than a slide with the words zero trust networking on it.

When a connection times out, the first guess is always that the app is sick. Train people to check the Hubble verdict before they scale up pods. A deny line carries the identity of who spoke, who was listening, and which rule stopped it. That is a ticket somebody can act on, and it turns network policy from black magic into something you read like a log.

Common ways this goes wrong: opening /0 egress "temporarily", stacking a Kubernetes NetworkPolicy and a CiliumNetworkPolicy that contradict each other, and half-writing L7 rules so an HTTP method slips through that should not. Temporary becomes permanent, every time. Contradictions fail open or closed depending on luck. Be boring instead: one policy family per namespace, and pull requests that show the Hubble output before and after.

Here is the trade-off. Identity-aware policy is sharper than an IP list, and it snaps when labels are sloppy. A team that renames app: checkout on a whim has edited a firewall on a whim, whether anyone noticed or not. Freeze the labels that carry identity, or generate them from the platform, so no microservice can rename itself into a hole.

DNS-aware rules are strong and easy to get wrong. Someone allows *.amazonaws.com because one SDK needed S3, and half the cloud is reachable again. Prefer exact FQDNs (fully qualified domain names, the complete api.vendor.com form) or patterns your platform generates from declared dependencies. Give a DNS allow the same scrutiny you give a CIDR allow. It is the same egress gate wearing a friendlier name.

If every flow from a pod is allowed because no policy selects it, that pod is unsupervised. Do not read the green as healthy. The workload is sitting outside your stated intent. Measure coverage instead: the share of pods in sensitive namespaces that at least one policy selects. That number tells you more than any screenshot of a busy flow map.

Policy without Hubble is guesswork. In the middle of an incident you want a list of denied flows with pod identities on them, not a feeling that something is blocked. Keep one staging namespace permanently on default-deny so your pipeline exercises the allow rules before they ever stand in front of production.

Try this

Run this on a lab cluster or a single staging node. Read what comes back, and resist the urge to edit production policy off the back of one look.

$ kubectl -n kube-system get ds cilium
NAME     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
cilium   3         3         3       3            3
$ cilium status --brief
OK   CP: Ok   EP: 48   IPAM: Ok
$ hubble observe --pod payments --last 3
Jul 24 04:50:01 denied egress payments -> 1.2.3.4:443

Takeaway

Cilium makes identity-aware network policy something a team can operate, with eBPF doing the enforcement and Hubble showing the result. Default-deny plus a named owner on every allow beats a tangle of wide rules nobody remembers writing.

Next step: put one namespace on default-deny with explicit allows, confirm each one against Hubble flows, then look at how enforcement modes get rolled out without taking the cluster down.

Related