Why the perimeter fails

Flat interiors, no clean edge, ephemeral workloads.

Advanced25 min · lesson 3 of 15

One security desk in the lobby. You badge in once, the turnstile clicks, and after that every floor is yours: the mail room, the server closet, the finance floor, the cabinet with payroll in it. Nobody asks who you are again, because the lobby already asked. Corporate networks took the same bet. A firewall and a VPN (virtual private network, an encrypted tunnel that drops your laptop onto the internal network) make a hard shell at the edge, and behind that shell everything trusts everything. The usual name for it is the castle-and-moat model, or the perimeter model.

For one data centre and forty servers, that was a fair trade. It stopped being fair, and walking through why is not history for its own sake. Every specific way the perimeter breaks turns into a specific control you build later in this course. So meet the failures the way an attacker meets them, at a shell prompt, rather than as bullets on a slide.

The interior is flat

Three words of vocabulary, then the demo. North-south traffic is the front door: a browser hitting your load balancer, your cluster calling a payment provider. East-west traffic is the corridor between offices, service talking to service on the inside, where the perimeter firewall never sees a packet. Lateral movement is what an attacker does with that corridor once they own one room, walking from the box they got to the box they wanted. In a microservice estate the corridor carries most of the traffic. One click in a browser fans out into a dozen internal calls, and only the first of them crossed the edge. The firewall inspects that one and waves the rest through by never seeing them.

Here is a cluster running an online shop. Start with the two numbers that describe your blast radius, meaning how much an attacker gets from a single foothold. Both come out of kubectl, the command-line client for Kubernetes.

terminal
$ kubectl get networkpolicies --all-namespaces
$ kubectl get svc --all-namespaces --no-headers | wc -l
output
No resources found
87

Eighty-seven services reachable, zero policies about who may reach them. Nobody misconfigured that. It is the Kubernetes default: every pod (one or more containers sharing a single address on the cluster network) can open a connection to every other pod, on every port, in every namespace (a namespace is a folder that groups objects and their names), until a NetworkPolicy object says otherwise. Nothing here has said otherwise.

You do not have to wait for a breach to see what that buys an attacker. Attach a debug container to the front-end pod. Every container in a pod shares one network namespace, so this shell sees the network exactly as code running inside that pod does, which is exactly what an attacker with code execution there gets. The image is nicolaka/netshoot, a container packed with network tools. The general profile is the default in current kubectl and grants no privileges beyond the pod's own; netadmin and sysadmin grant more, and you do not need them here.

terminal
$ kubectl debug -n shop -it frontend-7d9f8b6c4-q7x2p \
--image=nicolaka/netshoot --profile=general -- bash
output
Defaulting debug container name to debugger-5mzqp.
If you don't see a command prompt, try pressing enter.
frontend-7d9f8b6c4-q7x2p:~#

That prompt is worth a second glance. Netshoot sets its prompt to the hostname, and a pod's hostname is its pod name, so the shell is telling you which pod you are standing inside. Now knock on three doors that a web front end has no reason to knock on.

terminal
frontend-7d9f8b6c4-q7x2p:~# nc -zv -w2 payments-db 5432
frontend-7d9f8b6c4-q7x2p:~# nc -zv -w2 redis-sessions 6379
frontend-7d9f8b6c4-q7x2p:~# curl -s -m2 http://admin-internal/actuator/env \
| jq -r '.propertySources[].name' | head -4
output
Connection to payments-db (10.0.14.7) 5432 port [tcp/postgresql] succeeded!
Connection to redis-sessions (10.0.9.22) 6379 port [tcp/redis] succeeded!
server.ports
systemProperties
systemEnvironment
Config resource 'class path resource [application-prod.yml]' via location 'optional:classpath:/'

The front end renders web pages. It has no business with the payments database, none with the session store, and none with an admin service that hands over its own configuration to anyone who asks. All three answered on the first try, in under a second. That last one is Spring Boot Actuator, a bundle of management endpoints; only the health check is published over HTTP by default, so somebody widened the exposure setting to everything during a bad afternoon and never narrowed it again. Nothing raised an alarm anywhere, because from the network's point of view nothing unusual happened. A pod talked to a pod, which is what pods do here.

Reaching a service counts as permission to use it

Reachability is half the problem. The other half is that arriving at a service gets treated as being allowed to use it, the way a hotel corridor treats a door with no lock: turning the handle is the whole check. Internal services get shipped with no password and no encryption all the time, on the reasoning that the traffic never leaves the network. Two lines of typed text are enough to price that reasoning.

terminal
frontend-7d9f8b6c4-q7x2p:~# printf 'PING\r\nDBSIZE\r\n' | nc -q1 redis-sessions 6379
output
+PONG
:41273

Those two lines are RESP (the Redis Serialization Protocol, the plain-text format Redis speaks on the wire), typed by hand into a raw socket. The +PONG means the server answered. The :41273 is an integer reply, the count of keys in that database, which here is the number of live sessions. No password, no client certificate, no question about who was typing. Notice what the reply also settles about encryption. A port doing TLS (Transport Layer Security, the encryption behind the padlock in your browser) cannot answer plain text, because it would be waiting for a handshake first. This one answered plain text, so nothing on this hop is encrypted, and anything that can read packets between those two pods reads sessions. Redis behaves this way once someone turns off protected mode and sets no password, which is the standard fix the first time a service in another pod cannot connect.

The session store cannot tell a legitimate caller from a hostile one, because it never asked. It can write a source address into a log, and the next section is about how little that address is worth.

An address is a location, not a name

Perimeter-era controls write trust down as address ranges: allow 10.0.14.0/24 to reach the database on port 5432. That notation is a CIDR block (Classless Inter-Domain Routing, the /24 shorthand for a run of addresses), and the rule only means something if an address is a stable stand-in for a workload, the way a house number stands in for the family that has lived there thirty years. In a cluster, an address is a hotel room number. It tells you who is in 412 tonight. Tomorrow it belongs to somebody else.

Watch an ordinary deploy break the assumption.

terminal
$ kubectl -n shop get pods -l app=payments \
-o custom-columns=NAME:.metadata.name,IP:.status.podIP,NODE:.spec.nodeName
$ kubectl -n shop rollout restart deploy/payments
$ kubectl -n shop rollout status deploy/payments
$ kubectl -n shop get pods -l app=payments \
-o custom-columns=NAME:.metadata.name,IP:.status.podIP,NODE:.spec.nodeName
output
NAME IP NODE
payments-7c9d64f8b5-8k2mq 10.0.31.4 node-2
deployment.apps/payments restarted
Waiting for deployment "payments" rollout to finish: 1 old replicas are pending termination...
deployment "payments" successfully rolled out
NAME IP NODE
payments-5df9c74b86-q4xzt 10.0.22.31 node-5

The database allowlist that names 10.0.31.4 now points at nothing, and the payments service it was written for is locked out. That is the harmless failure, the one that pages somebody at 3am and gets fixed by breakfast. Here is the other one.

terminal
$ kubectl get pods -A --field-selector status.podIP=10.0.31.4 \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,NODE:.spec.nodeName,START:.status.startTime
output
NS NAME NODE START
analytics spark-exec-7f2c9-2 node-2 2026-07-21T10:31:06Z

Eleven minutes after the payments pod let that address go, the pool handed it to a Spark executor (Apache Spark, a batch engine that runs whatever query somebody scheduled) in a different namespace. Same node, which is the ordinary way it happens: most address managers give each node its own block and refill it from the addresses that node released. Your rule still says 10.0.31.4 may reach the database on 5432, so it may. An address rule fails in both directions at once. It goes stale, because the workload you meant to allow moved. It turns dangerous, because the hole you left open now belongs to something you never meant to allow.

Push past the cluster and the edge stops being a line at all. Workloads span clusters, VPCs (virtual private clouds, your own fenced-off slice of a provider's network) and providers. One request can cross three networks you only partly control. The managed database you migrated to is reached over the provider's network, not yours. Cloud firewalls make the address problem worse in a way that catches people out. When a pod talks to something outside the cluster, most setups source-NAT the traffic (network address translation, rewriting the packet's source address on the way out) to the node's address. So the database's firewall rule allowlists a node, or a whole subnet of nodes. Every pod scheduled on that node inherits the allow. The front end inherits it. The Spark executor inherits it. So does whatever an attacker lands on next.

Where the perimeter breaks, and what replaces it
Perimeter assumption
inside is trusted
one foothold reaches all 87 services
east-west needs no proof
cleartext, and nobody asks who is calling
an address names a workload
until the next deploy recycles it
there is one clean edge
clusters, VPCs and managed services say otherwise
Zero-trust replacement
default-deny east-west
NetworkPolicy first, then mesh policy
mTLS on every hop
both ends present a certificate
a name the workload can prove
a private key it holds, not an address
federated trust domains
one identity across clusters and clouds
Read it row by row. Each item on the right is a later lesson in this course, and none of them deletes the firewall. What changes is that being inside stops counting as a credential.

Close the interior first

The cheapest control against all of the above is to stop the interior being flat. A default-deny NetworkPolicy works like a guest list taped to a door. The door was open to the whole building, and the moment a list appears, only the names on it get in. It is an ordinary Kubernetes object in the networking.k8s.io/v1 API group, no service mesh and no extra software required, and it is the first thing to reach for on day one.

default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
namespace: shop
spec:
podSelector: {} # empty selector = every pod in this namespace
policyTypes:
- Ingress # isolated for ingress, and no rule lets anything back in

State the semantics precisely, because they surprise people. NetworkPolicy has no deny rule at all. Policies are additive: a pod is unrestricted until some policy selects it, and the moment one does, that pod is isolated for the directions listed under policyTypes, after which only traffic matching an allow rule reaches it. An empty podSelector selects every pod in the namespace. An empty ingress list allows nothing. Put those two together and you have deny all inbound. One more thing to pin down: these are namespaced objects, so this one covers shop and nothing else. You need one per namespace.

terminal
$ kubectl apply -f default-deny.yaml
$ kubectl -n shop describe networkpolicy default-deny-ingress
output
networkpolicy.networking.k8s.io/default-deny-ingress created
Name: default-deny-ingress
Namespace: shop
Created on: 2026-07-21 10:42:18 +0000 UTC
Labels: <none>
Annotations: <none>
Spec:
PodSelector: <none> (Allowing the specific traffic to all pods in this namespace)
Allowing ingress traffic:
<none> (Selected pods are isolated for ingress connectivity)
Not affecting egress traffic
Policy Types: Ingress

describe telling you the object exists is not proof that anything enforces it. Go back to the debug shell and try the same hop again.

terminal
frontend-7d9f8b6c4-q7x2p:~# nc -zv -w3 payments-db 5432
output
nc: connect to payments-db (10.0.14.7) port 5432 (tcp) timed out: Operation in progress

Timed out, not refused. That difference is your verification, and it is worth learning to read. "Connection refused" means the packet arrived somewhere and something there answered no, like knocking and being told to go away. A timeout means the packets are dropped on the floor and nobody answers at all, which is silence at the door, and silence is what a working policy looks like from the attacker's side. The trailing "Operation in progress" is the C library's wording for a connection that never finished; on a glibc-based image the same message reads "Operation now in progress". Silence has a cost you should plan for: newly blocked calls surface in your application as slow requests rather than fast errors. Set client connect timeouts before you roll this out, or your first symptom will be thread pools filling up.

Default-deny egress silently breaks DNS
Add the egress twin of this policy (an empty podSelector with policyTypes [Egress]) and your pods can no longer reach CoreDNS, the cluster's DNS service (Domain Name System, the phone book that turns payments-db into an address) in kube-system, because a pod isolated for egress may only reach what an egress rule names. Calls then fail at the lookup rather than the connection, so it reads like a broken application instead of a policy you applied twenty seconds ago. On glibc-based images the message is "Temporary failure in name resolution"; Alpine and the JVM word it differently, which is why the symptom is often misfiled. Pair default-deny egress with an explicit allow to UDP and TCP port 53 toward the DNS pods, and roll it into one namespace before the rest. If your cluster runs NodeLocal DNSCache, queries go to a link-local address on the node instead of a pod, and a pod selector will never match it, so allow that address explicitly.
allow-dns-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: shop
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector: # same list item as the namespaceSelector,
matchLabels: # so BOTH must match: DNS pods in kube-system
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53

That policy allows name lookups and nothing else. Every other outbound flow you actually need, the payments API (Application Programming Interface, the set of calls a service publishes for other software to use), the object store, the provider's database endpoint, gets its own rule stacked on top.

The API accepts policies your CNI may never enforce
kubectl apply succeeds whether or not anything in the cluster implements NetworkPolicy. Enforcement is the job of the CNI (Container Network Interface, the plugin that wires up pod networking). Calico and Cilium enforce it. Plain flannel does not, and some managed clusters ship a data plane where enforcement is a switch somebody has to turn on. The object will exist, describe will look correct, and every packet will still flow, which makes it a house rule with nobody employed to enforce it. The only honest check is the empirical one: probe a flow you expect to be blocked and confirm it times out.

What a label proves, and what it does not

Default-deny is a real win and you should ship it this week. Be precise about what it proves. NetworkPolicy matches on pod labels, address blocks, ports and protocols, which is layers 3 and 4 (the addressing layer, and the layer of TCP and UDP, the two protocols that carry connections and datagrams). It has no idea who is on the far end of a connection. Anything wearing the label app: frontend is admitted, and a label is a sticker the pod writes on itself. Say you have since added an allow rule so the front end can reach payments again. Ask two questions about it.

terminal
$ kubectl auth can-i create pods -n shop --as=system:serviceaccount:shop:ci-deployer
$ kubectl -n shop get networkpolicy allow-frontend-to-payments \
-o jsonpath='{.spec.ingress[0].from[0].podSelector.matchLabels.app}'
output
yes
frontend

Read those two answers together. The rule admits app: frontend. Anything that can create a pod in that namespace can start a pod wearing that sticker and inherit the access without exploiting a single bug, and in most clusters the list of things that can create pods includes your CI deployer (continuous integration, the automation that builds and ships your code) and every token it holds. That is Kubernetes RBAC (Role-Based Access Control, the rules about which identity may perform which operation), not networking, which is exactly the point: your network rule is now only as strong as your deploy permissions. Underneath, the CNI turns labels into addresses in its data plane, and that translation lands after the pod does, so during heavy churn enforcement works from a picture that is a moment out of date. You have moved from trusting a location to trusting a self-written sticker. Better. Still nowhere near the caller proving anything.

The obvious next move is to encrypt east-west traffic with mTLS (mutual Transport Layer Security, where both ends present certificates instead of only the server, like two people showing each other ID rather than one shop showing its licence). Be exact about what that buys, because it gets oversold. A successful mTLS handshake proves two things: the peer holds the private key for a certificate your trust root vouches for, and the bytes in flight are encrypted and unaltered. Now the things it does not prove. It does not prove the peer is uncompromised, because a popped front end still holds a perfectly valid certificate and will use it happily. It does not decide whether that identity may call this particular endpoint. It says nothing about what the request is asking for. On its own, mTLS encrypts lateral movement rather than stopping it. Authorization is a separate control, and it gets its own lessons later.

Which leaves the question the rest of this course answers. If a workload's name is not an address and not a label, what is it? It has to be something the workload proves with a private key only it holds, that survives being rescheduled, that expires fast enough to be worth little once stolen, and that means the same thing in every cluster and cloud you run. It looks like spiffe://acme.internal/ns/prod/sa/payments, a trust domain plus a path, defined by SPIFFE (Secure Production Identity Framework For Everyone) and carried in a short-lived certificate called an X.509-SVID (SPIFFE Verifiable Identity Document). Before the next lesson, run the first two commands from this one against a cluster you actually operate. The service count is your blast radius today. The policy count is everything standing between one foothold and all of it.

Quick check
01Your cluster runs 87 Services and zero NetworkPolicies. An attacker gets code execution in one front-end pod and opens a connection to the payments database. What does the perimeter firewall do about it?
Incorrect — Backwards: the Kubernetes default is that every pod reaches every pod on every port until a NetworkPolicy selects it.
Incorrect — There is nothing to record, because those packets never touch the edge device.
Correct — a perimeter device only sees traffic that crosses it, and this interior has no controls of its own.
Incorrect — Pod-to-pod traffic between nodes rides the cluster network, not the perimeter firewall.
02You apply a policy with an empty podSelector, policyTypes: [Egress] and no egress rules. Within seconds the application logs fill with name-resolution failures such as "Temporary failure in name resolution". What happened?
Correct — default-deny egress cuts off CoreDNS first, and the symptom is a lookup error rather than a connection error.
Incorrect — An empty rule list is valid, and it is exactly how you express deny-all for that direction.
Incorrect — NetworkPolicy objects are namespaced, so this one selects pods in its own namespace and cannot touch kube-system.
Incorrect — Enforcement is connection-tracked, so replies to a permitted outbound flow return without any ingress rule.
03You applied default-deny-ingress in the shop namespace. kubectl says it was created, describe reports "Selected pods are isolated for ingress connectivity", and a probe from the front-end pod to payments-db:5432 still succeeds instantly. What is the right next step?
Incorrect — Enforcement lives in the cluster data plane and applies to running pods within seconds, so a restart proves nothing.
Incorrect — NetworkPolicy has no deny rule at all, which is why selecting a pod with an empty rule list is how you deny.
Incorrect — That widens the change, breaks DNS without an allow for port 53, and still does not explain why the ingress block is not enforced.
Correct — policies are namespaced, and plain flannel accepts the object while enforcing nothing, so either one makes a real policy invisible in the data plane.

Try this

Run kubectl get networkpolicies --all-namespaces on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: default-deny egress silently breaks DNS. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related