Services: a stable address for Pods

ClusterIP, NodePort and LoadBalancer.

Beginner25 min · lesson 8 of 21
Lesson files
The scripts, test data and local test servers this lesson uses, exactly as they ran on the lab machine (5 files, 1 KB): services.tar.gz. The lab VM shares no folders with your computer, so fetch them inside the VM: cd ~/lab && curl -fsSLO https://secopslog.com/lab-files/k8s-fund/services.tar.gz && tar -xzf services.tar.gz, which creates ~/lab/services/. SHA-256: ce7ae12386affc00ab0cc302d048979edea82bcdc6ac585c9f591200890fc561

Say a shop front talks to a cart app at 10.244.1.3, the IP of the cart Pod. Then the cart Pod is replaced, as Pods often are, and its replacement comes up at a different address. The shop front now sends every request to an IP nobody answers on. With three cart Pods it would also have to decide which one to call.

A Service is the answer to that. It is one address that stays put, and behind it Kubernetes keeps an up-to-date list of the Pods that should get the traffic. First, see the problem for yourself.

Pod IPs do not last

This Deployment runs three nginx Pods. The args line writes the Pod's own name into the page nginx serves, so later you can see which Pod answered.

web.yaml
# Three nginx Pods. Each one writes its own name into the page it serves,
# so you can tell which Pod answered a request.
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 3
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: nginx
image: nginx:1.30.6-alpine
command: ["/bin/sh", "-c"]
args: ['echo "answered by $(hostname)" > /usr/share/nginx/html/index.html && exec nginx -g "daemon off;"']
ports:
- containerPort: 80
ubuntu@secopslog-k8s:~/lab/services · Kubernetes 1.37
$ kubectl apply -f web.yaml && kubectl rollout status deployment/web
deployment.apps/web created Waiting for deployment "web" rollout to finish: 0 out of 3 new replicas have been updated... Waiting for deployment "web" rollout to finish: 0 of 3 updated replicas are available... Waiting for deployment "web" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment "web" rollout to finish: 2 of 3 updated replicas are available... deployment "web" successfully rolled out
$ kubectl get pods -l app=web -o custom-columns=NAME:.metadata.name,IP:.status.podIP,NODE:.spec.nodeName
NAME IP NODE web-8597c5b645-9w8nf 10.244.2.3 lab-worker web-8597c5b645-wqwz8 10.244.2.4 lab-worker web-8597c5b645-zklvg 10.244.2.2 lab-worker
$ kubectl delete pods -l app=web && kubectl rollout status deployment/web && kubectl get pods -l app=web -o custom-columns=NAME:.metadata.name,IP:.status.podIP,NODE:.spec.nodeName
pod "web-8597c5b645-9w8nf" deleted from default namespace pod "web-8597c5b645-wqwz8" deleted from default namespace pod "web-8597c5b645-zklvg" deleted from default namespace deployment "web" successfully rolled out NAME IP NODE web-8597c5b645-dpn8p 10.244.1.2 lab-worker2 web-8597c5b645-n65kc 10.244.1.3 lab-worker2 web-8597c5b645-s5bx6 10.244.2.5 lab-worker

Compare the two tables. After kubectl delete pods, the Deployment made three new Pods with new names, and the IP column is different too. Nothing that remembered the old addresses would still work.

One stable address

A Service picks its Pods with a label selector, as lesson 7 (Labels) showed: here the app: web label the Deployment puts on every Pod. port is what clients connect to, targetPort is the port inside the Pod.

web-svc.yaml
# One stable address in front of every Pod labelled app=web.
apiVersion: v1
kind: Service
metadata:
name: web
spec:
selector:
app: web
ports:
- port: 80
targetPort: 80
ubuntu@secopslog-k8s:~/lab/services · Kubernetes 1.37
$ kubectl apply -f web-svc.yaml && kubectl get service web
service/web created NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE web ClusterIP 10.96.41.168 <none> 80/TCP 0s
$ kubectl get endpointslices -l kubernetes.io/service-name=web
NAME ADDRESSTYPE PORTS ENDPOINTS AGE web-nx8pq IPv4 80 10.244.1.3,10.244.2.5,10.244.1.2 0s

The Service is type ClusterIP, the default. Its CLUSTER-IP is a virtual address: no Pod or node owns it, and it only works from inside the cluster. The second command shows where the traffic really goes. Kubernetes keeps an EndpointSlice for each Service, and its ENDPOINTS column holds exactly the three Pod IPs from the table above.

Now call the Service from inside the cluster. kubectl run --rm -i starts a one-off busybox Pod, runs a command in it, prints the output and deletes the Pod again. The loop asks for the page six times:

ubuntu@secopslog-k8s:~/lab/services · Kubernetes 1.37
$ IP=$(kubectl get service web -o jsonpath="{.spec.clusterIP}") kubectl run client --rm -i --restart=Never --image=busybox:1.38.0 --env="IP=$IP" -- sh -c 'sleep 2; for i in 1 2 3 4 5 6; do wget -qO- -T 5 http://$IP; done'
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt. If you don't see a command prompt, try pressing enter. answered by web-8597c5b645-dpn8p answered by web-8597c5b645-n65kc answered by web-8597c5b645-s5bx6 answered by web-8597c5b645-n65kc answered by web-8597c5b645-s5bx6 answered by web-8597c5b645-s5bx6 pod "client" deleted from default namespace

The first two lines are kubectl's usual notes about attaching to a Pod. After a two-second pause (the sleep 2 gives kubectl time to attach before the loop prints anything) six answers arrived, from different Pods, although the client only ever used the one Service IP. kube-proxy, a program on every node, spreads connections across the endpoints. Which Pod answers is down to chance, so your list will look different. The last line is --rm cleaning up.

The endpoints follow the Pods

Replace all the Pods again and look at the Service and its EndpointSlice. Then ask for five Pods instead of three with kubectl scale, which lesson 9 (Scaling) covers in depth:

ubuntu@secopslog-k8s:~/lab/services · Kubernetes 1.37
$ kubectl delete pods -l app=web && kubectl rollout status deployment/web && kubectl get service web && kubectl get endpointslices -l kubernetes.io/service-name=web
pod "web-8597c5b645-dpn8p" deleted from default namespace pod "web-8597c5b645-n65kc" deleted from default namespace pod "web-8597c5b645-s5bx6" deleted from default namespace deployment "web" successfully rolled out NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE web ClusterIP 10.96.41.168 <none> 80/TCP 7s NAME ADDRESSTYPE PORTS ENDPOINTS AGE web-nx8pq IPv4 80 10.244.2.7,10.244.1.4,10.244.2.8 7s
$ kubectl scale deployment web --replicas=5 && kubectl rollout status deployment/web && kubectl get endpointslices -l kubernetes.io/service-name=web
deployment.apps/web scaled Waiting for deployment "web" rollout to finish: 3 of 5 updated replicas are available... Waiting for deployment "web" rollout to finish: 4 of 5 updated replicas are available... deployment "web" successfully rolled out NAME ADDRESSTYPE PORTS ENDPOINTS AGE web-nx8pq IPv4 80 10.244.2.7,10.244.1.4,10.244.2.8 + 2 more... 8s

The CLUSTER-IP is the same as before. The EndpointSlice kept its name but now lists three new addresses. Nobody edited it: the EndpointSlice controller watches Pods that match the selector and rewrites the list as they come and go. After scaling to five, the list has five entries; kubectl shows three and sums up the rest as + 2 more.... Each endpoint also carries a ready flag, and a Pod gets traffic only while its endpoint is marked ready. Lesson 13 (Probes) shows how an app decides that.

NodePort: a port on every node

A ClusterIP Service cannot be reached from outside the cluster. Type NodePort adds a port, here 30080, that is opened on every node and forwards to the same Pods. NodePorts must come from the range 30000-32767. Leave nodePort out and Kubernetes picks a free one.

web-nodeport.yaml
# The same Pods, also reachable on port 30080 of every node.
apiVersion: v1
kind: Service
metadata:
name: web-nodeport
spec:
type: NodePort
selector:
app: web
ports:
- port: 80
targetPort: 80
nodePort: 30080
ubuntu@secopslog-k8s:~/lab/services · Kubernetes 1.37
$ kubectl apply -f web-nodeport.yaml && kubectl get service web-nodeport
service/web-nodeport created NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE web-nodeport NodePort 10.96.1.130 <none> 80:30080/TCP 0s
$ kubectl get nodes -o custom-columns=NAME:.metadata.name,INTERNAL-IP:.status.addresses[0].address
NAME INTERNAL-IP lab-control-plane 172.18.0.4 lab-worker 172.18.0.3 lab-worker2 172.18.0.2
$ NODE_IP=$(kubectl get node lab-control-plane -o jsonpath="{.status.addresses[0].address}") curl -s --retry 5 --retry-connrefused --retry-delay 1 http://$NODE_IP:30080
answered by web-8597c5b645-z9zps

The PORT(S) column now reads 80:30080/TCP: port 80 on the cluster IP, port 30080 on the nodes. The curl went to the control-plane node, which runs none of the web Pods, and still got an answer from one of them. Every node forwards its node port to wherever the Pods are. kube-proxy needs a second or two to open a new node port, and a curl sent straight after kubectl apply can be refused. That is why the command retries. kind nodes are containers on a Docker network, so their IPs are reachable from the machine that runs them.

LoadBalancer: waiting for a cloud

On AWS, Google Cloud or Azure, type LoadBalancer asks the cloud for a real load balancer with a public address, which then sends traffic to the node ports. A kind cluster has no cloud behind it:

web-lb.yaml
# Asks the cluster for an external load balancer.
apiVersion: v1
kind: Service
metadata:
name: web-lb
spec:
type: LoadBalancer
selector:
app: web
ports:
- port: 80
targetPort: 80
ubuntu@secopslog-k8s:~/lab/services · Kubernetes 1.37
$ kubectl apply -f web-lb.yaml && kubectl get service web-lb
service/web-lb created NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE web-lb LoadBalancer 10.96.189.163 <pending> 80:32096/TCP 0s
$ kubectl describe service web-lb | tail -n 4
Session Affinity: None External Traffic Policy: Cluster Internal Traffic Policy: Cluster Events: <none>

EXTERNAL-IP says <pending> and stays that way. Kubernetes did its part: the Service got a cluster IP and a node port picked at random (80: followed by a port number in PORT(S)), because a LoadBalancer Service is built on top of the other two types. The missing piece is the program that talks to a cloud and fills in the address, and kind does not ship one. There are no events either, because nothing is even trying. A separate tool, cloud-provider-kind, can play that role on a laptop; this course does not need it.

TypeReachable fromTypical use
ClusterIPInside the cluster onlyOne app calling another
NodePortAny node's IP, port 30000-32767Testing, or behind your own load balancer
LoadBalancerAn address the cloud hands outPublic traffic on a cloud cluster

Hands-on

You do not always need a YAML file for a Service. kubectl expose copies a Deployment's selector into a new one.

1. Create a NodePort Service called web2 and look at it: kubectl expose deployment web --name=web2 --port=80 --type=NodePort && kubectl get service web2.

2. Read its node port and the IP of lab-worker2 into two variables, then curl them: PORT=$(kubectl get service web2 -o jsonpath="{.spec.ports[0].nodePort}"), then NODE_IP=$(kubectl get node lab-worker2 -o jsonpath="{.status.addresses[0].address}"), then curl -s --retry 5 --retry-connrefused --retry-delay 1 http://$NODE_IP:$PORT.

3. Scale the Deployment to zero, wait until its Pods are deleted, and look at the EndpointSlice: kubectl scale deployment web --replicas=0 && kubectl wait --for=delete pod -l app=web --timeout=90s && kubectl get endpointslices -l kubernetes.io/service-name=web2. kubectl wait --for=delete returns once the matching objects are gone.

4. In the same shell, curl again without retries: curl -sS --max-time 5 http://$NODE_IP:$PORT.

5. Clean up: kubectl delete deployment web && kubectl delete service web web-nodeport web-lb web2 && kubectl get all.

What you should see: in step 1, web2 gets a random node port between 30000 and 32767 (80:3xxxx/TCP). Step 2 prints answered by and the name of one Pod. In step 3 each of the five Pods reports condition met, and the EndpointSlice for web2 shows <unset> under both PORTS and ENDPOINTS. Step 4 fails with curl: (7) Failed to connect ... Could not connect to server: the Service still exists, but no Pod matches its selector, so the connection is refused. After step 5, kubectl get all lists only the built-in kubernetes Service.

Remember

Pods come and go with new IPs, so clients talk to a Service instead. The Service keeps one address, and its EndpointSlice lists the Pods that match its selector, sending traffic only to those marked ready. NodePort and LoadBalancer build on a ClusterIP, and a LoadBalancer needs a cloud to hand out its address.

Quick check
01A rollout replaced every Pod behind a Service. A client that uses the Service's cluster IP keeps working. Why?
Incorrect — Replacement Pods get new IPs. Both tables in the lesson changed.
Correct — The Service kept its address while the EndpointSlice switched to the new Pod IPs.
Incorrect — The client only ever knew the Service IP. Kubernetes did the bookkeeping.
Incorrect — Old Pods never come back. A replacement is always a new Pod.
02A Service shows <unset> under ENDPOINTS. What is the most likely reason?
Incorrect — The type decides who can reach the Service, not which Pods it finds.
Incorrect — The API server assigns the cluster IP when the Service is created.
Incorrect — Kubernetes creates and updates one for every Service with a selector.
Correct — Scaling to zero in the hands-on emptied the list the same way. Matching Pods that are not ready would still be listed.
03On a kind cluster you create a Service of type LoadBalancer. What happens?
Correct — It got a cluster IP and a node port. Nothing on kind fills in the external address.
Incorrect — The Service was created without any error.
Incorrect — kind ships nothing that assigns addresses, so it stayed <pending>.
Incorrect — The type stays LoadBalancer. It includes a node port, as every LoadBalancer Service does.

Related