>_SecOpsLog_
CoursesKubernetes fundamentalsScaling up and down

Scaling up and down

More copies, fewer copies, on demand.

Beginner8 min · lesson 9 of 24
In plain terms
Scaling is turning a dial: more customers, spin up more copies; a quiet night, spin some down. Same app, just more or fewer hands on deck.

A lunch counter at noon has six people working behind it. By three in the afternoon, two. Same menu, same kitchen, same griddle. The only thing that changed is how many pairs of hands are on the line. Scaling in Kubernetes is that idea, boiled down to a single number you can type.

You built a Deployment a couple of lessons back. A Deployment is the Kubernetes object that runs your app and keeps an agreed number of copies alive. Each copy is a Pod, the smallest thing Kubernetes runs (one Pod is one wrapped-up running instance of your app). The number of copies you asked for has a name: the replica count. Scaling means changing that number, then letting Kubernetes add or delete Pods until what is running matches what you asked for.

Turning the number up and down

There are two ways to change the count. The quick way is one command. You name the Deployment, name the new number, and Kubernetes handles the rest. Say hello is running 3 copies and the queue outside says you need 5.

Terminal — scale up to 5
kubectl scale deployment/hello --replicas=5
Output
deployment.apps/hello scaled

That word scaled means one thing only: Kubernetes wrote down the new target. It says nothing about whether the Pods are up and serving traffic. So go and look. The -l app=hello part narrows the list to Pods carrying the label app=hello, which is how you ask for this app's copies and nothing else.

Terminal — see the new Pods
kubectl get pods -l app=hello
Output
NAME                     READY   STATUS    RESTARTS   AGE
hello-7d9c6f8b4-2xk9p    1/1     Running   0          6m
hello-7d9c6f8b4-8ftll    1/1     Running   0          6m
hello-7d9c6f8b4-c4v2m    1/1     Running   0          6m
hello-7d9c6f8b4-lp7qz    1/1     Running   0          15s
hello-7d9c6f8b4-wm5rn    1/1     Running   0          15s

Five Pods. The columns read like this. READY 1/1 means the one container inside that Pod is up. STATUS Running is what you want to see. AGE is where the story sits: three Pods have been alive 6 minutes, two for 15 seconds. Those two youngsters are the copies Kubernetes started the moment you asked for 5.

Here is the machinery under that command. A Deployment does not handle Pods itself. It owns a ReplicaSet, a small controller with one job: keep an agreed number of Pods running. You moved that number to 5. The ReplicaSet counted 3, saw it was two short, and started two more. Scale the other way and it deletes the surplus instead. Compare what you want against what you have, close the gap, repeat. That loop is the whole of scaling.

What happens when you scale up
1You set replicasto 5One number changes2Deployment updatesits ReplicaSetNew target is 53ReplicaSetcompares countsHas 3, wants 54It starts 2 newPodsPlaced onto machines with room5Pods reach Runningand ReadyThey start receiving traffic

The quick command comes with a catch, and it is better to meet it here than at 2am. Your hello.yaml file on disk still says replicas: 3. (YAML is the plain-text file format Kubernetes uses to describe things.) Run apply against that file again and the count drops straight back to 3, because making the cluster match the file is exactly what apply is for. That is the declarative habit: the file is the source of truth, and the cluster gets dragged into line with it. So for a change you want to keep, edit the file. Here is the full Deployment with the count moved to 5.

hello.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  labels:
    app: hello
spec:
  replicas: 5
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
        - name: hello
          image: nginxdemos/hello:plain-text
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 100m
            limits:
              cpu: 250m
Terminal — apply the file
kubectl apply -f hello.yaml
Output
deployment.apps/hello configured

The word configured tells you the Deployment was already there and Kubernetes edited it to match your file.

Now a wall you will hit sooner or later. A replica count is a number you type, but every copy has to land on a real machine. Ask for more than your cluster has room for and Kubernetes will not argue with you. It writes the number down and then quietly fails to place them all.

Terminal — scale past capacity
kubectl scale deployment/hello --replicas=8
Output
deployment.apps/hello scaled

The same cheerful scaled line, because all the command did was record a target. Now look at the Pods.

Terminal — some Pods can't be placed
kubectl get pods -l app=hello
Output
NAME                     READY   STATUS    RESTARTS   AGE
hello-7d9c6f8b4-2xk9p    1/1     Running   0          18m
hello-7d9c6f8b4-8ftll    1/1     Running   0          18m
hello-7d9c6f8b4-c4v2m    1/1     Running   0          18m
hello-7d9c6f8b4-lp7qz    1/1     Running   0          12m
hello-7d9c6f8b4-wm5rn    1/1     Running   0          12m
hello-7d9c6f8b4-q4h2n    1/1     Running   0          22s
hello-7d9c6f8b4-rb8dz    0/1     Pending   0          22s
hello-7d9c6f8b4-t7k2s    0/1     Pending   0          22s

Six copies are Running and two are sitting at Pending. Pending means the Pod exists on paper but no machine has been picked to run it. To find out why, describe one of the stuck Pods and read the Events at the bottom of the output.

Terminal — ask why it's stuck
kubectl describe pod hello-7d9c6f8b4-rb8dz
Output — the Events at the bottom
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  24s   default-scheduler  0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

The scheduler is the part of Kubernetes that picks a machine for each Pod. It walked every node, found none with a spare 100m of CPU (the amount each copy requests), and stopped there. The Pod is not broken. It is unplaced, and it will keep waiting. You clear it by making room: add a node, lower the CPU request, or scale back down now the rush is over.

Terminal — scale back down
kubectl scale deployment/hello --replicas=2
Output
deployment.apps/hello scaled

Scaling changes the replica count on a Deployment (or on a ReplicaSet directly). It works sideways, which people call horizontal: you get more copies, never a bigger copy. Making one Pod bigger means raising its requests and limits, which is vertical sizing and a lesson of its own. For a stateless web app, reach for more copies when traffic spikes rather than pouring more CPU into a single one.

Autoscaling, which you set up at the end of this lesson, is the same knob with a machine turning it, driven by measurements instead of by your judgement. Typing kubectl scale by hand is still what you do in an incident when you need capacity in the next thirty seconds. Write down why you scaled, in the ticket or the incident channel, so the next person on call does not inherit a replica count of 12 with no story attached to it.

One warning for flash-sale day. Adding Pods to your app adds nothing to the database sitting behind it. If 200 connections already have that database on its knees, going from 6 copies to 20 moves the outage one hop down the chain instead of ending it. Scale the app, then go and watch how saturated everything it depends on is.

Try this

Do this on your own cluster. Scale a Deployment up, then back down, and watch the Pod count follow. Keep an eye on Ready against Desired while the new Pods come up.

terminal
$ kubectl create deployment scale --image=nginx:1.27 --replicas=1
deployment.apps/scale created
 
$ kubectl scale deployment/scale --replicas=4
deployment.apps/scale scaled
 
$ kubectl get deploy,pods -l app=scale
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/scale   4/4     4            4           20s
NAME                         READY   STATUS    RESTARTS   AGE
pod/scale-…                  1/1     Running   0          20s
... (4 pods)
 
$ kubectl scale deployment/scale --replicas=2
deployment.apps/scale scaled
 
$ kubectl get pods -l app=scale
NAME          READY   STATUS        RESTARTS   AGE
scale-…       1/1     Running       0          45s
scale-…       1/1     Running       0          45s
scale-…       0/1     Terminating   0          25s
 
$ kubectl delete deployment scale
deployment.apps "scale" deleted

Takeaway

kubectl scale moves one number: how many copies of your app run. It buys you more Pods, never a roomier Pod. Use it on purpose during an incident, trust the Ready count rather than the word scaled, and expect whatever your app talks to be the next thing that buckles.

Extra copies only help if any copy can answer any request
Scaling out assumes a request can land on any copy and get the same answer back. For web servers and APIs that is normally true. It breaks the moment one copy writes uploaded files to its own local disk, or holds a signed-in user's session in its own memory. The next request goes to a different copy, that data is not there, and people see strange, inconsistent results: an upload that vanished, a login that keeps dropping. Keep copies stateless by pushing files to shared storage and sessions to a database or cache. Do that and adding copies, plus autoscaling later, works without any of that pain.

Let the cluster scale for you

Turning the dial by hand is fine until you have to do it at every hour of the day. A thermostat solves the same problem for heating. You do not stand next to the boiler flipping a switch. You set a target temperature and the thermostat holds it, firing up when the room gets cold and easing off when it warms. A Horizontal Pod Autoscaler (HPA for short, the controller that changes your replica count for you) is a thermostat for Pods. You hand it a target, a floor and a ceiling, and it adds or removes replicas on its own to hold that target.

The one below watches CPU use across your Pods. CPU means the processor, so this is a reading of how hard your app is working. The HPA tries to hold that average near 60 percent, and it is fenced in on both sides: never fewer than 2 Pods, never more than 10. It is a complete file you can apply as it stands, and it aims at the hello Deployment by name.

hello-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: hello
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: hello
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
Terminal — turn on autoscaling
kubectl apply -f hello-hpa.yaml
Output
horizontalpodautoscaler.autoscaling/hello created
Terminal — check what the HPA sees
kubectl get hpa
Output
NAME    REFERENCE          TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
hello   Deployment/hello   cpu: 12%/60%    2         10        2          45s

Read the TARGETS column as current over goal. Here cpu: 12%/60% says the average Pod is using 12 percent of the CPU it asked for, against a goal of 60. Load is quiet, so the HPA parks on its floor of 2 Pods. Send real traffic at the app and that first number climbs. Once it crosses 60, the HPA starts adding replicas, up to its ceiling of 10, then takes them away again as the traffic drains. That is real elasticity: you run the capacity today needs, not the capacity one busy Friday a year might need.

Two things have to be true already or the HPA will sit there doing nothing at all. Your Pods need a CPU request (the resources.requests.cpu line in hello.yaml), because 60 percent has to be 60 percent of something. And your cluster needs metrics-server, a small add-on that reports each Pod's CPU and memory use. Without the request the HPA has no baseline to divide by. Without metrics-server it cannot see usage in the first place. Both are already in the files above.

Quick check
01During a quiet spell you run kubectl scale deployment/hello --replicas=2, but hello.yaml on disk still says replicas: 5. An hour later a colleague runs kubectl apply -f hello.yaml. What happens?
Incorrect — apply always pulls the cluster back to what the file says. It never starts ignoring it.
Correct — The file is what apply reconciles to, so the count climbs straight back to 5.
Incorrect — There is no conflict to report. apply sets the target to whatever number the file carries.
Incorrect — apply edits the Deployment in place. A changed value never triggers a delete.
02You apply the Horizontal Pod Autoscaler (HPA) above, aiming at 60% CPU, and the replica count never moves. Two things have to be in place or the HPA does nothing. Which pair?
Correct — Without a CPU request there is no baseline for 60 percent of anything, and without metrics-server the HPA cannot read usage.
Incorrect — A CPU-target HPA leans on neither of those.
Incorrect — The HPA takes the replicas count over, so pinning one is not a prerequisite.
Incorrect — Readiness probes and node count feed nothing in the HPA's CPU calculation.
03You scale hello to 8 replicas. Six Pods are Running, two sit in Pending, and describing one shows 'FailedScheduling ... Insufficient cpu.' What is going on?
Incorrect — A Pending Pod has not started a container yet, so nothing is crashing and there are no logs waiting.
Incorrect — A failed image pull shows as ImagePullBackOff, not Pending with FailedScheduling.
Correct — The scheduler found no machine with room for the CPU each Pod requests, so they wait. Not broken, unplaced.
Incorrect — Time alone does not cure it. They stay Pending until some node has room.

You have a pool of copies sharing the work now, and their names change every time that number moves. Nobody calling your app should have to keep track of that. What they need is one steady address sitting in front of all the copies, which is exactly what a Service is, and setting one up is next.

Deployments: the one you will useSelf-healing

Related

Cluster architectureKubernetes administrationSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep