Secrets: handling passwords

Like config, but sensitive.

Two of the settings your app needs are not alike at all. The port your database listens on, 5432, could be printed on a poster in the lobby and nothing bad would happen. The database password could not. Kubernetes takes that difference seriously enough to give the sensitive half its own kind of record, called a Secret. (Your cluster, the group of machines Kubernetes runs your apps across, is what stores these records and hands them out.) Here is an easy way to keep the two straight. A ConfigMap, which is Kubernetes' record for plain, non-sensitive settings, is a note pinned to the office bulletin board where anyone walking past can read it. A Secret is that same note, sealed in an envelope and locked in a drawer. Inside, both hold ordinary key-and-value pairs. What changes is how carefully the cluster treats the envelope, and how carefully you have to treat it too.

Reach for a Secret the moment a value would do real damage if a stranger read it. A database password. An API key (application programming interface key), a token that lets your app prove who it is to another service, working a bit like a key cut for one specific door. A TLS certificate, TLS being short for Transport Layer Security, the file sitting behind the little padlock in your browser that proves a website is who it claims to be. The rule you met with plain config still holds here. Keep the value out of your container image. A container image is the frozen, ready-to-run snapshot your app ships as, and every server that runs the app pulls its own copy. Bake a password into that snapshot and you have handed the password to everyone who can pull the image, permanently. Put it in a Secret instead, and Kubernetes passes the value to the app at startup, from outside.

Make one without opening an editor

You talk to Kubernetes through a command-line tool called kubectl, meaning a tool you drive by typing commands instead of clicking buttons. Half the industry says 'cube-cuttle', half says 'cube-control', and nobody has ever settled it. For your first Secret you do not need to write a file at all. One command creates the record and handles the fiddly encoding on your behalf. The --from-literal flag is you saying: the value is right here, typed on the line. (A flag is an extra option you tack onto a command to change what it does.)

$ kubectl create secret generic db-secret --from-literal=DB_PASSWORD=supersecret

secret/db-secret created

Now ask the cluster to read back what it stored. The value comes out looking like line noise, and that appearance fools people constantly. It is not encrypted. Encryption means scrambling something so that only a person holding the right key can unscramble it. What you are looking at is base64, a reversible way of rewriting text so that awkward characters survive being copied from one system to another. Anyone allowed to read the Secret can turn that string back into the original password in one step. You are about to do exactly that.

$ kubectl get secret db-secret -o jsonpath='{.data.DB_PASSWORD}'

c3VwZXJzZWNyZXQ=

$ echo 'c3VwZXJzZWNyZXQ=' | base64 -d

supersecret

So a Secret is a ConfigMap with different defaults and a lot more caution built up around it. Follow the bytes and you see why the caution matters. They end up in etcd, the key-value database where Kubernetes keeps a record of everything it manages, and depending on how your cluster was set up they may sit there still only base64-encoded rather than properly encrypted. A Secret object is a labelled envelope, not a bank vault. For passwords guarding real production data, keep the master copy in a dedicated external secret store and switch on encryption at rest so the copy in etcd is scrambled as well.

That leads to one hard rule about files. Never commit a Secret manifest holding a live value to Git. Running kubectl get secret -o yaml prints base64, and base64 in a commit is a costume, not a lock. The usual ways around it, in rough order of how often you will meet them: sealed-secrets, which lets you commit an encrypted blob only your cluster can open; a CSI driver (Container Storage Interface, the standard plug for attaching outside storage to a Pod) that fetches the value from a vault when the Pod mounts it; or your cloud provider's own secret manager.

And if a credential does leak, the order of operations matters. Rotate it first, at the source, so the old string stops working. Then patch the Secret with the new value and restart everything that consumes it, because a Pod that read the old value at startup is still holding it in memory. Last, audit RBAC (role-based access control, the Kubernetes permission system that decides who may read which objects) to work out who could have read that Secret in the first place. The object is only as safe as your RBAC rules and your storage encryption, and no safer.

Try it yourself

Run the whole loop end to end. Create a Secret, mount or inject it into a Pod, confirm the app really can see the value, and watch how asking the cluster for it only ever gives you base64. Clean up at the end so nothing is left lying around.

$ kubectl create secret generic db --from-literal=password=S3cr3t!
secret/db created
 
$ kubectl get secret db -o jsonpath='{.data.password}{"\n"}'
UzNjcjN0IQ==
 
$ echo UzNjcjN0IQ== | base64 -d
S3cr3t!
 
$ kubectl run sec --image=busybox:1.36 --restart=Never --overrides='{"spec":{"containers":[{"name":"sec","image":"busybox:1.36","command":["sleep","300"],"env":[{"name":"DB_PASSWORD","valueFrom":{"secretKeyRef":{"name":"db","key":"password"}}}]}]}}'
pod/sec created
 
$ kubectl exec sec -- printenv DB_PASSWORD
S3cr3t!
 
$ kubectl delete pod sec; kubectl delete secret db
pod "sec" deleted
secret "db" deleted

Takeaway

Sensitive values belong in a Secret rather than a ConfigMap, and keeping that line clean is worth the effort. The encoding is not what protects them. base64 comes off in a single command, as you saw. What actually guards the password is the short list of people and service accounts your RBAC rules let near that Secret, whether the cluster encrypts its data store, and whether the production credential lives in a dedicated store to begin with. Rotate the moment you suspect anyone saw it.

Hand the Secret to your Pod

Now let something actually use it. A Pod is the smallest thing Kubernetes runs for you: one or more containers wrapped together, placed onto a machine, sharing the same small sandbox. There are two ways in. You can mount the Secret as a file, where Kubernetes makes the value appear as a file inside the container at a path you choose. Or you can expose it as an environment variable, a named value the operating system hands your program the moment it starts, sitting there waiting to be read. The example below takes the environment-variable route. The file itself is a manifest, a YAML description of what you want the cluster to create. (YAML is a plain-text format for structured settings, built on names, values and indentation.) Look at stringData: writing the password there means Kubernetes does the base64 encoding for you, and you never type an encoded string by hand.

apiVersion: v1
kind: Secret
metadata:
  name: db-secret
type: Opaque
stringData:
  DB_PASSWORD: supersecret
---
apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  containers:
    - name: web
      image: nginx:1.27
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-secret
              key: DB_PASSWORD

$ kubectl apply -f app.yaml

secret/db-secret configured
pod/web created

Here comes the satisfying part. Step inside the running container and check that the password made it. The exec command runs a program inside a Pod for you, and printenv is a tiny utility that prints the value of one environment variable.

$ kubectl exec web -- printenv DB_PASSWORD

supersecret

DB_PASSWORD is now sitting in the app's environment exactly as if you had typed it there by hand, while the real value lives in the cluster and never in the image. Your code reads it the ordinary way, which in most languages is one line that looks up an environment variable by name. The app has no idea the value came from a Secret, and it does not need one. A caveat before you move on. Mounting a Secret as a file is usually the safer of the two routes, because environment variables have a habit of escaping: into log files, and down into every other program your app launches. Both routes work, and your code barely changes between them.

When the key name does not line up

One mistake catches nearly everyone in their first week, so it helps to meet it deliberately, on your own terms. The key you name under secretKeyRef has to match a key that really exists inside the Secret, spelled identically. Say your finger slipped and you wrote key: DB_PASS while the Secret holds DB_PASSWORD. Kubernetes takes the file happily and creates the Pod. The container never starts. Ask for the Pod's status and here is what comes back.

$ kubectl get pod web

NAME   READY   STATUS                       RESTARTS   AGE
web    0/1     CreateContainerConfigError   0          14s

CreateContainerConfigError is Kubernetes telling you it could not assemble the container's startup settings, because something they point at is missing. That names the symptom and stops there. For the cause, describe the Pod and read the Events list at the bottom, which is a play-by-play of everything that has happened to it.

$ kubectl describe pod web

Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  22s               default-scheduler  Successfully assigned default/web to node-1
  Normal   Pulled     6s (x3 over 21s)  kubelet            Container image "nginx:1.27" already present on machine
  Warning  Failed     6s (x3 over 21s)  kubelet            Error: couldn't find key DB_PASS in Secret default/db-secret

Read it top down, oldest line first. The scheduler, the part of Kubernetes that picks which machine a Pod runs on, did its job: Successfully assigned web to node-1. Then the kubelet, the agent running on that machine, hit the wall and said so in plain words: couldn't find key DB_PASS. Fix the key name in the manifest so it matches the Secret, apply again, and the container comes up. There is a close cousin of this error, secret "db-secret" not found, which shows up when the Secret is missing altogether, for instance if you applied the Pod before creating the Secret.

Try this

Run kubectl create secret generic db-secret --from-literal=DB_PASSWORD=supersecret on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Related