>_SecOpsLog_
CoursesSoftware supply chain securityDevSecOps: shift left, assume breach

DevSecOps: shift left, assume breach

Security as everyone’s job, in the pipeline.

Advanced12 min · lesson 3 of 18

A restaurant does not pass its health inspection because one person checks the plates on the way out the door. It passes because the cook chopping onions wipes the board, the person at the fryer watches the oil temperature, and someone tosses the cream that expired yesterday. Clean is built into every station, all shift. DevSecOps (short for development, security, and operations working as one group instead of three separate ones) asks software teams to work the same way. Security stops being a final gate that another team runs the day before release. It becomes a set of habits wired into every step, run by machines, and owned by everyone who touches the code.

For the software supply chain, the ingredients are the open-source libraries you pull in, the base image your container starts from, the scripts that build it, and the machine doing the building. Any one of those can be swapped, poisoned, or stolen from. So the supply-chain version of the idea is concrete: the checks that answer 'can we trust this artifact?' (scanning for secrets and known-bad code, signing what you build, recording where it came from, and verifying all of that before you run it) fire automatically on every change. Nobody has to remember. The pipeline remembers for them.

Shift Left Means Catching It While It's Cheap

A cracked egg costs nothing to throw away while you are holding it over the bowl. The same crack, baked into the cake and carried to the table, costs you the cake, the customer's evening, and a story they tell their friends. Bugs behave the same way. A hardcoded password caught in your editor is a ten-second fix. That same password, pushed to a public repository and scraped by a bot within minutes, is a 2 a.m. phone call and a rotation of every credential that ever touched it.

'Shift left' is that idea drawn on a timeline. Trace one change from left to right: you write it, you open a pull request, it gets built, it gets released, it runs in production. Left is early and cheap. Right is late and expensive. Shifting left means pushing each check to the earliest spot where it can catch its kind of problem. Secret scanning the moment you commit. Static application security testing (reading your source code for dangerous patterns without running it, shortened to SAST) on the pull request. Dependency and image scanning at build. Signing and provenance at release. Signature checks at deploy. Each guard stands at the earliest door it can actually watch.

Here is the earliest door in action. A pre-commit hook (a small program git runs on its own right before it records a commit) scans what you staged and refuses to save it if it finds a secret. Gitleaks (a tool that hunts for hardcoded passwords, tokens, and keys) is the common choice. You wire it in with a few lines:

.pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.21.2
    hooks:
      - id: gitleaks

Now a developer stages a config file that accidentally carries a live cloud key and tries to commit it:

terminal
git add config/settings.py
git commit -m "wire up staging config"
output
Detect hardcoded secrets.................................................Failed
- hook id: gitleaks
- exit code: 1
 
    Finding:     AWS_ACCESS_KEY_ID=REDACTED
    Secret:      REDACTED
    RuleID:      aws-access-token
    Entropy:     3.982543
    File:        config/settings.py
    Line:        14
    Fingerprint: config/settings.py:aws-access-token:14
 
11:04AM INF scanned ~2143 bytes (2.14 KB) in 8.19ms
11:04AM WRN leaks found: 1

The commit never happened, so the key never entered your history. That matters because git history is effectively permanent: pulling a secret back out later means rewriting every branch, and you still have to rotate the key because you must assume a bot already grabbed it. Caught at the commit, it is a one-line edit and nobody races to rotate anything.

A pre-commit hook is advice, not a wall
That hook runs on the developer's own laptop, and anyone can walk straight past it with git commit --no-verify. A tired engineer racing a deadline will. Client-side hooks are a helpful nudge, not an enforced control. The real barrier has to live somewhere the developer cannot switch off: the same secret scan, run again in continuous integration (the shared server that builds every push, CI for short), where a failure blocks the merge. Shift left, but keep a copy of the check on the right where nobody can turn it off.

Assume Breach

You lock your front door. You still do not leave your passport and car keys on the kitchen table, because part of you plans for the door failing. That is the mindset in one sentence: build as if something is already compromised. Assume one dependency is poisoned, one build token is stolen, one server is owned. A single failure like that should cost the attacker a little, not everything. The way you get there is old-fashioned: defense in depth (more than one independent barrier), least privilege (every token and account can touch only what it truly needs), and having each stage check the previous stage's work instead of trusting it.

Assume breach stays abstract until a real library goes bad, so make it concrete. Most of the code you ship is not yours; it is hundreds of open-source packages you pulled in, and each one is a door someone else built. To watch those doors you first have to know they exist. A software bill of materials (a full, machine-readable ingredients list of every package and version baked into your artifact, said as SBOM) gives you exactly that. Syft, from Anchore, generates one:

terminal
syft scan ghcr.io/acme/checkout:1.4.2 -o cyclonedx-json=checkout.cdx.json
output
 ✔ Loaded image                    ghcr.io/acme/checkout:1.4.2
 ✔ Parsed image                    sha256:9b2af1c0e2...e1c1
 ✔ Cataloged contents              sha256:4d2e8a77b1...9a
   ├── ✔ Packages                    [214 packages]
   ├── ✔ File digests                [1,203 files]
   ├── ✔ File metadata               [1,203 locations]
   └── ✔ Executables                 [78 executables]

Now hold that ingredients list up against the public catalog of known flaws. Every serious vulnerability gets a CVE (Common Vulnerabilities and Exposures, basically a shared catalog number like CVE-2021-44228 that everyone uses to name the same bug). Grype reads your SBOM and matches every package against that catalog:

terminal
grype sbom:checkout.cdx.json
output
 ✔ Vulnerability DB                [no update available]
 ✔ Scanned for vulnerabilities     [39 vulnerability matches]
   ├── by severity: 2 critical, 5 high, 20 medium, 12 low, 0 negligible
   └── by status:   35 fixed, 4 not-fixed
 
NAME        INSTALLED          FIXED-IN           TYPE          VULNERABILITY   SEVERITY
log4j-core  2.14.1             2.15.0             java-archive  CVE-2021-44228  Critical
log4j-core  2.14.1             2.16.0             java-archive  CVE-2021-45046  Critical
libssl3     3.0.2-0ubuntu1.15  3.0.2-0ubuntu1.16  deb           CVE-2024-0727   Medium

That top line is Log4Shell, the flaw that had half the internet patching through a December weekend in 2021. Your own code never changed; a library buried three levels deep did. Because this scan runs on every build and not once a quarter, the day the catalog learns about a new flaw in something you ship, your next build fails and hands you the package name and the exact fixed version that closes it. The attacker's whole play is to hide in a dependency you forgot you had. Your counter is to keep the ingredients list and re-check it constantly.

Don't Trust, Verify

A wax seal on a letter did two jobs at once: it proved who sent it, and it showed whether anyone had opened it on the way. A cryptographic signature does the same for a container image. Assume the registry you pull from has been breached and an attacker has quietly replaced your image with theirs. If you signed yours at release and you verify that signature before running it, the swap fails loudly. Cosign, from the Sigstore project, does keyless signing, meaning it ties the signature to your pipeline's own identity through OpenID Connect (OIDC, a standard way for one system to prove who it is to another) rather than a private key that some human has to guard. Verifying looks like this:

terminal
cosign verify \
  --certificate-identity-regexp '^https://gitlab.com/acme/checkout//.gitlab-ci.yml@refs/tags/v.*' \
  --certificate-oidc-issuer https://gitlab.com \
  ghcr.io/acme/checkout:1.4.2
output
Verification for ghcr.io/acme/checkout:1.4.2 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
 
[{"critical":{"identity":{"docker-reference":"ghcr.io/acme/checkout"},"image":{"docker-manifest-digest":"sha256:9b2af1c0e2...e1c1"},"type":"cosign container image signature"}}]

Read the two flags, because they carry the whole point. You are not asking 'is this image signed?' Anyone can sign anything. You are asking 'was this signed by my release pipeline, on a real version tag, and proven in the public transparency log?' (a public append-only ledger where every signature is recorded so it cannot be quietly forged after the fact). Point the identity at an attacker's fork and verification fails, even though their image is perfectly signed by them. Run that same check in your deploy step and an image that does not carry your pipeline's signature never starts. That is assume breach paying off: the registry can lie to you, and you still do not run the wrong thing.

Automate Trust, or It Rots

Every control in this lesson shares one rule: a machine enforces it, on by default, or it does not count. A check that depends on a person remembering to run it will be skipped the first busy Friday, and after that you have paperwork instead of protection. So the secret scan runs in CI, not only on laptops. The image scan fails the build on a critical finding instead of printing a warning nobody reads. The signature check runs at deploy and refuses to start an unsigned image. The reason you write each control down as a pipeline step is that the pipeline does not get tired and does not skip ahead.

There is a failure mode on the other side, though. Push too hard and you get the opposite of security. A gate that is slow, cries wolf with false alarms, or blocks a merge over a flaw with no available fix will get muted, wrapped in a permanent exception, or deleted outright, and then you are back to nothing. So every check earns its place by being fast, by pointing at a real and fixable thing, and by failing with a message that tells you what to do next. The strongest pipeline is not the one with the most gates on a slide. It is the one your team leaves switched on.

Each control at the earliest door it can watch
1Commit
gitleaks blocks secrets before they enter history
2Pull request
SAST reads the diff for dangerous code
3Build
syft builds the SBOM, grype scans it for CVEs
4Release
cosign signs the image, provenance recorded
5Deploy
cosign verify, fail closed on a bad signature
Quick check
01Every laptop on your team runs gitleaks as a pre-commit hook, and this morning it blocked a live AWS key in config/settings.py. Why is that hook still not an enforced control?
Incorrect — The blocked commit prints Entropy 3.982543 next to RuleID aws-access-token, so shaped rules fire alongside the entropy check. Detection quality is a tuning question anyway, not the reason this hook can be walked around.
Incorrect — Staged content is precisely what the commit is about to record, so that scope is the right one. An unstaged file is not part of the commit at all, so nothing rides along.
Correct — The hook lives on the machine of the person it constrains, which means that person decides whether it runs. Put the identical scan on the shared build server, where a failure blocks the merge, and the decision stops being theirs.
Incorrect — Rotation is the expensive half. Once a key reaches a public repo you assume a bot already read it, and the value stays in history until every branch gets rewritten, so the cheap moment is before the commit exists.
02You run cosign verify with --certificate-identity-regexp '^https://gitlab.com/acme/checkout//.gitlab-ci.yml@refs/tags/v.*' and --certificate-oidc-issuer https://gitlab.com. What do those two flags let you demand?
Correct — The regexp pins the workflow file and the refs/tags path, the issuer pins who vouched for that identity, and the output confirms the claims exist in the transparency log. An attacker's fork fails on the identity even with a flawless signature.
Incorrect — Pinning a base image by digest is a separate habit you enforce in the Dockerfile and the build. These flags constrain who signed the artifact and from which workflow reference, and never read your FROM line.
Incorrect — Nothing binds a signature to scan results. Grype answers the vulnerability question against your SBOM, and a pipeline is free to sign an image full of criticals, so verification cannot stand in for scanning.
Incorrect — A valid certificate is the weak version of the question, and anyone can obtain one for their own build. The regexp is exactly what turns a yes-it-is-signed answer into a yes-my-pipeline-signed-it answer.
03On a build where none of your own source changed, grype prints 'log4j-core 2.14.1 FIXED-IN 2.15.0 CVE-2021-44228 Critical'. What is the reading, and what do you do?
Incorrect — Read the NAME column: the flaw sits in log4j-core 2.14.1, a package you pulled in rather than a file you wrote. Rewriting your logging calls leaves the vulnerable library exactly where it is.
Incorrect — This line is a match between your ingredients list and the public catalogue, which says a shipped library has a known flaw. A clean runner rebuilds the same 2.14.1 and reports the same critical.
Incorrect — The vulnerable code travels inside your artifact whichever way your own source uses it, and grype only marks a finding not-fixed when no fix exists. Here it names 2.15.0, so there is a fix to take.
Correct — This is Log4Shell surfacing in a dependency buried a few levels down. Your source stayed still and the catalogue moved, which is the whole reason the scan runs on every build rather than once a quarter.

Concrete first move if you own a pipeline today: add cosign verify to your deploy job with your own --certificate-identity, and set the job to fail closed so a failed check stops the rollout instead of warning past it. An unsigned or swapped image gets turned away at the door before it can run, and 'assume breach' stops being a slogan and becomes one line of YAML you can point at.

Try this

Run git add config/settings.py on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a pre-commit hook is advice, not a wall. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Anatomy of real supply-chain…Securing the source

Related

What CI is, and where Jenkins fitsJenkins foundations, done rightThe .gitlab-ci.yml modelSecure CI/CD with GitLabSupply-chain threat modelSoftware supply chain in depthCI/CD interview questionsInterview prep