Two teams you have never met rebuild your release from the same commit and land on byte-for-byte identical output. A colleague asks what that actually buys you. Which reading holds up?

Neither build added anything that was not on the ingredient list you handed it, though the committed source could still be malicious. Identical bytes out of two unrelated machines is strong evidence that nothing was injected while the build ran. What the code itself does is a separate question you answer by reading it.

diffoscope narrows the two archives down to one line in the file list: app/main.py stamped 09:14:24 in build-a.tgz and 09:16:58 in build-b.tgz. You then export SOURCE_DATE_EPOCH="$(git log -1 --format=%ct)" and pass --mtime="@${SOURCE_DATE_EPOCH}" to tar. Why do the digests line up after that?

Every builder of that commit derives the same number from git, so the stamp written into the archive no longer depends on when you build. Git log -1 --format=%ct reads a property of the commit, so you, your teammate, and a stranger all compute the identical value. That is what takes the clock out of the digest.

You wrap a build step in unshare --user --map-root-user --net and it dies in about a second with 'curl: (6) Could not resolve host: deb.example.com', though the same step passes on your normal builder. What is the right response?

Treat the red build as a finding: fetch and verify that dependency earlier, then run the sealed step against what you already checked. The seal turned a silent build-time download into a visible failure. Pull the toolchain in an earlier phase, check it against a fingerprint you pinned, and the sealed step has nothing left to reach for.

Courses Software supply chain securityHermetic & reproducible builds

Hermetic & reproducible builds

Same inputs, same bytes, no network.

Advanced14 min · lesson 8 of 18

Hermetic Builds: Seal The Kitchen

A hermetic build is a sealed kitchen. Before the doors lock, you measure out every ingredient and set it on the counter: this exact flour, this exact sugar, weighed and labeled. Then the doors shut. No delivery van pulls up mid-bake. No one leans through a window to hand the cook a mystery packet. Whatever comes out was made from what you laid out, and nothing else. In build terms, a hermetic build runs in isolation with every input declared ahead of time and no access to the network or to whatever happens to be installed on the machine. It cannot download a dependency while it runs, read a secret out of the environment, or quietly use a tool that some other job left behind on the builder.

That seal is the whole point for a defender. If a build can reach the internet while it runs, then whoever controls what it fetches controls what it ships. Your build server can be patched, your pipeline audited, your artifact signed, and none of it saves you, because the signature faithfully wraps whatever the attacker served. The provenance (a signed record of how and from what an artifact was built) becomes a lie told with a straight face. So the defender's move is to make reaching out impossible, and to make any attempt to reach out fail loudly, where you can see it.

Here is what that looks like. Take a build step that grabs a toolchain (the compilers and helper programs a build needs) while it runs, and wrap it in a private network namespace (a Linux kernel feature that gives a process its own view of the network, in this case an empty one with no route out).

terminal

# The build wants to download a toolchain while it runs.
# Run that step sealed off from the network and watch what happens.
$ unshare --user --map-root-user --net -- \
    curl -sS --max-time 5 -o toolchain.tar.gz \
    https://deb.example.com/toolchain.tar.gz

output

curl: (6) Could not resolve host: deb.example.com

The --user --map-root-user part is what lets an ordinary account do this at all. It hands you root inside a throwaway user namespace, which is the one privilege you need to create that empty network namespace, without ever being real root on the box. Inside it, curl has nowhere to send a DNS lookup, so the name never resolves and the fetch dies in about a second.

That failure is a feature. On a normal builder the fetch would have quietly succeeded, and you would never learn that your build leaned on a server you do not control. Sealed off, the hidden dependency turns into a red build you can fix. The fix is not to open the network back up. It is to pull that toolchain in a separate, checked step, before the sealed build ever starts.

Reproducible Builds: Same Inputs, Same Bytes

A reproducible build is a recipe so precise that any cook, in any kitchen, on any day, turns out a cake identical down to the crumb. Same weight. Same shape. Same everything. In software, the same source and the same declared inputs always produce the byte-for-byte identical file, no matter who runs the build or when. This is a verification superpower. If two strangers build from your source and land on the exact same bytes, you have strong evidence that neither of them slipped anything in, because a single injected byte would change the fingerprint.

The fingerprint here is a hash, or digest: sha256 (a function that turns any file into a short, fixed-length string, where changing one bit of the input changes the string completely and unpredictably). Reproducibility is hard because builds are full of hidden clocks and dice. Timestamps get baked in. File order depends on the filesystem's mood. Your locale (the machine's language and region settings) changes how names sort. Watch a plain tar archive (tar, the standard Unix tool that bundles many files into one) give you away.

terminal

$ mkdir app && echo 'print("hi")' > app/main.py
$ tar czf build-a.tgz app/
# A fresh checkout on another runner stamps every file with the current time.
$ touch app/main.py
$ tar czf build-b.tgz app/
$ sha256sum build-a.tgz build-b.tgz

output

54dd12e5628bd1b7e84cad636100300e2b304f82e58d46cbd18e8497df25778c  build-a.tgz
ea8516a4bd15e1dd7291528083e0d131b7abfe47f6999ea0592f9176c46e93c1  build-b.tgz

Same source, same file contents, two different fingerprints. Nothing in the code moved, so what did? A tar archive stores more than your file contents. For every file it also writes down the modification time, the owner, and the permissions. The second checkout stamped main.py with a fresh "now," and that one moved timestamp ripples through every byte after it. Point a microscope at the two archives and you can see the exact culprit.

terminal

$ diffoscope build-a.tgz build-b.tgz

output

--- build-a.tgz
+++ build-b.tgz
├── build-a.tar
│ ├── file list
│ │ @@ -1,2 +1,2 @@
│ │  drwxr-xr-x build/build         0 2026-07-17 09:14:24 app/
│ │ --rw-r--r-- build/build        12 2026-07-17 09:14:24 app/main.py
│ │ +-rw-r--r-- build/build        12 2026-07-17 09:16:58 app/main.py

diffoscope (a tool that unpacks two files as deep as it can and reports every difference in plain terms) settles it. It decompressed both archives and surfaced exactly one difference, and it was not the gzip framing (gzip, the compressor that squeezes a .tgz down to size). The difference sits inside the tar: a single line in the file list, main.py's timestamp, a couple of minutes apart. Kill the clocks and the builds converge. The standard control is an environment variable that most build tools already honor.

SOURCE_DATE_EPOCH is a number of seconds that tells every tool: pretend the build happened at this exact moment. Set it to your commit time (a value that is the same for everyone who builds that commit) and every tool that respects it stops writing "now." Combine that with sorted file order, normalized ownership, a fixed locale, and a gzip told to keep its mouth shut.

terminal

$ export SOURCE_DATE_EPOCH="$(git log -1 --format=%ct)"
$ export LC_ALL=C
$ tar --sort=name \
      --mtime="@${SOURCE_DATE_EPOCH}" \
      --owner=0 --group=0 --numeric-owner \
      --pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime \
      -cf - app/ | gzip -n > build-1.tgz
$ tar --sort=name \
      --mtime="@${SOURCE_DATE_EPOCH}" \
      --owner=0 --group=0 --numeric-owner \
      --pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime \
      -cf - app/ | gzip -n > build-2.tgz
$ sha256sum build-1.tgz build-2.tgz

output

c1f5a9d3e8b7460a2d9c4f1e6b83a7d05c2e9f4180a3b6c7d8e9f0a1b2c3d4e5  build-1.tgz
c1f5a9d3e8b7460a2d9c4f1e6b83a7d05c2e9f4180a3b6c7d8e9f0a1b2c3d4e5  build-2.tgz

Each flag closes one leak. --sort=name fixes the order files enter the archive, instead of leaving it to the filesystem. --mtime pins every file's timestamp to that one pinned moment. --owner=0 --group=0 --numeric-owner strips out whose machine built it. The --pax-option line tames tar's extended headers: it gives them a fixed name instead of one built from the process id, and deletes the access and change times tar would otherwise tuck inside. gzip -n tells gzip to write no timestamp and no original filename into its header. In this pipeline gzip already reads from a pipe and leaves those blank, so -n turns a lucky default into a guarantee, and it becomes essential the moment anyone compresses a named file instead. Now the two builds match, and so would a build on your laptop, your teammate's, and a stranger's.

The clock is not the only source of randomness

Pinning the timestamp fixes the most common break, not every one. Absolute build paths get compiled into binaries (normalize them with a flag such as -ffile-prefix-map on GCC and Clang, the two main C and C++ compilers). Parallel steps can finish in a different order. Some tools seed a hash table from a random value at startup. Set LC_ALL=C so sorting does not shift with the machine's language. Reproducibility is the habit of hunting down every hidden clock and die, not one switch you flip.

Seal The Fetch, Then Lock The Build

Put the two ideas together with a loading dock. Deliveries arrive at the dock, where you check each crate against the manifest before it comes inside: right supplier, right weight, tamper seal intact. Only then does it roll into the kitchen, and only then do the doors lock for the bake. A trustworthy build works the same way, in two phases. First a fetch phase with the network on, where you pull every dependency and check each one against a fingerprint you pinned in advance. Then a build phase with the network off, running only against what you already verified.

Pinning by fingerprint is what makes the dock check real. You do not ask for "the latest requests library." You ask for exactly this version with exactly this hash, and if the bytes that arrive do not match, you turn the delivery away at the door. Here is that pin written down.

requirements.txt

# Every dependency is pinned to a version AND a content hash.
# pip will refuse anything whose bytes do not match.
requests==2.31.0 \
    --hash=sha256:942c5a758f98d790eaed1a29cb6eefc7ffb0d1cf7af05c3d2791656dbd6ad1e1 \
    --hash=sha256:58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f

Now run the install in a mode that treats those hashes as law. pip (the Python package installer) computes the hash of every file it pulls and refuses anything that does not match. If a mirror has been poisoned or a package swapped underneath you, the numbers stop lining up and pip stops cold.

terminal

$ pip install --require-hashes --no-deps -r requirements.txt

output

Collecting requests==2.31.0 (from -r requirements.txt (line 3))
  Downloading requests-2.31.0-py3-none-any.whl (62 kB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    requests==2.31.0 from https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl (from -r requirements.txt (line 3)):
        Expected sha256 942c5a758f98d790eaed1a29cb6eefc7ffb0d1cf7af05c3d2791656dbd6ad1e1
        Expected     or 58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f
             Got        3b8e6faa1c9d0b2e7f4a55d0c1e9a8b7d6c5f4e3a2b1c0d9e8f7a6b5c4d3e2f1

That is the whole game in one error message. The attacker did their part, the poisoned file showed up at the door, and the build refused it before a single line of it ran. With the fetch verified, the build phase can be sealed with confidence, because there is nothing left for it to reach out and grab.

A build that fetches at build time cannot be trusted

The single most common break in build integrity is a step that downloads something while it runs: a script piped straight into a shell, a binary from an unpinned URL, a dependency resolved at "latest." That fetch is an input the attacker controls. It does not matter how high your SLSA (Supply-chain Levels for Software Artifacts, a graded checklist for how tamper-resistant your build is) level climbs or how clean your signature looks, because a signed, high-level artifact will faithfully package whatever the attacker served. Vendor and pin your inputs, verify them in a fetch phase, and run the real build with the network off.

Two-phase build: verify, then seal

1Pin inputs

version + sha256 in a lockfile

2Fetch phase

network on, check every hash

3Seal the build

network off, clock pinned

4Deterministic artifact

same bytes every run

5Rebuild to verify

two builders, matching digest

Checking Your Own Builds

You do not need Bazel or Nix (build systems designed so that hermetic, reproducible behavior is the default instead of something you bolt on) to start. You need three habits. Build the artifact twice and compare the digests; if they differ, you have non-determinism to hunt. Run your build under network isolation in continuous integration (CI, the automated system that builds and tests every change on the way in) and let it fail loudly the day someone adds a sneaky fetch. When two builds do not match, point diffoscope at them and it will tell you exactly which byte moved and why.

The reward is a receipt anyone can check. When your build is hermetic and reproducible, "trust our build server" turns into "here is the source, build it yourself, and confirm you get the same bytes I signed." That is the strongest form of build integrity there is, and it is the technical floor under the higher SLSA levels.

Wire the second build into CI: rebuild the artifact on a clean runner, diff its digest against the one you shipped, and fail the pipeline if they disagree. A reproducibility check that no one runs is only a claim. One that runs on every commit is a tripwire that trips the moment tampering happens.

Try this

Run tar czf build-a.tgz app/ on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the clock is not the only source of randomness. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

The SLSA framework & levelsTrusted, isolated builders

What CI is, and where Jenkins fitsJenkins foundations, done right The .gitlab-ci.yml modelSecure CI/CD with GitLab Supply-chain threat modelSoftware supply chain in depth CI/CD interview questionsInterview prep

Hermetic & reproducible builds

Hermetic Builds: Seal The Kitchen

Reproducible Builds: Same Inputs, Same Bytes

Seal The Fetch, Then Lock The Build

Checking Your Own Builds

Try this

Takeaway

Related