Hermetic & reproducible builds
Pinned inputs, isolation, bit-identical output.
In March 2024 a Microsoft engineer went looking for half a second of unexplained delay when logging into a server over SSH (Secure Shell, the standard way to open a remote command line). That half second led him to a backdoor in XZ Utils, a compression library that ships on nearly every Linux machine. It was catalogued as CVE-2024-3094 (CVE, Common Vulnerabilities and Exposures, is the public register of known flaws). The malicious code had gone out weeks earlier in releases 5.6.0 and 5.6.1, and here is the part worth sitting with: it existed only inside the release tarballs, the packaged .tar.gz files people download, and never appeared anywhere in the project's Git history. The attacker got in through the build. A doctored helper file called build-to-host.m4 and a scrambled payload rode along in the tarball that autotools produces (autotools being the script generator that turns a project's source tree into a ready-to-compile package), while the source everyone reviewed stayed clean. The trick worked because almost nobody rebuilds a release from source and checks that the bytes they get back match the bytes that were published. Two build properties would have exposed it. A reproducible build lets an outsider rebuild the release and compare it byte for byte. A hermetic build keeps anything outside the declared, reviewed inputs from reaching the build at all. Provenance, which fills the rest of this section, is only ever as trustworthy as the build it describes.
Reproducible: same source in, the same bytes out
A recipe that says 350 degrees for 40 minutes should give you the same cake in any kitchen. A build is reproducible when it holds to that same standard: the same source and the same declared inputs produce output that is identical byte for byte, whoever runs it, on whatever machine, at whatever hour. It is a strong claim, and most toolchains fail it out of the box. Watch two archives of a directory nobody touched come out different.
$ tar czf app.tar.gz build/ ; sha256sum app.tar.gz9f2c0b6b1d8e4a77c0b2e1f5a3d9c8b41e7f0a2c6d5b3e9f1a4c7d0b2e5f8a1c3 app.tar.gz$ sleep 2$ tar czf app.tar.gz build/ ; sha256sum app.tar.gz41ab77e0c3d9f2b5108c6a4e7d1b0f93a2c5e8d7b6f4a1c0e3d9b2f5a8c1e4d70 app.tar.gz# ^ same files in build/, different bytes on disk
Nothing in build/ changed, yet the two digests disagree. A digest is the sha256 fingerprint printed above, a short string that changes completely if a single byte of the file changes. The nondeterminism here, meaning the build refusing to settle on one answer, is baked into everyday tools. gzip writes the source filename and the current time into its header. tar records each file's mtime (modification time), its uid and gid (the numeric user and group that own it), and whatever order the filesystem happened to return when the directory was listed. C compilers embed __DATE__, the absolute path of the directory you built in, and symbol orderings that shift with your locale settings. Every one of those is a clock or a machine detail leaking into the file you ship.
Pinning the clock: SOURCE_DATE_EPOCH and a fixed toolchain
The Reproducible Builds project settled on one lever for the timestamp problem: SOURCE_DATE_EPOCH, an environment variable holding a UNIX timestamp (a plain count of seconds since the start of 1970, which is how computers store a moment in time). Tools that honor it clamp any timestamp newer than that value down to it, so "now" stops leaking in. Set it from the source itself, the date of the commit, rather than from the wall clock. Then pin whatever nondeterminism your archiver has left.
export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) # commit time, not wall clocktar --sort=name --format=posix \--pax-option='exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime' \--mtime="@$SOURCE_DATE_EPOCH" \--owner=0 --group=0 --numeric-owner \-cf app.tar build/gzip -n -9 < app.tar > app.tar.gz # -n: omit filename + timestampsha256sum app.tar.gz# --- run the whole script twice, an hour apart, on two machines ---$ bash pack.sh ; bash pack.shb7e0f3a2c14d59f6e8a1b0c3d7f2a5e9c4b6d1f8a0c2e5b3d7f9a1c4e6b8d0f32 app.tar.gzb7e0f3a2c14d59f6e8a1b0c3d7f2a5e9c4b6d1f8a0c2e5b3d7f9a1c4e6b8d0f32 app.tar.gz# ^ bit-for-bit identical
Run that script twice, an hour apart, on two different machines, and the two sha256sum lines come out identical. Each flag closes one leak. --sort=name fixes the file order. --mtime with SOURCE_DATE_EPOCH fixes the time. --owner=0 --group=0 --numeric-owner fixes ownership. gzip -n stops gzip stamping a name and a timestamp into its header. Pinning the toolchain matters every bit as much, because the digest only holds if everyone compiles with the same compiler version, and code generation and symbol ordering drift between releases. This is where reproducibility sits alongside the rest of the section. A signature tells you who produced an artifact. Provenance tells you how it was built. A reproducible build lets anyone confirm that "how" by rebuilding and comparing, taking nobody's word for it. Debian and Arch run rebuilder farms that do exactly this at scale, flagging any published binary that fails to match a fresh rebuild.
Reading a mismatch with diffoscope
A digest that does not match tells you about as much as a smoke alarm going off in an empty house: something is wrong somewhere, good luck finding it. diffoscope walks you to the smoke. It unpacks both artifacts recursively (archives, filesystem images, ELF sections, which are the labelled chunks inside a compiled Linux binary, even gzip streams buried inside other files) and prints a readable diff of the first thing that genuinely differs.
$ diffoscope app-1.tar.gz app-2.tar.gz--- app-1.tar.gz+++ app-2.tar.gz├── filetype from file(1)│ │ @@ -1 +1 @@│ │ -gzip compressed data, was "app.tar", last modified: Thu Jul 16 09:41:02 2026│ │ +gzip compressed data, was "app.tar", last modified: Thu Jul 16 09:41:04 2026├── app.tar│ ├── file list│ │ @@ -1,3 +1,3 @@│ │ --rw-r--r-- 0 alice alice 512 2026-07-16 09:41:02 build/main.o│ │ +-rw-r--r-- 0 alice alice 512 2026-07-16 09:41:04 build/main.o
Here the diff points straight at timestamps. The gzip header's "last modified" and each file's tar mtime sit two seconds apart. That is a determinism bug rather than tampering, and SOURCE_DATE_EPOCH fixes it. diffoscope earns its keep on the uglier cases. If it turned up an extra object file, a changed symbol table, or an embedded build path like /home/alice/…, you would be looking at either an input that got in undeclared or a genuine injection, and you would want to know which one before you signed anything.
Hermetic: sealed off, every input pinned
Hermetic is the old word for airtight, the way a sealed preserving jar keeps the outside out. Reproducibility is about consistency; hermeticity is about control, and having one does not hand you the other. A build is hermetic when it declares every input up front and runs sealed off from everything else: no network, no ambient host tools, no undeclared files lying around, no reading of the clock. Nothing outside the reviewed input set can influence or contaminate what comes out. That is what makes provenance believable at all, because a builder can only honestly attest to inputs it fully controls. SLSA (Supply-chain Levels for Software Artifacts) already requires an isolated builder emitting unforgeable provenance at Build L3. Its v1.0 spec lists hermetic and reproducible builds above that baseline, as future directions that harden provenance further by shrinking what an undeclared input could ever smuggle in. Two systems enforce hermeticity seriously today. Nix builds each derivation (its term for one build step plus its declared inputs) in a sandbox with no network access, makes a fixed-output derivation declare its content hash up front, and names every output after a hash of all its inputs.
$ nix build .#app --rebuild$ nix path-info ./result/nix/store/2q8v6h7k1r0mxz3d9c4b5a1f8s6g7h2j-app-1.0# --rebuild builds a second time in a fresh, network-less sandbox and# aborts if the result is not bit-identical. On a nondeterministic output:$ nix build .#app --rebuilderror: derivation '/nix/store/…-app-1.0.drv' may not be deterministic:output '/nix/store/2q8v…-app-1.0' differs from'/nix/store/2q8v…-app-1.0.check'
nix build --rebuild is a reproducibility check wired into the tool itself. Bazel takes the same line for its actions: a sandbox that denies network by default, and external dependencies pinned by sha256. The trade-off is real. Hermetic builds run slower and feel stricter, because every dependency has to be vendored or hash-pinned instead of pulled over the network whenever it is convenient, and reproducibility becomes ongoing whack-a-mole against new sources of nondeterminism. That cost is precisely what buys you a build a compromised worker cannot quietly rewrite.
# .bazelrc — force hermeticity on every buildbuild --incompatible_strict_action_env # fixed PATH, scrub the ambient envbuild --sandbox_default_allow_network=false # actions get no network# MODULE.bazel — every external input pinned by content hashbazel_dep(name = "abseil-cpp", version = "20240722.0")http_archive = use_repo_rule("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")http_archive(name = "zlib",sha256 = "9a93b2b7dfdac77ceba5a558a580e74667dd6fede4585b91eefb60f03b72df23",strip_prefix = "zlib-1.3.1",urls = ["https://zlib.net/fossils/zlib-1.3.1.tar.gz"],)# A clean rebuild yields the same binary digest:$ bazel build //:app && sha256sum bazel-bin/app$ bazel clean && bazel build //:app && sha256sum bazel-bin/app3c1d9a02f7e5b4c8d0a6f1e3b9c7d2a5… bazel-bin/app3c1d9a02f7e5b4c8d0a6f1e3b9c7d2a5… bazel-bin/app
An http_archive with no sha256, or a curl | bash sitting inside a build step, is the exact hole both tools were built to close: an input nobody hashed, free to change under you between one build and the next. That is the class of gap the XZ tarball walked through. When a rebuild's digest fails to match, diffoscope plus a short decision tree tells you whether you are looking at a dull determinism bug or something worth escalating.
git log -1 --pretty=%ct so "now" stops leaking into the artifact.The next lesson takes the digest you now know how to reproduce and turns these build properties into evidence you can hand to someone else: the provenance a SLSA builder emits, and how to read the predicate that records which pinned inputs and which toolchain version produced that exact digest.
Try this
Run tar czf app.tar.gz build/ ; sha256sum app.tar.gz on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.
Takeaway
The trap worth remembering here: container images stamp times into places the digest can see. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.